D-Lib Magazine
March 2001

Volume 7 Number 3

ISSN 1082-9873

The HeadLine Personal Information Environment

Evaluation Phase One

Anne Gambles
HeadLine Project Officer
The Library
London School of Economics and Political Science
United Kingdom
[email protected]

Introduction

This article briefly describes the HeadLine Personal Information Environment (PIE) and Resource Description Model (RDM). The results of the Phase One user evaluation study are presented, and the forthcoming Phase Two evaluation is discussed.

HeadLine (Hybrid Electronic Access and Delivery in the Library Networked Environment) is one of the hybrid library projects funded under the Electronic Libraries (eLib) Phase 3 program of the UK Higher Education Joint Information Systems Committee. The project is led by the London School of Economics and has the London Business School and the University of Hertfordshire as partners.

The aims of the Headline project

• To develop the concept of the hybrid library, going beyond resource discovery to resource access;

• To establish a working model for consistent access to library materials regardless of physical form;

• To implement the model in different types of working environments.

The project has focussed on providing access to information resources in two main subject areas: "Economics" and "Business and Management" but aims to demonstrate that the HeadLine model is scaleable to the wide range of subjects (and material types) to which libraries provide access.

The HeadLine PIE

HeadLine's hybrid library model, the PIE, is a web-based system that provides the user with a single interface to both print and electronic resources and services. The system aims to allow the user to go beyond resource discovery to resource access (from the bibliographic record to a means of accessing the material). Library materials are delivered to the user in a variety of formats: electronic, print, and document delivery via interlibrary loan.

The personal portal interface paradigm is a useful model for the hybrid library. What library users really want is their own personal library, containing all the resources they need and none of the resources that they don't need. Like hybrid libraries, portals are organized, mediated, and configurable, and they provide access to a wide range of materials. Personalized portal-type technology (such as that used by My Netscape and My Yahoo) enables the PIE to present an information environment that is tailored to user needs and allows user personalization.

The HeadLine hybrid library model contains metadata about its users -- their department, courses, etc. Hence, it is possible for the PIE to reduce information overload by presenting each user with a page (or pages) of information resources that are relevant to their subject area or course (e.g., Microeconomics). Users can create and customize their own PIE pages. They can build their own collections of resources and, if they wish, they can share these with other PIE users (such as researchers in the same field, students taking the same course, etc.).

The PIE interface

Figure 1 illustrates Rachel Smith's PIE account "default" page, which is named "r.j.smith's Page". Rachel owns this page, and she can both read and edit it. She can add or remove lists and resources from her page. Rachel is an Economics student; therefore, she is provided with access to the Economics page. The Economics page is owned and maintained by the Economics subject librarian. "Read" rights to the Economics page can be restricted to Economics staff and students, thus reducing information overload for those without interest in the subject. The HeadLine "All Resources" list is owned and maintained by the library. The "All Resources" list displays all resources currently available in the PIE (e.g., bibliographic and full-text databases, statistical datasets, and newspapers). In addition to reducing information overload, the PIE supports independent research; all PIE users have "read" rights to the "All Resources List".

Users can add a resource to their default page from another page by clicking on the blue arrow to the right of the resource title. This new resource will automatically appear in the "New Resources" list on the user's default page. The resources in the "New Resources" list can then be added to lists on any page to which the user has edit rights. To do this, and to perform other personalization functions, the user needs to access the customization screen by clicking on the "Customise" link at the top right of the page.

Additional personalization functions include:

• Creating new pages

• Renaming and deleting pages to which the user has edit rights

• Changing which page is the default page

• Filtering a page listing (This allows users to remove pages such as the "Economics" page from the navigation section on the left of the screen.)

• Adding a new Web link to one's lists (The user can add his or her own Web links -- resources that don't currently exist in the PIE.)

• Editing, creating, and deleting lists on the pages that can be edited

• Editing and deleting resources on the pages that can be edited.

Figure 1. The Headline PIE interface

The HeadLine Resource Description Model (RDM)

The HeadLine RDM describes collection level resources. Its relational structure can accommodate different versions of a single resource, each of which occurs at a different location and has different holdings (date range(s) available) and depth (bibliographic, abstract, full-text, etc.). A resource such as the journal American Economic Review may be available in more than one location -- perhaps on the shelf in the library as well as electronically via a web-based database like JSTOR. Resources that have been described within the RDM are presented to the user on pages in the PIE. In Figure 1, the full text of the American Economic Review can be accessed by clicking on the "www" location icon. The location of the American Economic Review on the library shelves can be found by clicking on the "shelf" icon. This takes the user to the HeadLine Shelfmark resource locator (SHERLOC, one of HeadLine's discovery to access tools), which presents a map of the LSE library's ground floor with a "+" marking the Periodicals Display Area.

To decide which resources are most relevant to their needs, users can search the PIE database (the resource metadata contained within the RDM). A search for resources that contain the word "Economics" in the keyword field, and that are of the type "Journal" will produce a results list containing all the economics journals that have been catalogued within the RDM.

HeadLine PIE: Evaluation Phase One

The first phase of the evaluation involved an analysis of PIE log data collected between May 2000 and January 2001, and a questionnaire survey of end users at each of the partner sites during November/December 2000. The Phase One evaluation aimed to report on the usage and usability of the HeadLine PIE from the end user's point of view and to enable recommendations to be made for improvements or further functionality. The Phase Two evaluation will be informed by the results of the Phase One evaluation.

HeadLine began in 1998 and was originally a three-year project. A six month extension has recently been granted and the project will now end on the 31st July 2001. The second phase of the evaluation will take place between April and July 2001 and will involve a more detailed evaluation of PIE end users at each site. In addition, the project will use focus groups to help evaluate administrators' use of the PIE. The second phase will also evaluate the wider community's response to the PIE.

PIE Log Data Analysis: Numbers of PIE Users

Currently there are three PIEs, one for each partner institution (London School of Economics, London Business School, and the University of Hertfordshire). PIE users at the partner sites include undergraduate and postgraduate students, academics, researchers, administrative staff and library staff. Guest users (from within the wider library community) are also using the PIEs. An analysis of PIE log data collected between May 2000 and January 2001 enabled the numbers of users using each PIE to be ascertained.

University of Hertfordshire (UoH) PIE users

In early May 2000, the very first version of the PIE was made avaialble to UoH Library staff and piloted to selected groups of undergraduate and postgraduate students. Forty students used the PIE between May and June 2000. The pilot testing enabled the project to identify necessary improvements to the PIE's functionality and to schedule these for later PIE versions.

In early November 2000, the UoH PIE was made available to undergraduate Microeconomics students during compulsory library training sessions that included demonstrations of the PIE and "hands on" exercises.

Between May 2000 and January 2001, the following numbers of people used the UoH PIE:

London School of Economics (LSE) PIE users

Towards the end of July 2000, the PIE was made available to academics and researchers in the LSE's Economics department, and to LSE Library staff. This PIE was subsequently presented to undergraduate Economics students in early October 2000.

The undergraduates, researchers, academics and administrators in the Economics department were invited via email to use the PIE. The email invitation described the PIE's functionality and pointed users towards the PIE online guide. Due to undergraduates' busy schedules at that time of year, it was not feasible to provide undergraduates with PIE training at the time of the PIE launch. However, training sessions are planned for undergraduates during the 2001 Spring term.

Between July 2000 and January 2001, the following numbers of people used the LSE PIE:

London Business School (LBS) users

In late July 2000, the LBS PIE was launched to LBS Library staff, and in early August 2000, this PIE was also launched to academics and Ph.D. students in the LBS Institute of Finance and Accounting. It was launched to full-time students in the LBS Masters in Finance course, and to MBA students taking Finance electives in late September 2000.

LBS target groups were also invited via email to use the PIE. Some of these groups received a PIE demonstration during a meeting with the liaison librarian.

Between July 2000 and January 2001, the following numbers of people used the LBS PIE:

Guest users

In mid October 2000, the three institutional PIEs were launched to guest users within the wider library community. In total, 58 guests used the three PIEs between October 2000 and January 2001. Many of the guests used more than one of the three PIEs. There were 46 individual guest users. Thirty-six guest users had only used one PIE, eight guest users had used two PIEs and two guest users had used all three PIEs.

Total number of PIE users

Between May 2000 and January 2001, 352 people used the three PIEs.

The Questionnaire Survey

The questionnaire aimed to ascertain:

• Who the respondents were, e.g., whether they were an undergraduate or postgraduate student;

• From which country they mainly accessed the PIE, and whether they accessed it on-campus or off-campus, as well as from personal PCs or shared PCs;

• How often they had used the PIE;

• How frequently particular library resources were used and how frequently the same resources had been used via the PIE;

• How often they had used particular pages within the PIE;

• How often personalization and search functions had been used;

• Whether there were any additional resources that the respondents would like the PIE to include;

• Respondents' opinions on the usefulness of personalization functions and their opinions of the PIE as a whole.

A set of core questions was included in each of three web-based questionnaires -- one for each institution. The questionnaires did not contain any compulsory fields. Several questions were tailored to the particular institution. For example, the question "Please indicate how often you have used the following PIE pages" included the list of PIE pages that were contained within the particular institution's PIE.

The three questionnaires can be found via the following links:

LSE questionnaire <http://www.headline.ac.uk/lsepie/LSEPIEquaire11-2000.html>
LBS questionnaire <http://www.headline.ac.uk/lbspie/LBSPIEquaire11-2000.html>
UoH questionnaire <http://www.headline.ac.uk/uhpie/UHPIEquaire11-2000.html>.

Sample group

The sample group was comprised of the following groups of PIE users:

* During the PIE training sessions, the UoH's Microeconomics students were asked whether they would like to take part in the evaluation. The 59 who agreed to take part were included in the sample group. The UoH students that were involved in pilot testing the first version of the PIE during May 2000 were not sampled. Most of these students had completed their studies by the time of the survey, and none of them had used the PIE since June 2000.

Invitations to complete the questionnaire were emailed to the sample group in late November and early December 2000. The invitations included a reminder of how to access the relevant PIE -- its URL and authentication instructions. As an incentive to complete the questionnaire, all respondents who correctly entered their email addresses were entered in a prize drawing (one drawing per institution) for a �10 Amazon book voucher. Shortly before the deadline for returning the questionnaire, reminders were emailed to those who had not yet completed the questionnaire. Questionnaire responses were received by email via a "forms mailer" based at the University of Birmingham.

Questionnaire response rate

Only 19 out of the 153 people in the sample group completed the questionnaire, a response rate of 12%. The response rate was slightly disappointing, which can perhaps be attributed to the time of year. Students may have been busy with end of term course work and/or exams. Table 1 shows the response rate by institution.

Table 1. Response rate by institution

Results and Discussion

User type

The majority of respondents (79%) were undergraduate students, 11% were Masters students, 5% were academics and 5% were administrators. Table 2 illustrates the number of respondents of each user type per institution.

Table 2. Number of respondents by user type

From which country do users mainly access the PIE?

All respondents indicated that they mainly accessed the PIE from the United Kingdom. Respondents were asked to list any other countries from which they had accessed the PIE. Only one respondent (a UoH undergraduate) listed other countries (Germany and Dubai). These results were expected, as none of the student target groups were distance learners.

From what type of PC did the users access the PIE?

Respondents were asked to indicate whether they accessed the PIE from a shared PC on campus, from a personal PC on campus, or from a PC at home. More than one option could be selected. Figure 2 shows the percentage of respondents who had used the PIE from each location. The majority of respondents had used the PIE from a shared PC on campus. The sample group included a high percentage of undergraduates, and undergraduates tend not to have access to a personal PC on campus. A higher percentage of respondents had used the PIE from home than had used the PIE from a personal PC on campus.

Figure 2. Percentage of respondents that had used the PIE from each type of PC

Respondents were asked to rate the usefulness of accessing the PIE from off-campus locations. They were given the following options: "very useful", "useful", "not useful", or "no opinion". 74% indicated that they thought off-campus access was either "very useful" or "useful". None of the respondents indicated that off-campus access was "not useful".

How many times had the respondents used the PIE?

The questionnaire asked respondents to indicate whether they had used the PIE "once", "2 to 5 times", "6 to 9 times", "10 to 19 times", or "more than 20 times". Respondents who had only used the PIE once were asked to explain why this was so. Figure 3 illustrates the number of times that respondents said that they had used the PIE.

None of the respondents indicated they had used the PIE more than nine times. Approximately two-thirds of respondents indicated they had used the PIE between two and five times. The highest frequency of PIE usage was observed in the UoH respondents. The four LSE respondents who had used the PIE only once each explained why this was the case:

"Am not sure as to its uses."
"I don't understand how to use it and I don't understand what its purpose is."
"I do not know how to use it or what it does."
"I was curious but haven't had time since."

As mentioned earlier, the undergraduates, researchers, academics and administrators in the LSE's Economics department had not received any PIE training. The LSE respondents' comments indicate that training which includes a statement about the purpose of the PIE and information on its functionality is needed.

Figure 3. Number of times that respondents had used the PIE

Resource Usage

Resource usage (not via the PIE)

Figure 4 shows the percentage of respondents who had used each particular library resource (not via the PIE). Exam papers were the most popular resource (88% of the 17 UoH and LSE respondents indicated that they had used these), followed by library web pages and the library catalogue or OPAC (74% of the total number of respondents had used these). 67% of the nine LSE respondents indicated they had used reading lists (they may have used them in paper, or electronic form either via the library catalogue or via course web pages). 63% of the total number of respondents had used electronic full-text journals. Bibliographic databases were the least used resources; only 37% of respondents indicated they had used bibliographic databases.

Figure 4. Percentage of respondents that had used each resource type

Respondents were asked to list any additional library resources they used. Two LSE undergraduates listed additional resources. One mentioned using the "Guardian/Independent" CD-Roms. The other respondent had used "solutions to math exercises" and "lecture notes".

Resource usage via the PIE

Figure 5 shows the percentage of users who had used each resource type via the PIE. Again, exam papers were the most popular resources (41% of the 17 UoH and LSE respondents indicated that they had used exam papers via the PIE), followed by the library web pages, financial/company information, and/or electronic full text journals (each had been used by 37% of respondents). Reading lists were the least used resource via the PIE. Only one (11%) of the nine LSE respondents mentioned using them.

Figure 5. Percentage of respondents that had used each resource type via the PIE

Were there any additional resources that users would like to access via the PIE?

Just one respondent (an LBS Masters in Finance student who had used the PIE from home) specified additional resources he would like to be able to access via the PIE: 'Reuters and Datastream etc.'. These two resources are available within the LBS PIE, but they are restricted to on-campus use, hence the respondent couldn't use them from home. The respondent asked "Why bother putting it (a resource) on if I can't access it?"

The project "user requirements" study found that users want off-campus access to resources. The project team had discussed whether resources restricted to on-campus use (e.g., IP restricted services or stand-alone applications such as some CD-Roms) should be shown to users accessing the PIE from outside the campus IP. We decided that the PIE should show users all the resources they are entitled to use, but resource locations restricted to on-campus use should be greyed out for off-campus users and should include a brief note (within the 'alt' text) explaining why they are restricted. Users would be able to identify all resources relevant to their research, regardless of the location from which they accessed the PIE. This would enable users to plan on using a relevant on-campus, restricted resources when they are on campus.

How many times had respondents used particular PIE pages?

The questionnaire asked how many times respondents had used their "default" page, the "All Resources" page, the "Customize" page, the "Help" page and the "Subject" page specifically set up for them (i.e., the "Microeconomics" page in the UoH PIE, the "Economics" page in the LSE PIE and the "Finance and Accounting" page in the LBS PIE). Table 3 illustrates the percentage of the total sample group who had used each of these pages a particular number of times.

Overall, the most frequently used page was the user's "default" page, followed by the "Subject" page (relevant to the sample group) and then the "All Resources" page. The "Customize" and "Help" pages were the least frequently used. Only one or two respondents from each institution had used these pages, and none had used them more than once.

Table 3. Percentage of total respondents who had used particular PIE pages

Personalization and Search Functionality

Frequency of usage

Respondents were asked how often they had used the following PIE functions:

• Add resource to default page
• Change name of default page
• Add web links to default page
• Create additional PIE pages
• Share pages with other PIE users
• Search for resources contained within the PIE database

None of the respondents indicated that they had used any of the functions more than five times. "Adding a resource" was the only personalization function used by LSE and LBS respondents. "Adding a resource" is perhaps the simplest personalization function; it doesn't require the user to visit the "Customize" page. The UoH respondents had made full use of all the personalization functions.

Table 4 illustrates the percentage of the total sample group who had used each function a particular number of times. "Adding a resource" was the most popular personalization function; 47% of all respondents indicated that they had used this function at least once. The next most popular personalization function was "adding a web link" (16%), followed by "changing the default page name" (10%) and "sharing pages" with other PIE users (10%). Creating additional PIE pages was the least popular personalization function; only 5% of users indicated they had created additional PIE pages.

Forty-eight percent of all respondents indicated they had searched for resources contained within the PIE database. The two (100%) LBS respondents, 53% of UoH respondents and 33% of LSE respondents had used the search functionality.

None of the respondents listed any additional functions they would like the PIE to include.

Table 4. Percentage of total sample group that had used each function a particular number of times

Usefulness of personalization and search functionality

Respondents were asked to indicate how useful they found the personalization and search functions. Table 5 illustrates how the total sample group rated the usefulness of the personalization and search functionality. "Adding a resource" and "searching for resources" contained within the PIE database were rated as the most useful functions (68% of respondents listed these functions as either "useful" or "very useful"). "Adding a web link" was the next most useful personalization function (63%), followed by "sharing pages" with other PIE users (58%), "creating additional pages" (53%) and, lastly, "changing the default page name" (42%). Relatively low percentages of respondents thought that "changing the name of a page", "adding a web link" and "creating additional pages" were "not useful" functions.

Table 5. Percentage of the total sample group that gave each rating for the usefulness of the personalization and search functionality

Usability of the PIE interface

Respondents were asked whether they found the PIE interface "Easy to use", "Satisfactory", "Difficult to use" or had "No opinion" on its usability. Figure 6 illustrates the percentage of respondents from each institution, and the percentage of total respondents who gave each rating.

Figure 6. Usability of the PIE interface

Fifty-eight percent of the total sample group indicated the PIE interface was either "easy to use" or "satisfactory". Respondents from both the LSE and the LBS indicated that the PIE interface was difficult to use (32% of the total sample group). None of the UoH respondents indicated the PIE interface was difficult to use. In fact, 60% of the UoH respondents indicated the PIE was easy to use. Perhaps this was a consequence of the UoH PIE training and the lack of PIE training at the LSE and the LBS.

Respondents were asked to comment on the usability of the PIE interface. Their comments reflected the results above, and the comments suggest that a lack of training can result in a low level of satisfaction with the usability of the PIE interface.

The PIE as a tool

The respondents were asked to rate the usefulness of the PIE tool. Figure 7 illustrates the percentage of respondents from each institution, and the percentage of total respondents who gave each rating ("Very good", "Good", "Satisfactory", "Poor", "Very poor" and "No opinion").

Figure 7. The PIE's rating as a tool to help its users

UoH respondents were the only ones that rated the PIE as a "very good" tool. Seventy-six percent of UoH respondents either indicated the PIE was a "very good" or a "good" tool. Just one LSE respondent (an undergraduate) rated the PIE as a "good" tool. Only one respondent (an LBS student) rated the PIE as a "poor" tool.

Thirty-seven percent of the total sample group thought that the PIE was either a "very good" or a "good" tool and 37% thought that it was a "satisfactory" tool. Five percent thought that the PIE was a "poor" tool and 21% had "no opinion".

One LBS respondent, a Masters in Finance student, completed the "further comments" section at the end of the questionnaire. He said:

"LBS Forum is quicker. Ultimately, for this site to work (become user-accepted) it has to offer more than the alternative (LBS Forum). So far it doesn't -- being able to add some web links doesn't do much for me; I can just as easy add a favorite on my PC."

(The Forum is the system through which the LBS provides access to Library resources and services.)

One of the UoH undergraduates commented that the PIE "System is very slow." The UoH PIE currently contains the largest number of PIE pages, so downloading and customizing pages can be slow.

Conclusions

The demand for access to library resources from off-campus locations is strong. Personalized library portals, such as the HeadLine PIE, can help to facilitate resource access, with the ultimate aim of enabling seamless access to resources regardless of the (authorized) user's location.

It was disappointing that, overall, the frequency of PIE usage had been quite low, but encouraging to see that those users who had been trained on the use of the PIE used the PIE more frequently than those who hadn't received training. It is hoped that increased promotion of the PIE to target groups and further training in the use of the PIE will increase the number of return users.

The statistics on library resource usage and usage via the PIE indicate the PIE is providing access to a broad enough range of resources for its current users. Users' views on the PIE's provision of restricted resource locations will be investigated in the Phase Two evaluation study. Perhaps the PIE should include an additional personalization function -- one that enables users to select whether or not the PIE should display resource locations not available to them from their current locations (e.g., from outside the campus IP).

The Phase Two evaluation will continue to monitor the use of particular PIE pages, personalization and search functions. It will attempt to find out whether usage changes as users become more experienced. For example, once users have their prime resources located on their default pages (through adding them from "subject" pages, the "All resources" page and from search results lists) does their use of subject pages decrease? Of course, this may depend on the type of user (researcher, academic, undergraduate student, etc.).

Perhaps certain experienced users will set up searches for new resources in their subject areas (saving them as lists on their default page) and set them to run at frequent intervals. The Phase Two evaluation will investigate the types of searches that PIE users perform, the range of search functionality used and the usability of the PIE's search interface.

The results of the questionnaire survey indicate the PIE isn't very easy to use by those who haven't received training. The design of the PIE's interface, its performance speed and competition with existing library services (such as the LBS' Forum) are important issues that require further investigation. The Phase Two evaluation will seek to ascertain which aspects of the PIE interface and PIE performance could be improved. The development of the PIE software has now been frozen. Therefore, these investigations will inform the development of any future hybrid library systems. The extent to which competition with existing library services influences the level of PIE usage by target groups at each institution will also be evaluated.

Create your PIE

All technology associated with, or used by, the HeadLine PIE has either been developed by the project or is currently freely available to academic institutions. HeadLine will make the PIE and all associated software available (at no charge) via the HeadLine website <http://www.headline.ac.uk> before the end of the project.

Guest access

The Headline Team is keen to get feedback on the PIE from the wider library community. Guest access to the PIE (subject to any copyright restrictions associated with the resources that it contains) is available. To register as a Guest user, you need to complete the web form: <http://www.headline.ac.uk/guest/regform.html>. You will then be emailed a username and password along with instructions for accessing each of the three PIEs. Guest users will be invited (but by no means obliged) to participate in the Phase Two evaluation.

Further information

For further information about the HeadLine Project, please either visit the HeadLine website: <http://www.headline.ac.uk> or email the HeadLine Team: <[email protected]>.

For further information concerning the development and launch of the HeadLine PIE and the design of the RDM, see the article by Anne Gambles, "The development and launch of the HeadLine Personal Information Environment," in Information Technology and Libraries, volume 19, number 4 (December 2000): pages 199-205.

Information about the HeadLine extension may be found at <http://www.headline.ac.uk/extension/>.

Copyright 2001 Anne Gambles

Top | Contents
Search | Author Index | Title Index | Back Issues
Previous Article | In Brief
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

DOI: 10.1045/march2001-gambles