Search   |   Back Issues   |   Author Index   |   Title Index   |   Contents



D-Lib Magazine
June 2003

Volume 9 Number 6

ISSN 1082-9873

Google Meets eBay

What Academic Librarians Can Learn from Alternative Information Providers


Anne R. Kenney, <>
Nancy Y. McGovern, <>
Ida T. Martinez, <>
Lance J. Heidig, <>
Cornell University

Red Line



In April 2002, the dominant Internet search engine, Google™, introduced a beta version of its expert service, Google Answers, with little fanfare. Almost immediately the buzz within the information community focused on implications for reference librarians [1]. Google had already been lauded as the cheaper and faster alternative for finding information, and declining reference statistics and Online Public Access Catalog (OPAC) use in academic libraries had been attributed in part to its popularity. One estimate suggests that the Google search engine handles more questions in a day and a half than all the libraries in the country provide in a year [2]. Indeed, Craig Silverstein, Google's Director of Technology, indicated that the raison d'être for the search engine was to "seem as smart as a reference librarian," even as he acknowledged that this goal was "hundreds of years away" [3]. Bill Arms had reached a similar conclusion regarding the more nuanced reference functions in a thought-provoking article in this journal on automating digital libraries [4]. But with the launch of Google Answers, the power of "brute force computing" and simple algorithms could be combined with human intelligence to represent a market-driven alternative to library reference services.

Google Answers is part of a much larger trend to provide networked reference assistance. Expert services have sprung up in both the commercial and non-profit sector. Libraries too have responded to the Web, providing a suite of services through the virtual reference desk (VRD) movement, from email reference to chat reference to collaborative services that span the globe [5]. As the Internet's content continues to grow and deepen — encompassing over 40 million web sites — it has been met by a groundswell of services to find and filter information. These services include an extensive range from free to fee-based, cost-recovery to for-profit, and library providers to other information providers — both new and traditional. As academic libraries look towards the future in a dynamic and competitive information landscape, what implications do these services have for their programs, and what can be learned from them to improve library offerings?

This paper presents the results of a modest study conducted by Cornell University Library (CUL) to compare and contrast its digital reference services with those of Google Answers. The study provided an opportunity for librarians to shift their focus from fearing the impact of Google, as usurper of the library's role and diluter of the academic experience, to gaining insights into how Google's approach to service development and delivery has made it so attractive.

What is Google Answers?

Google Answers is an expert answering service that is staffed by over 800 freelance researchers who have been vetted by Google. For fifty cents, users can pose questions on Google Answers, and like Ebay®, they determine what the product is worth to them. They set the amount of money they are willing to pay for the correct answer (minimum of $2) and the amount of time they will wait for a reply. Any registered user may offer comments on the question but only Google-affiliated researchers may provide an answer. When a researcher indicates an interest in responding, the question is "locked in" for an hour. If the response is satisfactory, the user pays Google the agreed-upon fee — 75% of which goes to the researcher and 25% of which the company pockets. If the user is not satisfied with the response, s/he can request clarifications or reject the response and the question goes back into the queue for other researchers. The user can request a full refund (minus the fifty cent registration fee) if ultimately not satisfied with the results. Google Answers also allows users to rate responses. Researchers who receive too many negative reviews may have their privileges revoked by Google. The service provides an FAQ, a searchable and browseable database of questions, commentaries, and answers to previously asked questions; tips for users; a training manual for researchers; and terms of service.

The Cornell Taste Test

Of all the expert services available on the Web, Google Answers represents a good subject for comparative review with academic VRD services for many reasons. The search engine enjoys a large and growing market share of users, including those who rely on academic libraries. The service is well-documented and the question database contains many research questions and responses that parallel those answered by reference librarians. The information available about Google Answers supports a comparison of costs and response attributes and provides the means for assessing quality based upon external ratings. Google Answers is not specific to a domain, which makes it broadly relevant as a counterpoint for considering virtual library reference services. Finally, use of Google Answers has increased steadily over the past year, indicating a certain staying power for this type of service [6].

Like many research libraries, Cornell University Library has been experimenting with digital reference for a number of years. The offerings include:

A review of the Google Answers service indicated that it parallels more closely email reference than chat reference. Like email reference, Google Answers may involve a sequence of exchanges, but it is still asynchronous. Chat reference is an interactive session during which a reference librarian may respond to questions, but the software also enables reference staff to demonstrate the use of library resources to patrons. Our study was designed, therefore, to compare Google Answers to the email reference service used at Cornell. The study consisted of three stages: posting a set of questions to both Google Answers researchers and to Cornell reference staff; conducting a blind review of each pair of responses by reference staff from across the CUL system; and evaluating the results. In comparing the nature of the responses, we were interested in posing and answering the following set of questions.

  • Is there a formula or a consistent format to the responses?
  • Are there appreciable differences in approach between academic reference staff and freelance researchers? Based on their respective missions, it seemed logical that reference librarians would be more inclined to instruct users on how to do research and Google Answers researchers would be more inclined to deliver the answer, but what else would the comparison suggest?
  • Are there discernible patterns to the evaluations between the two groups?
  • What cost and time comparisons are possible, given the data available?
  • What lessons from this study can help reference librarians do their jobs better?

Study Methodology

We selected a set of twenty-four questions for the study, representing the range of research questions, from simple to complex, typically encountered by reference librarians [7]. The study did not include directional or how-to questions. The questions came from three different sources. The first batch (Maryland set) consisted of the twelve questions used by Neal Kaske and Julie Arnold in their study of online chat services, which they kindly agreed to allow us to use [8]. This test set had already been assessed as study-worthy and had been drawn from real questions posed at a variety of reference desks. All of the questions could be answered readily using basic reference materials, including web-accessible resources, but each required some effort to answer correctly and completely. The second set (Cornell set) comprised six questions that Cornell library users had asked over the past six months and that were deemed by the Cornell reference staff to be representative of questions they routinely receive. The third set (Google Answers set) consisted of six comparable questions selected from the list of unanswered questions on Google Answers.

At Cornell, a senior reference librarian, a beginning reference librarian, and a reference assistant each prepared answers for their allotment of the twenty-four questions and noted the amount of time spent on each question. Each librarian was asked to prepare their responses as if they were answering an ordinary email reference request. The Cornell reference librarians also assessed the degree of difficulty for each question based upon an informal rating of question complexity and anticipated time required to respond. They used a simple system ranging from one as the easiest to four as the most difficult. Table 1 lists the distribution of questions by difficulty rating, and Table 2 shows the difficulty ratings by source of the question.

Table 1. Distribution of Questions by Difficulty Rating
Difficulty rating Number of questions
1 5
2 8
3 10
4 1


Table 2. Difficulty Rating by Source
  Difficulty Rating
Source 1 2 3-4
1. Maryland (12) 4 7 1
2. Cornell (6)     6
3. Google Answers (6) 1 1 4

To gather the Google Answers responses to the study questions, one of the reference librarians created email accounts using multiple free email service providers. She registered with Google Answers as a user, submitted the questions, responded to requests for clarifications, collected the responses, and completed the loop by rating the Google Answers researchers' answers. The answers to the questions that were posted to Google Answers were priced as shown in Table 3.

Table 3. Pricing of Questions Posted to Google Answers
Difficulty Rating Price
1   $2.00
3-4  $10.00

Google researchers declined to bid on two of the more difficult questions at the $10.00 fee. Both questions had been assigned a difficulty rating of three and were among the questions drawn from those received by Cornell reference staff. These two questions were ultimately resubmitted to Google Answers with an increase in price to $50.00 and $100.00 respectively. At these rates, both questions were then answered.

Once the final set of 24 questions and completed responses from CUL and Google Answers were in hand, reference librarians across the CUL system blind-reviewed the responses. At least two reference librarians evaluated both sets of responses to each question, and no one librarian evaluated all of the questions. For several questions, we received three or four evaluations. Most evaluators reviewed two questions, but several reviewed as many as five questions. The evaluators varied in terms of levels of reference experience and academic ranking within the CUL system. They were not given extensive instructions, but were asked to evaluate and rate the responses using this scale: 1 = Poor, 2 = Fair, 3 = Good, 4 = Very Good, and 5 = Excellent. They were also aware that they were evaluating responses by a reference librarian and a Google Answers researcher, but they did not know which response came from which group.

Study Findings

This study was more exploratory than scientific in nature. It was intended to provide some insights into the value and utility of alternative information service providers in order to enhance the academic reference function and to identify areas for further study. The evaluations, therefore, reflect a research library bias as to what constitutes a good response. The study did not focus on or measure user satisfaction, one of the three ways quality service is generally defined — the other two being accuracy and utility [9]. For the purposes of the study, we assumed that users were satisfied with the responses they received. The evaluations implicitly reflect, then, reference librarians' notions of a quality answer, which includes accuracy and utility, but do not necessarily correspond to user satisfaction. For instance, we know that users who rated answers taken from the Google Answers database, assigned each a five-star rating (the maximum possible number). Cornell librarians, on the other hand, were more critical of the freelance specialists' responses, rating them on average at 2.7 out of a possible 5.00.

In addition to using the simple rating scale, many reviewers provided comments that offer valuable insights into their evaluation process.

Positive ratings were given to responses that:

  • were presented using clear and simple language
  • made a strong correlation to the question
  • indicated that the answer was possible or probable, not definitive
  • were thorough and relevant but did not include extraneous information
  • demonstrated a good search strategy and process
  • linked authoritative and reliable sources to the answer provided
  • included clear and complete citations
  • provided links that did not require downloading software to view resources

Negative comments noted that responses:

  • referred the user to relevant sources, but did not provide the answer
  • left the asker with a list of to-do's
  • lacked specificity or insufficient context for the response
  • failed to provide instructions on how to proceed
  • offered too much or too little information
  • included unnecessary commentary
  • were incomplete or inaccurate
  • provided links that did not result in a clear referral
  • were written in a manner that was hard to follow, grammatically incorrect, or overly complicated
  • did not exhibit a creative or comprehensive search strategy

In addition, the participating reference staff and several reviewers commented that the responses might well have been improved if a reference interview had been possible. Some noted that many responses were too Web-based, thus failing to provide the best references. They also took exception to Google Answers researchers who did cite resources beyond those freely available on the Internet but often conveyed a somewhat negative connotation on the necessity of having to visit a library to access the material.

Evaluation scores

The evaluations indicate that there was no clear "winner" in terms of responses to the questions posed. Cornell reference librarians fared better in the overall average and were rated slightly higher in answering the Cornell reference questions (study set 2) and substantially higher for the Google Answers questions (study set 3). Google Answers researchers led Cornell staff by a significant margin on the Maryland questions (study set 1), which represent half of the question set used in this study. The results of the evaluations based upon question source are presented in Figure 1.

Bar chart comparison

Figure 1. Average Rating by Question Source

Figure 2 presents the distribution of evaluation scores by level of difficulty. Cornell leads by a small margin in both the level 1 and level 3 questions, but is well below Google Answers for level 2 questions. The Maryland set of questions contained the majority of level 2 questions.

Bar chart comparison

Figure 2. Average Score by Difficulty of Answer

It is interesting to note that the reference librarians, whose strive to instruct users as well as provide answers, scored higher on level one questions, which tend to call for more straightforward factual answers, and level 3 questions, which require more in-depth research. Google Answers researchers do not experience such tension; their sole aim is to answer the question satisfactorily. This desire in some cases, the reviewers' noted, may have led some Google Answers researchers to offer responses based on opinions rather than facts. Another possible explanation for the difference in scores may be financial in nature. Google Answers researchers are paid by the response; reference librarians are paid by the hour. Perhaps the Google Answers researchers spent less time answering the simplest questions because they could earn only 75% of the average price paid for Level 1 questions ($2.20 in our study). The most difficult questions (level 3-4) paid the most (averaging $20.36 in our study) but required a lot more effort to prepare a successful response. Level 2 questions, netting on average $4.88, may have struck a balance between effort expended and reward earned. The number of questions in our study, however, was too small to draw any firm conclusions.

Another interesting finding from the evaluations is that the ratings for Google Answers responses were much more consistent from one reviewer to the next than were the evaluations for Cornell responses. For instance, on one question, the two reviewers rated the Google Answers response at 3 and 4 respectively, a mean difference of 1. The Cornell response was rated 2 and 4, for a mean difference of 2. The overall mean difference in scores between reviewers for the same question was 1.3 for Cornell, but only 1.0 for Google Answers. The standard deviation for both was 0.8, indicating that the shape of the distribution of ratings was the same for both. However, the Google Answers ratings curve was shifted to a lower value (e.g., more responses rated the same by reviewers). Figure 3 presents a histogram of the differences in ratings between the responses from Cornell and Google Answers. Although freelance researchers may not offer better answers than Cornell librarians, their responses were considered more consistent.

Bar chart comparison

Figure 3: Histogram of Differences in Ratings

A final point on the evaluations involves sources. Google researchers are experts at locating hard-to-find information on the Web. Their answers, therefore, tend to be limited to freely available networked resources. Reference librarians, on the other hand, are experts at locating and evaluating information in print and digital forms, and have access to a vast array of credible resources they physically own or license. Although reviewers complimented the reference librarians' use of non-Web materials, the rating scores themselves do not reveal any significant advantage being gained through their use. Again the number of questions is too small to draw any real conclusions, but the findings suggest that the quality of response based on resources used should be further studied.


Cost proved difficult to measure. Google Answers currently allows users to set a price for answers between $2.00 – $400.00, only 75% of which goes to the successful researcher. Early versions of the tips for users on the Google Answers site suggested using an hourly rate of $25.00 for factoring appropriate prices for questions based on anticipated time to complete the answer. Although it may seem logical that Google Answers researchers would be less likely to take on questions that would require more work than they would be compensated for, there is no explicit measure to evaluate. Hobbyists who serve as Google Answers researchers may well take on a question for the challenge.

Research libraries, on the other hand, have overhead costs, licensing fees, salaries, benefits, and other expenses that would need to be included in realistic pricing. We looked at some relevant pricing schemes using Cornell examples. Hostline charges $135.00 per hour, including the delivery of citations, with a 10% discount for alumni of the Cornell Hotel School. Cornell's Ask-a-Librarian reference service for alumni and friends costs between $75 to $135 per hour, depending on the subject. Like a number of expert services and unlike Google Answers, the librarian provides an estimate for the user before undertaking the work.

For the purpose of this study, we calculated costs two ways. For the Google Answers responses, we used the price set for each answer. For the Cornell cost, we divided the preparation time for each answer by the average salary plus benefits of the three participating reference librarians, which came to $42.24 per hour. Table 4 correlates level of question difficulty with time spent by reference librarians preparing a response. The average time spent on the easiest questions was 15 minutes, which equates to $10.56 in staff salary and benefits. The cost to respond to Level 2 questions averaged $16.05; Level 3 questions averaged $42.24.

Table 4. Cornell Time Spent on Responses
Level of Difficulty
(number of questions)
Cornell Preparation Time
range (average) in hours
1 (5)   .17 - .33 (.25)  
2 (8)   .17 - 1.25 (.38)  
3-4 (11)   .33 - 2.75 (1.00)  

Figure 4 compares the costs for Cornell and Google Answers based on the two methods of calculation presented above. Although the figures do not represent exact or comparable measures, they are nonetheless suggestive.

Bar chart comparison

Figure 4. Average Price by Difficulty of Question

Figure 5 charts cost against average rating per response. This chart illustrates again that Google Answers prices were much lower than Cornell's, but for neither group is there a strong correlation between price and quality of response.

Graph showing cost and rating for Google Answers and Cornell Reference

Figure 5. Cost and Rating Analysis


This study offered a quick, limited review of an emerging phenomenon — market-driven external reference services — that will ultimately affect the role played by reference librarians in academic settings. We see three immediate lessons learned from this study.

First, the study revealed the importance of self-assessment. Although Cornell reference librarians scored higher overall than did the freelance researchers working for Google, their scores were not significantly better. Both groups were rated overall in the "good" category, but one might have expected that highly trained and comparatively expensive information professionals would have scored consistently higher. There are many plausible explanations for why the librarians did not score higher than the freelance researchers, some attributable to the flawed or limited nature of our study. In addition, studies evaluating reference service present a mixed picture. Peter Hernon and Charles McClure, among others, estimate that on average librarians provide correct answers 55% of the time as judged by other information experts [10]. But in a recent study, John Richardson suggests that librarians provide an accurate source or strategy in response to users' questions approximately 90% of the time [11]. And users themselves consistently rate their reference encounters as highly satisfactory [12]. The lesson to be taken from our study is not the relative rating achieved but the importance of ongoing review as part of a strategy of self-assessment. A recent SPEC Kit on reference assessment noted that only 3% of the 77 responding libraries to a 2002 ARL survey regularly assess the quality of transactions [13]. Just as public school teachers evaluate each other's performance throughout the school year, reference librarians could improve their services through peer review. In addition, Google Answers' practice of encouraging users to rate and publicly post evaluations of responses received should be considered. A similar precedent already occurs in the academy as student ratings of professors' classes are posted with the course description at some institutions.

Second, academic libraries should make a practice of regularly monitoring developments in the broader information landscape. These developments will not only have a pronounced impact on information provision but they can also help reference librarians assess their own programs. Although it is still early days, we can envision a point in the future where some forms of reference service will be outsourced in a manner similar to the outsourcing of other library functions, such as copy cataloging. If for instance, an outside provider can adequately address simple reference questions at one-fifth the cost of doing so in-house, why duplicate the service? Reference librarians need to analyze more thoroughly how much time is spent by function performed. By freeing themselves from more routine tasks, they can focus their efforts on aspects of complex information discovery and use in which they clearly excel.

Finally, what lessons can academic libraries draw from the ancillary services offered by commercial enterprises? It has already been noted that is used by many in lieu of public access catalogs and that their methods of recommending like materials, providing sample page images for review, and posting reader reviews encourages customers to purchase materials. Can academic libraries entice greater use of intellectually vetted material by adopting similar practices? What can we learn from the practice of customers assigning fair market value for products and services that is at the heart of or Google Answers? Are there other ways to quantify the value of information services to users beyond pricing? Answers to such questions will help research libraries justify their value to academic administrators who are responding to the current economic crisis by casting about for programs that can be cut or eliminated. Academic librarians must become more savvy in articulating their value to the educational enterprise in order to prosper in a rapidly changing information environment. Commercial enterprises determine their services in part by assessing their competitors and going one better.


Special thanks go to those Cornell staff members who participated in this study, in particular Lynn Thitchener, and to Dr. John Kenney (Department of Physics, East Carolina University) for his assistance in data analysis.

Notes and References

[1] For an overview of reactions to Google Answers, see <>.

[2] Barbara Quint, "Some Advice for Google Answers," Information Today, June 2002, Vol. 19, Issue 6, pg. 8.

[3] "Google: Reference Librarians can Keep Their Jobs," Library Journal Academic News Wire, May 13, 2003.

[4] William Y. Arms, "Automated Digital Libraries: How Effectively Can Computers be Used for the Skilled Tasks of Professional Librarianship?" D-Lib Magazine, July/August 2000, Vol. 6, No. 7/8, <doi:10.1045/july2000-arms>.

[5] For a discussion of the various options, see Joseph Janes, Chrystie Hill, and Alex Rolfe, "Ask-an-Expert Services Analysis," Journal of the American Society for Information Science and Technnology, 52(13):1106-1121, 2001.

[6] For information on the growth of this service, see <>.

[7] For a listing of the questions, cost, time spent, ratings, and evaluation comments, see <>.

[8] Neal Kaske and Julie Arnold's study: "An Unobtrusive Evaluation of Online Real Time Library Reference Services," paper presented at the Library Research Round Table, American Library Association Annual Conference, Atlanta, GA, June 15, 2002 <>.

[9] John V. Richardson, Jr., "Reference Is Better Than We Thought," Library Journal, April 15, 2002, 127 (7), 41-42.

[10] Peter Hernon and Charles McClure, "Unobtrusive Reference Testing," Library Journal, 111 (April 15, 1986) p. 37-41.

[11] Richardson, Op. cit., p. 42.

[12] Results from a fall 2002 survey of chat reference users at Cornell indicated that over 80% of them received a full or nearly full answer to the questions they posed; 94% indicated they would use this service again.

[13] Association of Research Libraries, Reference Service Statistics & Assessment, SPEC Kit 268, September 2002, <>.

(On June 17, 2003, the name of Google's director of technology was corrected to read Craig Silverstein.)

Copyright © Anne R. Kenney, Nancy Y. McGovern, Ida T. Martinez, and Lance J. Heidig

Top | Contents
Search | Author Index | Title Index | Back Issues
Previous Article | Next Article
Home | E-mail the Editor


D-Lib Magazine Access Terms and Conditions

DOI: 10.1045/june2003-kenney