Sarah Giersch - iLumina Digital Library
A two-day workshop on Participant Involvement in Digital Libraries (PIDL), sponsored by the National Science Foundation (NSF) and hosted by The Math Forum at Drexel University  was held in early February 2004. The workshop brought together 35 experienced practitioners and evaluators from projects funded by NSF's National Science Digital Library (NSDL) program  and from the area of online community development to describe and discuss the nature of participant involvement in NSDL  and NSDL-funded projects, as well as methods that provide feedback about participants to these projects.
The PIDL workshop had the twin goals of 1) exploring the practical issues of building participant involvement in digital libraries (DLs) as well as bringing together the research to date on participant involvement in NSDL and 2) focusing directions for future workshops. The goals of the PIDL workshop included sharing information about work already completed and outlining the challenges involved in establishing educational DLs  as active communities of practice. Workshop attendees developed strategies that will help NSDL projects, and DLs in general, incorporate involvement of participants from the communities the DLs serve; this "participant involvement" in turn will help grow and support the DLs. Workshop attendees formulated recommendations that anticipate involving participants both as users of the DL and as producers of new knowledge for the community. The full PIDL workshop report can be found on the workshop website .
The concise report presented here details the relevance of participant involvement to DLs; provides a summary of issues and action items identified during the workshop; and offers recommendations for the direction of future research on participant involvement and community-building. While the PIDL workshop focused specifically on NSF's NSDL program, the ideas developed in the workshop have broader application to the larger DL community.
The Case for Participant Involvement
Educational digital libraries are faced with several challenges, among them: to be widely adopted; to contribute meaningfully to educational improvements; and to thrive following start-up funding. They also have the potential to be more than simply a collection of resources. Educational DLs have the potential be the junction for active, engaged communities.
While there are several public and private sector models for building and maintaining DLs, a model that involves participants who have varying strengths, needs, experiences, and interests should make a significant difference in the design, development, and utility of educational DLs. Development of a model or models that describe participant involvement should enhance the likelihood that multiple audiences and constituencies can be effectively involved in all facets of educational DLs. How DLs choose to invest in involving participants is affected by:
One challenge in developing a model for participant involvement that applies across all DLs is that efforts to foster interaction often evolve organically and remain localized to a particular event (e.g., resource contribution or site design) or to an individual DL. One way to overcome this challenge is to support the development of communities of practice that bring together people with various proficiencies to effectively and efficiently solve problems and develop tools, benefiting all constituents. Further, such communities can mobilize expertise, train the next generation of experts, and meet the needs of other less skilled participants. Adopting a strategic, centrally organized, participant involvement model could result in the growth of active communities of practice (e.g., around NSDL).
Participant Involvement in Practice
Workshop discussion focused on types of activities that comprise "building" digital libraries. This section identifies types of participation and activities for involving participants as well as identifies associated benefits, challenges, and recommendations related to participant involvement.
Creating, contributing and describing content
Since its inception in FY2000, the NSDL program has had a funding track to create collections of digital resources. This support has resulted in several models of how DLs could grow their collections through participant interaction. Two common models have emerged. They are:
One challenge related to individuals creating and contributing resources is that DLs need to demonstrate to participants that their contributions are valued.
Often, the line between creating information about a resource and creating the new resource itself is blurred, as resources may be modified to meet one-time needs  or separate resources might be combined to create completely new content . While the technology supporting these models of contributing to DL collections or describing resources is still in the early developmental phase, one challenge will be that we need to remain focused on the primary goal of using technology to support participant involvement in creating, contributing, and describing content.
Involvement in human-moderated services
Human-moderated services rely primarily on direct human intervention to fulfill users' needs. In a distributed digital environment, many human-moderated services are delivered via technology, and at some point, may overlap with technology-mediated services. Human-moderated services can support interaction between DL staff and participants and can foster interaction between expert and novice participants of communities of practice.
Involvement in technology-mediated services
Technology-mediated services can include recommender systems , aggregation and harvesting services , federated search [14, 15] and hosting content [16, 17]. One point that human-moderated and technology-mediated services share is that technology should supplement, rather than substitute for, human interactions. While it is relatively easy to create technologies that support participant interaction, the challenge lies in supporting sustained interaction across complex social and cultural networks. Another ongoing challenge is that often technology drives the development process  without understanding the users' ultimate needs.
Sustaining and expanding involvement
"Sustaining"  a DL has often been equated with "maintaining" the DL. This could be a limiting idea because "sustaining" can also connote nourishment and growth. From that perspective, the DL becomes but one link within an ecosystemboth providing and requiring "sustenance". Participants who contribute to a DL's sustenance in turn receive nourishment. To be sustained and sustaining, DLs should develop business strategies or models that enable them to endure beyond research funding. The following recommendations for sustaining DLs emerged from the PIDL workshop:
Summary of Participant Involvement Issues
This section describes issues and challenges associated with involving participants in DLs. Highly developed DLs such as DLESE, ENC, Math Forum, MERLOT, and others demonstrate the importance of participant involvement in the growth and development of their sites. Following are operating principles shared by DLs that involve their "users" as participants:
However, as workshop attendees examined, word by word, the question: "If we build it, will they come?" attendees identified assumptions that limit effective action to involve participants (see Figure 1).
Involving Participants Requires an Investment
Ideally, participants should be involved from the earliest stages of building a DL. The short-term benefits of early and continuous involvement are that the infrastructure, content, and services meet the immediate strengths, needs, experiences, and interests of participants. Additionally, the DL can become responsible for sustaining its own growth at an earlier stage. In the long-term, continued involvement can create a sense of trust in participants, leading to long-term commitments. In order to achieve this continued involvement, DLs invest time, money and human capital. At the same time, they also need to acknowledge and reward participants' investment of time and effort (discussed in more detail below). Two key challenges include: being responsive to participants' needs while progressing steadily through the DL building process and scaling human-intensive development processes or services.
Involving Participants in Projects
Involving participants in NSDL projects may be optimal for participant growth; the question is how to make this experience optimal for projects. Working with participants takes time and does not necessarily yield immediate results that may become criteria for evaluating DL effectiveness (e.g., papers, site resources, etc.). Some issues that surfaced in discussions include the following:
Participants' time and effort need to be recognized and valued. This is more complicated than just providing rewards for participation, contribution, and so forth. DLs must first understand participants' strengths, needs, experiences, and motivating interests: (e.g., a participant might ask, "How will participating in the DL enhance my educational or professional life?"). If participants are involved in every phase of DL building, the DL is more likely to reflect and match the strengths, needs, experiences, and interests of participants. At the same time, DLs need to build in rewards for participating that are meaningful and valued. Experiences of workshop participants suggest that:
It is a Challenge to Support the Evolution of Participation
DLs operate at various stages of development. Similarly, participants engage DLs at various stages of individual growth and need. Over time participants will shift roles or will adopt additional roles as their needs evolve; technology will mature beyond the cycle of test and development; and communities will coalesce around areas of interest, practice or need. The challenge is to recognize that DLs evolve and that participation in DLs varies and requires appropriate human and technical scaffolds to accommodate these transitions. This evolution is influenced by 1) the activities and needs of current and potential participants, 2) the knowledge and skills that exist within the participant community, 3) the types of content and services offered by a DL, and 4) the stage of development, resources and services of the DL.
Participant and DL Collaboration
Collaborations can take many forms (e.g., distributed / centralized, formal / informal). Results of collaboration can vary based on the depth of involvement from DL staff and participants, with more intensive collaborations requiring a larger investment of time and resources but potentially yielding more enduring results. A number of successful efforts at localized collaboration among specific projects, audiences, disciplines or geographic areas were identified during the workshop.
Collaborations typically occur via both relatively simple technology tools and face-to-face. Two factors that motivate DL staff and participants to collaborate are 1) that they share common needs and 2) that they value meeting their needs together. Obvious benefits for DLs include (a) leaders working and worrying together about similar issues and (b) leveraging the experiences of other DLs and participants. On a larger scale, collaboration is beneficial for the diffusion of innovation . Collaboration also includes challenges:
DLs are an important new tool that can have a significant effect on people's behavior and opportunities to learn. So, it is important to discover who is using DLs, to understand the constraints that affect use and to identify what would help individuals change their existing patterns of interaction and allow these individuals to become regular participants.
In undertaking evaluation, DLs need to know who their participants are, what the social context is for their participation in DLs and what scaffolds different types of participants need to aid their learning. As these questions are answered, DL staff can identify evaluation goals for the DL and work with researchers and evaluators to identify whether or not significant social change is occurring. This level of exploratory research is critical for all DLs, and particularly for NSDL at this time, because such research will provide an understanding of how DLs fit into participants' lives and how creative participants are using the DL to further educational and learning goals.
As it is better understood who DL participants are and why they come to a particular DL, it becomes possible to do larger, targeted and detailed descriptive studies of participants' activities and of what those activities mean to participants. This work provides a detailed understanding of what the DL means to different participants. For NSDL, broadly speaking, this level of research is expected to come after substantial exploratory work is conducted in NSDL and other DLs and would require some coordination of research efforts across different DLs. For these reasons, descriptive work may not be part of the evaluation plans of a single DL but rather the coordinated effort of evaluators across several mature DLs.
It is hoped that after exploratory and descriptive evaluation there will be large-scale explanatory studies that follow an experimental model, which would allow for assessment of educational impact. These studies could adhere to the tradition of double-blind experimental research by establishing research hypotheses, testing hypotheses with a large representative sample and explaining the results. Explanatory research with reliable results is critical for NSDL in that it is necessary to know what impact new educational communities have on teacher and student learning and practice . While explanatory research should be a goal for NSDL, like descriptive research, explanatory research is often larger in scale and requires a high degree of coordination. It also requires a much greater knowledge than currently exists of what is happening in DLs. NSDL should develop plans to support coordinated explanatory studies to be undertaken by teams of researchers across different mature DLs. However, explanatory research is not likely to be part of the regular evaluation plan of individual DLs.
Because research on DL participants is in an early phase, evaluators often are not aware of what is known about social activity in DLs. In addition, research into the patterns of activity of participants in DLs and how they incorporate DLs into their daily lives is complexit involves investigating the new social worlds created by the overlap of virtual and physical spaces. Evaluators and researchers can take advantage of new information technologies, and by studying DLs, close this gap between spaces. It is suggested that the results of evaluation be shared across NSDL so that researchers can see what others have learned and consider its application to their own DL.
Workshop attendees concluded that in order to better inform themselves and others about involving participants in DLs, the following actions need to be taken:
This work was sponsored by The National Science Foundation under grant no. DUE-0226284. Opinions expressed in this piece are not necessarily endorsed by the National Science Foundation or Drexel University.
Notes and References
 In this report, digital library (DL) is an inclusive term, encompassing NSDL, NSDL-funded projects and other online sites of communities.
 User-contributed content is a goal of the Chemistry Collective, <http://www.chemcollective.org/>. It provides authoring tools for both virtual labs and scenario-based learning, and is starting the project with a base of six authors who have contributed roughly one third of the activities.
 The Gender Science Digital Library (GSDL), <http://www.gsdl.org/>, has spent time developing sample lesson plans out of the items in the collection, based on the focus group feedback of K-12 teachers. This has been the most used and most commented on section of the siteimmediately usable content with a clear context for use.
 Each ComPADRE, <http://www.compadre.org/>, collection has an Advisory Panel of recognized experts/practitioners in the field covered by the collection. Their job is to recommend authors and projects that can provide high quality content.
 At Unidata, the Internet Data Distribution (IDD), <http://my.unidata.ucar.edu/content/software/idd/index.html>, system enables hundreds of universities nationwide to receive vast quantities of real-time data (of many types in many forms) and to store custom subsets that reflect specific departmental and faculty interests. The stored data subsequently may be shared via conventional Web images or direct-access methods such as are being developed (in part with NSDL funding) under the THREDDS project, <http://my.unidata.ucar.edu/content/projects/THREDDS/index.html>).
 Instructional Architect, <http://ia.usu.edu/>, is a program designed to help participants use existing instructional resources from digital libraries in order to create engaging and interactive educational web pages.
 SMETE.ORG Recommender Service,
 SMETE.ORG Technology, <http://www.smete.org:80/smete/?path=/public/about_smete/activities/
 The search engine found at ENCdl.org allows single or multiple ENC and ENC collaborative collections to be searched. It is not quite a federated search, but it is a model for how multiple collections that have been described using a variety of metadata can be displayed via a common access point.
 MERLOT's Federated Search <http://fedsearch.merlot.org/main/search.jsp>, currently hosts federated search with SMETE.ORG and EdNA Online - Education Network Australia. This model allows users to search all three digital libraries, seamlessly.
 Hosting materials submitted by users is a vexing issue. The Michigan Teacher Network (MTN; <http://mtn.merit.edu/>) hosted the results of the Michigan Department of Education lesson plan contest. Teachers were paid $100 for each lesson plan submitted. The process did not involve a consistent template or any review process. Quality is essential when dealing with submitted material.
 The Learning Matrix, <http://thelearningmatrix.enc.org/>, has the ability to accept participant submitted materials. The participants are asked to use the ENC cataloging tool to complete metadata fields.
 This was not the case with ComPADRE: On the ComPADRE student site, <http://www.compadre.org/student/>, the national board of the Society for Physics Students (a group of both students and SPS chapter advisors) was consulted on the design and features for the site during construction. Working groups were selected to generate ideas about various pieces of the site.
 Two previous NSDL workshops have addressed sustainability topics: 1) Establishing Relationships between Educational Publishers and the NSDL (October 2002), <http://publishers.comm.nsdl.org/deliverables/>, McArthur, D., Giersch, S., Wittenberg, K. and M. Luby. NSDL Educational Publishers Workshop: Workshop Report) and 2) Exploring Business Options for NSDL identified potential business models to sustain NSDL and NSDL-funded projects (November 2003), <http://nsdlbizmodel.comm.nsdl.org/docs/>, McArthur, D. NSDL Report: A Review of Business Options for the National Science Digital Library)
 The MERLOT awards program, <http://taste.merlot.org/awards/>, is an annual award given out for 'best in discipline' (Classics Award) and 'best overall' (Editors' Choice Award)selected by editorial boards. Materials that rate highly in the peer review process are considered for the award. The Awards program receives funding from sponsors. Editors' Choice awardees receive a small stipend and travel to the MERLOT International Conference where the award is presented. Classics award winners receive free registration to the MERLOT International Conference. All Awardees who attend the conference showcase their materials, and they are highlighted on the MERLOT site.
 See work by: Malcom Gladwell, The Tipping Point; Hagel and Armstrong, NetGain; Geoffrey Moore, Crossing the Chasm; Sami Paavola, <http://newmedia.colorado.edu/cscl/228.html>. Also, <http://inquiry.uiuc.edu/cil/documents.php?cilid=99&folderid=227>, contains a list of papers about collaboration and diffusion of innovation.
 The Math Forum <http://www.mathforum.org>, has been studying the impact of site services on students' mathematical thinking and has developed rubrics for identifying problem difficulty that allow weighted comparison of student work on one problem with work on another problem. Study has also been conducted on mentor-participant exchanges in the Ask Dr. Math Service, <http://www.mathforum.org/dr.math/>. Studies of this type have been undertaken in response to staff members' questions, and findings are used to further develop site services. These studies serve a dual purpose in that they provide needed information to the site and provide summative studies of service impact. These studies follow earlier descriptive study of the same services.
 K.A. Renninger & W. Shumar (Eds.), Building virtual communities: Learning and change in cyberspace. New York, NY: Cambridge University Press.
 See, Renninger, K.A. & Shumar, W. (2002). Community building with and for teachers: The Math Forum as a resource for teacher professional development. In K.A. Renninger & W. Shumar (Eds.), Building virtual communities: Learning and change in cyberspace (pp 60-95). New York, NY: Cambridge University Press.
Copyright © 2004 Sarah Giersch, Eugene A. Klotz, Flora McMartin, Brandon Muramatsu, K. Ann Renninger, Wesley Shumar, and Stephen A. Weimar