From the 140th ARL Membership Meeting
ARL Committee on Statistics and Measurement
Wednesday, May 22, 2002
8:30 a.m. - 11:30 a.m.
Room: Ocean II
Fairmont Miramar Hotel
Santa Monica, CA
Note: The parenthetical times shown are estimates only to aid in moving the meeting along. If an issue warrants, we will take as much time as necessary.
Welcoming new committee members: Joan Gotwals, Diane Perushek, Sandra Yee. Three members were renewed for 2002-2004: Eileen Hitchingham, Carolynne Presser, and Sherrie Schmidt (5 minutes)
Attachment 1: ARL Statistics and Measurement Program Plan 2002 [PDF]
Outcome: Understanding and supporting the Program Plan.
Outcome: Approval of minutes.
The ARL E-Metrics Project, an ARL New Measures Project, led by Sherrie Schmidt (Arizona State) and Rush Miller (Pittsburgh), completed an important piece of the scheduled work under contract with the Information Use Management and Policy Institute at Florida State University. The goals of the project were (a) to document the state of affairs regarding statistics and performance measures to describe electronic information services and resources; (b) to test a variety of data elements and conclude with some recommendations regarding data elements to collect and next steps, and (c) to provide a framework for measuring outcomes. All these goals have been accomplished and documented in detail in the reports generated by the principal investigators and are available at http://www.arl.org/arldocs/stats/aboutstats/may2002/contract00-01.pdf.
Project results were disseminated at the 4th Northumbria International Meeting last August, at the October ARL Membership Meeting, and more recently published in Global Issues in 21st Century Research Librarianship. A forthcoming article by McClure and Fraser will also appear in the portal journal. All project deliverables will also be available in print from the ARL Publications Distribution Center after the May membership meeting (see attachment for Table of Contents). A paper entitled "Toward a Framework of Library and Institutional Outcomes" is also available at http://www.arl.org/arldocs/stats/aboutstats/may2002/ARL.Emetrics.Outcomes.Paper.Final.Jan.8.02.pdf.
Clearinghouse for Library and Information Center Networked Statistics. The Florida State University, Information Institute is developing a National Clearninghouse for Library and Information Center Networked Statistics. This Clearinghouse builds on and extends the work completed for ARL by Charles McClure and his staff at the Institute. The proposed Clearinghouse intends to identify, access, maintain, and update statistics and related data that describe libraries use, users, services, and resources in the networked environment. The Clearinghouse will provide customized reports and analyses for sponsors and others willing to pay required fees. EBSCO Publishing, NCLIS, and NISO are current sponsors and additional sponsors are actively being sought http://www.ii.fsu.edu/Announce/clearinghouse.html
Brinley Franklin is also doing work in the area of electronic resources and will be available to discuss this and other research with the committee members. His proposal entitled "Patterns of Patron Use of Networked Electronic Services at Four Academic Health Sciences Libraries" will be presented at Northumbria Lite later this summer. His paper on "Academic Research Library Support of Sponsored Research in the United States" was presented at the 4th Northumbria and is published in the proceedings.
ARL plans to use the upcoming survey coordinators meeting on June 14 (Sheraton Atlanta Hotel Atlanta 4/5 Room) as the main meeting for dissemination of the project findings and launch the next phase of this new measures pilot project. The proposed data and statistics will be collected and further tested by interested ARL libraries as part of the continuing effort to further refine and develop these new measures. A call for participation will be issued after the ARL May Membership Meeting. The 24 libraries that have funded the project by contributing $10,000 each to support the research under contract with Florida State University's Information Use Management and Policy Institute can participate without additional financial obligation. New participating libraries will be asked to contribute $2,000 each to support this continuing R&D effort. Gordon Fretwell (University of Massachusetts) will be serving as consultant in this continuing effort.
ARL is also participating in COUNTER, an international multi-agency organizational effort in the area of electronic resources and vendor/publisher statistics tasked to develop an achievable and widely supported common code of practice for vendor-based online usage statistics. This effort aims to cross-fertilize the work done by the ARL E-Metrics project, the UK PALS Usage Statistics Working Group as well as other parallel efforts led by ICOLC, NISO, publishers and libraries. Sherrie Schmidt (Arizona State) represents ARL in this effort and attended a small group meeting in London in December and participated in subsequent conference calls. The ARL Executive Committee has approved ARL's participation in this effort. Dr. Peter Shepperd, a consultant with experience in the publishing industry, will help this community of publishers and libraries move forward in the coming months. He will visit experts to get face-to-face feedback; writing up papers and proposals for circulation, approval and comment, leading to the creation of the detailed code of practice; work with sponsors and with other organizations to co-ordinate a truly international and multi-agency approach; and work on getting industry buy-in and developing a framework for ensuring compliance.
Attachment 3a: Measures for Electronic Resources (E-Metrics): Table of Contents [PDF]
Attachment 3b: Recommended Statistics and Measures [PPT]
Attachment 3c: Project COUNTER [PDF]
Attachment 3d: "Academic Research Library Support of Sponsored Research in the United States" by Brinley Franklin (excerpt from 4th Northumbria Proceedings) [PDF]
Outcome: Update committee members on the status of the project and identify continuing interest from past participants and potential future participants
LibQUAL+ Currently in its second year of the FIPSE three-year funding cycle LibQUAL+ had 163 libraries that collected data on more than 70,000 library users across North America. Results will be presented to the participating libraries in an upcoming meeting in conjunction with ALA in Atlanta. Individual libraries will be presented with their individual library results in addition to four aggregate notebooks: (a) for all spring 2002 participating libraries, (b) ARL, (c) OhioLINK, and (d) AAHSL (Fred Heath) (15 minutes).
Outcome: Understanding current developments.
Paul Mosher will share with the committee the range of activities being accomplished with data collection and analysis both as they relate to the ARL E-Metrics and LibQUAL+ projects. (45 minutes)
Outcome: Discuss the possibilities of learning from and building upon the work done at the University of Pennsylvania.
The Learning Outcomes Project is attempting to identify measures that libraries can use to demonstrate their contribution to campus learning outcomes. The Working Group met on January 19 in New Orleans and identified the following actions for promoting the agenda: (a) identify assessment expertise on campuses, these can be people who are involved in the accreditation process at the institution level and/or departmental/professional level; (b) work with national campus-wide surveys to identify which ARL institutions have participated in these efforts; and, (c) work with the Standardized Assessment of Information Literacy Skills (SAILS) research team at Kent State University for further development of an instrument to measure information literacy skills
The Project for the Standardized Assessment of Information Literacy Skills (SAILS) is an initiative of Kent State University. Its purpose is to create an instrument to measure information literacy that is standardized, easily administered, has been proven valid and reliable, assesses at institutional level, can be administered at any type of institution, and provides for both external and internal benchmarking. A team of two librarians and a measurement expert has worked for more than two years to create this tool that has been tested and utilized at Kent State. (30 minutes)
Outcome: Understanding current developments.
Preliminary discussions have taken place among Joseph M. Moxley, Mary Case, Martha Kyrillidou, and others, about submitting to FIPSE a joint proposal on evaluating the National Digital Library on Theses and Dissertations. Julia Blixrud will be attending the ETD conference the week after the membership meeting and have the opportunity to discuss how can this proposal be modified to better reflect ARL's interests. Attached is a working draft of some of the preliminary ideas that were discussed. (10 minutes)
Attachment 7: Draft Proposal tentatively entitled 'National Graduate Development Research Initiative' by Joseph Moxley (and others)
A grant proposal has been submitted by ARL to IMLS (DeEtta Jones and Martha Kyrillidou) for a three year grant to establish a full-year curriculum on collaborative assessment strategies for institutional teams. ARL will probably be notified later in August/September if the application is successful. (10 minutes)
Outcome: Understanding goals and planned activities.
Update, replicate, and expand the 1997 ARL ILL/DD Performance Measures Study to obtain current data on the performance of mediated and unmediated interlibrary loan/document delivery operations in research libraries. This project will be undertaken as part of ARL's New Measures Initiative. (10 minutes)
Outcome: Explore interest for the proposed project
A. Allocation of Staff Resources. The need to explore the applicability of comprehensive cost accounting frameworks is being explored with a model originally developed and currently refined by Eileen Hitchingham (Virginia Tech). Libraries interested in this project include: U. of Miami, U. of Manitoba and Notre Dame. Eileen Hitchingham will work with these interested libraries on the project in the latter part of 2002.
B. Measures to identify the library contribution to research outcomes. Visiting Program Officer Doug Jones made a presentation of his latest finding at the Survey Coordinators meeting last January. He plans to have a paper draft by the end of July.
Attachment 10B: Powerpoint presentation from January meeting [PPT]
C. Impact of assessment on library management decision making. Susan J. Beck, Head of Public Services in the Rutgers University Paul Robeson Library, is serving as Visiting Program Officer investigating the impact of assessment on library management decision-making processes and the degree to which assessment data has influenced change.
D. Service Quality Evaluation Academy. ARL is sponsoring a Service Quality Evaluation Academy, an intensive five-day workshop emphasizing basic concepts and skills in measurement and data analysis that are applicable to service quality evaluations (27-31 May 2002). The Academy curriculum focuses on qualitative and quantitative methods for collecting and analyzing library service quality data. Time will be devoted to relevant software skills, including the use of ATLAS.ti to analyze the content of interviews or responses to open-ended survey questions and the use of SPSS for quantitative data analysis. The participants are:
Bonnie Allen, Oregon State University
Susan Beck Rutgers, University
Robert Bellanti, University of California, Los Angeles
Francine DeFranco, University of Connecticut
Irma Dillon, University of Maryland
Frank Elliott, University of Minnesota
Sue Gervais, Brown University
Doris Hendrickson, Radford University
Steve Hiller, University of Washington
Sarah McCord, Washington State University
Kurt Murphy, Arizona State University
Laurie Probst, Pennsylvania State University
Brian Roberts, Brigham Young University
Steve Shorb, University of Florida
Judy Siebert Maseles, University of Missouri-Columbia
Janice Simpson, University of Alabama
Joan Stein, Carnegie Mellon University
Linda White, University of Virginia
Carla Stoffle, ARL Statistics and Measurement Committee Chair and Dean of Libraries at the University of Arizona, Eileen Hitchingham, Dean of University Libraries at Virginia Tech, and Sherrie Schmidt, Dean of University Libraries at Arizona State University, served as the Academy Selection Committee.
Outcome: Understanding goals of the Academy
Excerpt from Working Paper on CRL Strategic Objective #3:
"The legacy economics of research libraries' maintaining homogeneous and redundant print holdings are becoming increasingly unsustainable. Given CRL's members' stake in the future of print CRL can and should provide a cooperative solution to this problem. A cooperative approach to preservation of print materials will have to surmount many obstacles including lack of infrastructure, cost and traditional notions of ownership."
Outcome: Understanding of issues and developing recommendations
A.Academic Libraries Advisory Committee/NCES. Proposed revision of the ALS/NCES survey will make the national surveys more similar. Advisory committee is meeting in conjunction with ALA meeting and chaired by Mary Jo Lynch, Director of the ALA Officer of Research.
B. ACRL Library Statistics data collection for 2000-2001 is currently underway. We are working more closely with ACRL to eliminate unnecessary duplication of effort by exchanging data with institutional permission. We are also working on formalizing our relation with ACRL so that any revisions to the ARL Statistics survey are filtered into the ACRL version of the instrument. Last, there are many revisions in the ARL Statistics interpretation of the definitions that need to be communicated in a uniform way to the larger post-secondary education community and need to explore ways for doing this.
C. NISO The Library Statistics Standard, ANSI/NISO Z39.7-1995, was first released in 1968, and revised in 1983 and 1995, and is currently under revision again. With each revision the Standard has grown and changed. When the 1995 edition of the Library Statistics Standard was released, the committee that developed it acknowledged that the standard did not address two important emerging areas: measurement of electronic resources and performance measures. The current revision is trying to address these issues. Sherrie Schmidt is serving on the revision committee.
D. ISO Performance Measures and Statistics Standards Revision Process. Fred Heath, Denise Davis and John Carol Bertot are the US representatives to the ISO related committees.
E. DLF/Outsell Inc Study. With support from the Andrew W. Mellon Foundation, the Digital Library Federation and Outsell, Inc. are engaged in a multi-institution study that compares the information-seeking behaviors of more than 3,200 students and faculty across academic disciplines in liberal arts colleges and research/doctoral universities. Preliminary results were presented at the Spring CNI Task Force meeting. A copy of the project proposal is available at: http://www.diglib.org/use/grantpub.pdf . Leigh Watson Healy identified self-sufficiency as an important aspect of user behavior in her CNI presentation confirming findings from the LibQUAL+ study.
F. Known to this group as the Technical Services Cost Study by Dilys Morris is moving toward final product release. The software product is a cost accounting software and can be used for other service areas in addition to technical services. TCA DecisionBase, the current branding of this product, is currently being tested and the product will be released officially by Chronus Group Inc. for $2,000 per institution later in the year. ARL will receive a copy of the software and the documentation once the product is officially released.
G. International interest by the UKSG in doing an international survey, possibly working through IFLA, on users and their use of e-resources as suggested by Kristiina Hormia-Poutanen, Head of National Electronic Library Services, Helsinki University Library, the National Library of Finland. See their survey suited to Finnish environment at http://www.lib.helsinki.fi/finelib/kysely01_eng.html. Interest is in doing a short (6-8 core question survey on - who the user is (questions in group 1);- where, how often they are using e-resources (4.2, 4.1); - why (4.3); - attitudes to cancelling print (2.2); - training needs - problems.
H. ARL's Policy for Protecting Human Subjects. With the development of services like LibQUAL+ where ARL collects data directly from users the need to establish a policy for protecting Human Subjects was becoming increasingly important. See http://www.arl.org/arldocs/stats/aboutstats/may2002/privacy.pdf.
Information Items:
(a) LibQUAL+ presentation at the SUNY Council of Library Directors by Duane Webster (April 10-12)
(b) "Service Quality Evaluation Academy" an Intensive Five-Day Program in Qualitative & Quantitative Methods, 27-31 May 2002 San Antonio, Texas.
(c) Measuring Service Quality Online Lyceum workshop developed by Danuta Nitecki and Toni Olshen to be offered from 24 June - 2 August and 7 October - 15 November.
(d) LibQUAL+ presentation by Julia Blixrud (June 2-6, Kansas City, MO, IATUL (International Association of Technological University Libraries)
(e) "LibQUAL+TM Process Management" by Consuella Askew Waller, Jonathan Sousa and Amy Hoseth, an ARL workshop, 14 June 2002, Atlanta, GA.
(f) NCES/ALS Advisory Committee (June 13-14, Atlanta
(g) New Measures Learning Outcomes Working Group (June 14, Atlanta)
(h) ARL survey coordinators and SPEC Liaisons meeting to discuss E-Metrics issues and next steps (June 14, Atlanta)
(i) "LibQUAL+TM Spring 2002 Participants Meeting", 15 June 2002, Atlanta, GA.
(j) "Measuring the New Measures" presentations by Carla Stoffle, Joe Zucca, Jim Self, Julia Blixrud, etc. (part of ALCTS/ALA Program, 17 June, Monday afternoon)
(k) LibQUAL+ presentation by Colleen Cook, Yvonna Lincoln and Bruce Thompson at the American Association of Higher Education (June 20-22)
(l) Web Development with XML: Design and Applications, a workshop by Matthew Gibson, Christine Ruotolo and Patrick Yott (June 24-28, Charlottesville, VA)
(m) LibQUAL+ presentation by Martha Kyrillidou to the U. Mass Director's Group (June 25)
(n) "Creating a Culture of Assessment" an ARL OLMS workshop 18-19 July 2002, Houston, TX
(o) "User Based Service Quality Assessment Across Multiple Institutions: LibQUAL+ (TM)" panel proposal submitted to the NCES Forum and Summer Data Conference (July 22-26)
(p) "IFLA Satellite Pre-conference, poster session for statistics in practice - "Measuring and Managing" by Julia Blixrud (August 13-14, Loughborough, UK)
(q) "Northumbria Lite" will have presentation by Brinley Franklin, Julia Blixrud, Colleen Cook, Duane Webster, Steve Hiller, etc. (August 22)
(r) "OhioLINK LibQUAL+ meeting" (August 29)
(s) Conference on Assessment Related issues cosponsored by SLA, ARL and FEDLINK in Monterey, CA (September 13-18, 2002)
(t) "New Ways of Listening to Library Users: Tools for Measuring Service Quality," an ARL workshop, 27-28 September 2002, Washington, D.C.
(u) LibQUAL+ presentation submission at the HLA (Hawaii Library Association) 2002 Annual Conference (27-28 September, 2002)
(v) LibQUAL+ presentation by Fred Heath and Duane Webster, EDUCAUSE 2002"Developing Tools to Assess Digital Libraries"
(w) "Service Quality Assessment" presentation by Julia Blixrud at the Swedish Library Association Conference (October 25, Stockholm, Sweden)
(x) AAHSL LibQUAL+ meeting (November 12, San Fransisco)
(y) NSF/NSDL All Projects Meeting by Fred Heath and Duane Webster (NSF/NSDL All Projects Meeting)
(z) FIPSE Project Directors Meeting by Fred Heath and Duane Webster (Fall 2002)