www.arl.org > statistics & assessment > committee & staff

Association of Research Libraries (ARL®)

Statistics & Assessment

Committee & Staff

ARL Statistics & Measurement Committee Agenda, October 2001

From the 139th ARL Membership Meeting

ARL Committee on Statistics and Measurement
Wednesday, October 17, 2001
8:30 a.m. - 11:45 a.m.
Room: Burlington A
Jurys Washington Hotel
Washington, DC

Agenda

Note: The parenthetical times shown are estimates only to aid in moving the meeting along. If an issue warrants, we will take as much time as necessary.

1. Introductions and Overview. (5 minutes)

2. Approval of Minutes. (5 minutes)

Attachment 2: Minutes of the Meeting of the ARL Statistics and Measurement Committee, May 2001 [PDF]

Outcome: Approval of minutes.

3. CLIR/DLF Report by Denise Troll, Associate University Librarian for library information technology at Carnegie Mellon University Libraries and DLF Distinguished Fellow

According to the CLIR October 2000 press release, Denise Troll "is the third fellow that CLIR and the DLF have appointed since the program was established in May 2000. Ms. Troll is spearheading that part of the DLF's program which aims to identify and evaluate measures that are appropriate for assessing the use and effectiveness of digital library collections and services." Carla Stoffle and the ARL Statistics and Measurement Program have invited Denise Troll to discuss with the ARL Statistics and Measurement Committee members preliminary results and recommendations emerging from her recent investigation on how libraries are changing and the challenges these institutions are facing in relation to accountability, performance and assessment. Daniel Greenstein, Director of DLF, will attend the committee meeting on her behalf due to a last minute cancellation. (1.5 hours)

Attachment 3: Draft introduction to forthcoming CLIR/DLF publication by Denise Troll [PDF]

Outcome: Understanding and discussion of issues with the purpose of informing the recommendations coming out of the forthcoming publication.

4. ARL Program Review Survey, Spring 2001

What does the committee see in the findings of the member Survey that could contribute to the committee's agenda or the activities of the program? There is some willingness to invest more in this program area. Is the strategy of self-funding projects working adequately while serving both member libraries and ARL satisfactorily? (20 minutes)

Attachment 4a: ARL Program Review Survey, Spring 2001 (as mailed to Directors on September 14, 2001)
Attachment 4b: Summary tables and chart showing results broken down by type of library [PDF]
Attachment 4c: ARL Statistics and Measurement Program Activities Report 2001

Outcome: Feedback on the program review results and advising on 2002 programmatic activities

5. New Measures: LibQUAL+

A review of recent developments and issues by Fred Heath (Texas A&M), Kaylyn Hipps, Jonathan Sousa and Consuella Askew Waller (ARL) (15 minutes)

5a. LibQUAL+. There are more than 100 libraries participating in spring 2002; Memoranda of Understanding have been signed with OhioLINK and AAHSL for consortial implemtations of LibQUAL+. Jonathan Sousa has been hired as the Technical Applications Development Manager, and Consuella Askew Waller has been working as a Communications Coordinator full-time on LibQUAL+ activities.

5b. NSDL LibQUAL+. Effective 1 October, ARL received a three-year grant of $245,737 from the National Science Foundation for a project to apply the LibQUAL+ assessment process to the National Science Digital Library (NSDL). This project seeks to reground the LibQUAL+ protocol for the needs of the user community of Science, Math, Engineering and Technology Education Digital Library (NSDL) services, suggesting the development of the NSDL LibQUAL+ protocol. The NSDL LibQUAL+ project goals include: (a) define the dimensions of digital library service quality from the perspective of the users; (b) develop a tool for measuring user perceptions and expectations of digital library service quality across NSDL digital library contexts; (c) identify digital library 'best practices' that permit generalizations across operations and development platforms; (d) enhance student learning by managing effectively user perceptions and expectations of digital library services; (e) establish a digital library service quality assessment program as an integral part of the library service quality assessment program at ARL; (f) institutionalize continuous product and process evaluation efforts directed toward positive and timely management of outcomes. The co-principal investigators on the project are: Duane Webster, Executive Director, ARL; Fred Heath, Dean and Director, Texas A&M University Libraries; Colleen Cook, Executive Associate Dean, Texas A&M University Libraries; Yvonna Lincoln, Professor and Program Director of Higher Education, Department of Educational Administration, Texas A&M University; and Bruce Thompson, Professor and Distinguished Research Scholar, Department of Educational Psychology, Texas A&M University.

5c. UK LibQUAL+ pilot. At the international level, there is also interest in exploring the LibQUAL+ process. Libraries in the UK are exploring ways to implement a joint UK/US pilot project.

Outcome: Understanding and advise on current developments

6. New Measures: ARL Learning Outcomes Working Group

Reporting discussion results from the October 15 meeting of the Working Group on Learning Outcomes Assessment. (15 minutes)

This working group has been formulated in response to the recommendations coming out of the report by Ken Smith, Eller Distinguished Service Professor of Economics at the University of Arizona, and author of the ARL White Paper on Learning Outcomes http://www.arl.org/arldocs/stats/aboutstats/oct2001/HEOSmith.pdf.

Attachment 6: ARL Learning Outcomes Working Group Meeting Agenda [PDF]

Outcome: Understanding and advice on current developments

7. New Measures: E-Metrics

Reporting discussion results from the October 16 project participating libraries meeting and preparing for the plenary discussion on October 18 during the ARL membership meeting by Sherrie Schmidt (Arizona State) and Rush Miller (University of Pittsburgh). (30 minutes)

The ARL E-Metrics project is conducted in three phases covering the following:

Phase I: Knowledge Inventory of ARL Libraries and organizing an ARL Working Group on Database Vendor Statistics (May, 2000 - October, 2000) * completed - see project web page http://www.arl.org/stats/initiatives/emetrics/index.shtml

Phase II: Development of statistics and performance measures (Nov, 2000 - June, 2001) * completed - Phase II report distributed to all member libraries and available on the web at http://www.arl.org/bm~doc/e-metrics.pdf.zip

Phase III: Institutionalizing Statistics and Performance Measures and Developing a proposal for institutional outcomes assessment (July 2001-Dec 2001)
* Presentation at CNI Task Force Meeting, San Antonio, TX, December, 2001
* Meeting in conjunction with ALA Midwinter on Jan 19, 2002

Attachment 7a: "E-Metrics: Preliminary Findings from the ARL Project" a facilitated discussion to be held on October 18, 2001, by Rush Miller, Sherrie Schmidt, and Carla Stoffle
Attachment 7b: Preface to the Phase II report [PDF]

Outcome: Understanding and advise on current developments

8. Council of Australian University Librarians (CAUL)

CAUL and their work on statistics and performance measures by Diane Costello. (15 minutes)

For an overview of related work in Australia, see Guidelines for the application of best practice in Australian University Libraries: Intranational and International Benchmarks. By Anne Wilson, Leeanne Pitman, Isabella Trahn. Evaluations and Investigations Programme 00/11 Higher Education Division Department of Education, Training and Youth Affairs. © Commonwealth of Australia 2000 ISBN 0 642 44477 3, ISBN 0642 444757 (Online version) Hard copies can be obtained by calling the Department on (02) 6240 9280. Available at: http://www.detya.gov.au/archive/highered/eippubs/eip00_11/00_11.pdf

Outcome: Understanding of performance measures context in Australia.

INFORMATION ITEMS:

9. Related Workshops/Presentations/Event Calendar

(a) Measuring Service Quality Online Lyceum workshop (8 October-16 November 2001) Danuta Nitecki and Toni Olshen

(b) "Creating a Culture of Assessment," an ARL OLMS workshop, 12-13 November 2001, Washington, D.C. Amos Lakos (Waterloo) and Shelley Phipps (U. of Arizona)

(d) E-metrics presentation (CNI Fall Task Force meeting, December 2001, TX) Charles McClure, Sherrie Schmidt (Arizona State) and Rush Miller (U. of Pittsburgh)

(e) Beyond LibQUAL+ (CNI Fall Task Force meeting, December 2001, TX) Duane Webster, Julia Blixrud and Fred Heath (Texas A&M)

(f) ARL survey coordinators meeting (January 18 2002, New Orleans)

(g) E-Metrics project participants meeting (January 19, 2000, New Orleans)

(h) Using LibQUAL+ Results 2001 Participating Libraries Workshop, 20 January 2002, ALA Midwinter, New Orleans, LA.

(i) New Ways of Listening to Library Users: Tools for Measuring Service Quality, An ARL LibQUAL+ 2002 Participating Libraries Workshop, 21-22 January 2002, ALA Midwinter, New Orleans, LA.

(j) Electronic Publishing of Datasets workshop with Patrick Yott (Brown University) (to be scheduled)

(k) Library Cost Allocation Model workshop (to be scheduled) Robert Hayes

(l) Conducting Focus Groups in Libraries workshop/online lyceum (not scheduled) Seeking VPO

(m) Research Methods and Statistics workshop/online lyceum seeking VPO

(n) User Survey for Academic Libraries workshop/online lyceum seeking VPO

10. Other News

(a) Academic Libraries Advisory Committee/NCES. Academic Libraries NCES now offers interactive data analysis capabilities of the Academic Libraries 1998 data. See: http://www.nces.ed.gov/surveys/libraries/academicpeer/

(b) International Standard ISO/DIS 2789, Information and documentation -- International Library Statistics has been voted at a recent meeting in Paris. Fred Heath (Texas A&M) represented the U.S., NISO, and ARL at that meeting.

(c) ARL is expecting to receive an official request from ACRL to use the ARL annual statistical surveys in their annual data collection of academic libraries. Their Academic Library Statistics and Trends publication for 2000 is under production. We are also continuing to identify effective ways for cooperating more closely in this area.

(d) 4th Northumbria International Conference on Performance Measures in Libraries had a lot of interesting presentations in relation to performance measures and outcomes assessment in libraries (IFLA preconference, August 12-16, 2001, Pittsburgh, PA). See also the web site at http://www.arl.org/arldocs/stats/aboutstats/oct2001/north-program.pdf. ARL will be publishing the proceedings next year.

(e) Outsell Inc. and DLF will be conducting a national survey of about 3,000 undergraduate students, graduate students, and faculty in relation to their information seeking behavior and library use activities. The project is financially supported by The Andrew W. Mellon Foundation.

(f) New Measures: Technical Services Cost Study by Dilys Morris and five test sites. Dilys Morris as well as other representatives from the participating test sites (including Christian Boissonas) made a presentation of the pilot projects and the software at an ARL meeting at ALA in San Francisco. Product development and refinement scheduled to finish by the end of the year and made available in 2002 as DecisionBase. The University of California at Santa Barbara has now input their data and Olha Buchel (Ph.D student at U of Illinois in Library science but working at UCSB) has done some comparisons between Cornel, Iowa State, Vanderbilt and UCSB and lots of graph files on UCSB.