The E-Metrics project, one of the ARL New Measures Initiatives, is an effort to explore the feasibility of defining and collecting data on the use and value of electronic resources. ARL has some experience in tracking expenditures for electronic resources through the ARL Supplementary Statistics, but there is a widely held recognition that more work needs to take place in this area. A group of 24 ARL libraries funded and participated in a study that took place from May 2000 to December 2001. The project was under contract with Florida State University's Information Use Management and Policy Institute and was directed by Charles R. McClure, Wonsik "Jeff" Shim, and John Carlo Bertot under the leadership of project co-chairs, Sherrie Schmidt, Dean of University Libraries, Arizona State University, and Rush Miller, University Librarian and Director, University of Pittsburgh.
In October 2001, the project team completed the Phase II report that presents the findings from the field-testing of various statistics and measures and presents a list of recommended ones. The complete report is entitled "Measures and Statistics for Research Library Networked Services: Procedures and Issues: ARL E-Metrics Phase II Report" and is available on the ARL website http://www.arl.org/stats/newmeas/emetrics/index.html.
This study provides one approach, a beginning approach, for describing and measuring some of the resources, uses, and expenditures for supporting networked services in a research library setting. Such statistics and measures are essential for collections decisions; cost analysis; justification of services; services planning and evaluation; and a host of other activities. The Phase II report presents a first effort to accomplish these objectives and to standardize data collection techniques, definitions, and procedures related to networked and electronic resources and services.
Based on a substantial field-testing process (described in detail in the report), the project team recommends the following network statistics (Table 1) and performance measures (Table 2). The statistics and performance measures provide indicators of library networked services and resources.
The performance measures are composite and/or combinations of the network statistics along with, in some cases, non-network statistics already collected by ARL libraries (e.g., number of visitors to the library).
Table 1 |
|
Patron Accessible Electronic Resources |
|
Use of Networked Resources and Services |
U1 Number of electronic reference transactions |
Expenditures for Networked Resources and Related Infrastructure |
|
Library Digitization Activities |
D1 Size of library digital collection |
Table 2
|
|
Performance Measures |
|
The recommended network statistics and performance measures, either independently or in some combination, can assist research libraries in describing a number of aspects of their networked resources and services. There is a section in the report that provides libraries with some guidance regarding the use to which the network statistics and measures can be put.
ARL libraries may currently collect measures that can provide some indication of the success of a particular program or service provided to customers, such as user satisfaction surveys. However, it is important to think broadly--with the desired state in mind--and not simply use the measures at hand because they are easy to collect or because a lot of time and effort has been devoted to collecting them.
In most cases, a single measure on its own is not enough to indicate whether a research library is successful in a given area. To accurately indicate the success or quality of an academic library, measurement should be implemented at three key levels: outcome level, use/capacity level (output), and resources level (input). (See Table 3 for illustrative questions at each level.)
Given the rapidly changing technology environment, the changing milieu affecting higher education, changing organizational structures within ARL libraries, and the complexity of measuring such networked services, it is almost certain that the statistics and measures proposed in this study will continue to evolve. The measurement tools offered in the Phase II report, however, will provide research librarians with important techniques to count, describe, and report networked services and resources in their libraries.
Table 3
|
||
Outcome Level |
Use/Capacity Level
|
|
|
What are the results of a program or process? |
||
|
How successful or effective is the library? |
||
|
How effective do customers perceive your programs to be? |
||
|
What beneficial effects are you having on your customers? |
||
|
How could a program be changed to better suit the needs of your customers? |
||
"Measures and Statistics for Research Library Networked Services: ARL E-Metrics Phase II Report," ARL, no. 219 (December 2001): 8–9, http://www.arl.org/resources/pubs/br/br219/br219emetrics.shtml.