by Wonsik "Jeff" Shim, Charles R. McClure, and John Carlo Bertot, Information Use Management and Policy Institute, School of Information Studies, Florida State University

The E-Metrics project, one of the ARL New Measures Initiatives, is an effort to explore the feasibility of defining and collecting data on the use and value of electronic resources. ARL has some experience in tracking expenditures for electronic resources through the ARL Supplementary Statistics, but there is a widely held recognition that more work needs to take place in this area. A group of 24 ARL libraries funded and participated in a study that took place from May 2000 to December 2001. The project was under contract with Florida State University's Information Use Management and Policy Institute and was directed by Charles R. McClure, Wonsik "Jeff" Shim, and John Carlo Bertot under the leadership of project co-chairs, Sherrie Schmidt, Dean of University Libraries, Arizona State University, and Rush Miller, University Librarian and Director, University of Pittsburgh.

In October 2001, the project team completed the Phase II report that presents the findings from the field-testing of various statistics and measures and presents a list of recommended ones. The complete report is entitled "Measures and Statistics for Research Library Networked Services: Procedures and Issues: ARL E-Metrics Phase II Report" and is available on the ARL website http://www.arl.org/stats/newmeas/emetrics/index.html.

This study provides one approach, a beginning approach, for describing and measuring some of the resources, uses, and expenditures for supporting networked services in a research library setting. Such statistics and measures are essential for collections decisions; cost analysis; justification of services; services planning and evaluation; and a host of other activities. The Phase II report presents a first effort to accomplish these objectives and to standardize data collection techniques, definitions, and procedures related to networked and electronic resources and services.

Recommended Statistics and Performance Measures

Based on a substantial field-testing process (described in detail in the report), the project team recommends the following network statistics (Table 1) and performance measures (Table 2). The statistics and performance measures provide indicators of library networked services and resources.

The performance measures are composite and/or combinations of the network statistics along with, in some cases, non-network statistics already collected by ARL libraries (e.g., number of visitors to the library).

<td width="65%" valign="TOP" > 
  <p>R1&#9;Number of electronic full-text journals<br>
    R2&#9;Number of electronic reference sources<br>
    R3&#9;Number 
    of electronic books </p>
  </td>

<td width="65%" valign="TOP" > 
  <p>C1 &#9;Cost of electronic full-text journals<br>
   C2&#9;Cost of electronic reference sources<br>
    C3&#9;Cost of electronic books<br>
   C4&#9;Library expenditures for bibliographic 
    utilities, networks, and consortia<br>
    C5&#9;External 
    expenditures for bibliographic utilities, networks, and consortia 
  </p>
  </td>

    U4&#9;Items requested in electronic databases<br>
    U5&#9;Virtual 
    visits to library&#146;s website and catalog </p>
  </td>

    D2&#9;Use of library digital collection<br>
   D3&#9;Cost 
    of digital collection construction and management </p>
  </td>

<td width="70%" valign="TOP"> 
  <p>P1&#9;Percentage of electronic reference transactions 
    of total reference<br>
  P2&#9;Percentage of remote library visits 
    of all library visits<br>
    P3&#9;Percentage of 
    electronic books to all monographs </p>
  </td>

Using the Network Statistics and Performance Measures

The recommended network statistics and performance measures, either independently or in some combination, can assist research libraries in describing a number of aspects of their networked resources and services. There is a section in the report that provides libraries with some guidance regarding the use to which the network statistics and measures can be put.

ARL libraries may currently collect measures that can provide some indication of the success of a particular program or service provided to customers, such as user satisfaction surveys. However, it is important to think broadly--with the desired state in mind--and not simply use the measures at hand because they are easy to collect or because a lot of time and effort has been devoted to collecting them.

In most cases, a single measure on its own is not enough to indicate whether a research library is successful in a given area. To accurately indicate the success or quality of an academic library, measurement should be implemented at three key levels: outcome level, use/capacity level (output), and resources level (input). (See Table 3 for illustrative questions at each level.)

Given the rapidly changing technology environment, the changing milieu affecting higher education, changing organizational structures within ARL libraries, and the complexity of measuring such networked services, it is almost certain that the statistics and measures proposed in this study will continue to evolve. The measurement tools offered in the Phase II report, however, will provide research librarians with important techniques to count, describe, and report networked services and resources in their libraries.

<td width="31%"> 
  <h3>Resource Level<br>
    (Input Measures) </h3>
</td>

<td width="37%" valign="TOP"> 
  <p>How much is a service, resource, or program being 
    used? 
</td>
<td width="31%" valign="TOP"> 
  <p>What do we need to ensure success? 
</td>

<td width="37%" valign="TOP"> 
  <p>Who is using a service, resource, or program? 
</td>
<td width="31%" valign="TOP"> 
  <p>What funding level is appropriate or necessary 
    for a particular program? 
</td>

<td width="37%" valign="TOP"> 
  <p>Why are people using a particular program? 
</td>
<td width="31%" valign="TOP"> 
  <p>Do we need more of a particular resource in order 
    to have a more effective program? 
</td>

<td width="37%" valign="TOP">&nbsp;</td>
<td width="31%" valign="TOP">&nbsp;</td>

To cite this article

"Measures and Statistics for Research Library Networked Services: ARL E-Metrics Phase II Report," ARL, no. 219 (December 2001): 8–9, http://www.arl.org/resources/pubs/br/br219/br219emetrics.shtml.