Association of Research Libraries (ARL®)

James G. Neal

Sheridan Director
Milton S. Eisenhower Library
The Johns Hopkins University

This paper outlines a series of quantitative and qualitative models for understanding and evaluating the use of electronic scholarly journals, and summarizes data based on the experience of Project Muse at Johns Hopkins University and early feedback received from subscribing libraries.

Project Muse is a collaborative initiative between the Press and the libraries at Johns Hopkins University to provide network-based access to scholarly journals including titles in the humanities, social sciences and mathematics. Launched with electronic versions of forty titles still published in print, Project Muse coverage has now been expanded to include electronic-only publications. Funded initially by grants from the Mellon Foundation and the National Endowment for the Humanities, Project Muse seeks to create a successful model for electronic scholarly publishing characterized by affordability and wide availability. It has been designed to take advantage of new technical capabilities in the creation and storage of electronic documents. It has been developed to provide a range of subscription options for individual libraries and consortiums. It is based on a very liberal use and re-use approach that encourages any non-commercial activity within the bounds of the subscribing organization.

Project Muse has been produced from the outset for usability, with a focus on user-centered features. This has evolved as a participative and interactive process, soliciting input and feedback from users, and integrating user guidance components into the system. An online survey is available to all users and libraries are providing information about the local implementation and the results of campus and community focus group discussions on Project Muse. As the number of subscribing libraries expands and the activity grows, a valuable database of user experiences, attitudes and behaviors will accumulate. A new feature will be the ability to track and analyze individual search sessions and to observe closely user activities. This will enable monitoring the impact of new capabilities and the efficiency of searching practices.

Six models of use analysis are discussed in this paper which cover both the macro or library-level and the micro or individual user-level activity:

1. subscribing organizations - which libraries are subscribing to Project Muse and how do they compare with the base of print journal customers

2. subscriber behaviors - how do libraries respond as access to electronic journals is introduced and expanded, and in particular, how are acquisitions like Project Muse accommodated in service and collection development programs and budgets

3. user demography - what are the characteristics of the individual user population, in such areas as status, background/experience, motivation, attitudes and expectations

4. user behaviors - how do individuals respond to the availability of scholarly materials in electronic format as they explore the capabilities of the system and execute requests for information

5. user satisfaction - what objectives do users bring to network-based access to scholarly information, and how do users evaluate system design and performance and the quality of search results

6. user impact - how are user research and information-seeking activities being shaped by access to full-text journal databases like Project Muse

One of the objectives of Project Muse is to achieve full cost recovery status by the completion of the grant funding period in 1998. Therefore, it is important to monitor the growth in the base of subscribing libraries and to evaluate the impact on the print journal business of the Press. An analysis of those libraries subscribing to the full Project Muse database as of June 1997 (approximately 400 libraries) demonstrates a very significant expansion in the college, community college and now public library settings with very low or no history of subscriptions to the print journals. The result is a noteworthy expansion in access to Hopkins Press titles with 70 percent of the subscribing libraries currently purchasing less than 50 percent of the titles in print, and over one-fourth acquiring no print journals from the Hopkins Press.

PROJECT MUSE
SUBSCRIBING LIBRARIES

ARL Universities          -     65 libraries
Other Universities        -    128 libraries
Liberal Arts Colleges     -    101 libraries
Community Colleges        -     53 libraries
      Public Libraries          -      3 library systems


PROJECT MUSE
CUSTOMER PRINT SUBSCRIPTIONS

  No. of Print                 Percentage of
Subscriptions                  Libraries
     0                           27.8
    1-4                           7.6
    5-9                           6.8
   10-14                         15.8
   15-19                         13.0
   20-24                         11.8
   25-29                          9.0
   30-34                          6.2
   35-40                          2.0

One of the explanations for these patterns of subscription activity is the purchase arrangement for Project Muse. Over 90 percent of the libraries are subscribing to the full Project Muse database of 42 titles. And due to very favorable group purchase rates, nearly 80 percent of Project Muse subscribers are part of consortial contracts. The cooperative approach to providing access to electronic databases by libraries in a state or region is widely documented, and the Project Muse experience further evidences this phenomenon.

Another objective of Project Muse is to enable libraries to understand the use of collections and thus to make informed acquisitions and retention decisions. The impact on collection development behaviors will be critical, as libraries do indicate intentions to cancel print duplicates of Muse titles and to monitor carefully the information provided on individual electronic title and article activity. Use information is beginning to flow to subscribing libraries, but there is no evidence yet of journal cancellations for Hopkins Press titles.

An important area of analysis is user demography, that is the characteristics of the individuals searching the Project Muse database. An online user survey and focus group discussions are beginning to provide some insights:

-- The status of the user, that is undergraduate student, graduate student, faculty, staff, community member, or library employee. As Project Muse is introduced, library staff are typically the heaviest users, followed by a growth in student use as campus awareness and understanding expands.

-- Type of institution, that is research university, comprehensive university, liberal arts college, community college, or public library setting. As Project Muse subscriptions have increased and access has extended into new campus settings, there has initially been heavier use in the research universities and liberal arts colleges where there is either traditional awareness of Project Muse titles or organized and successful programs to promote availability.

-- The computer experience of users, that is familiarity with searching full-text electronic databases through a Web interface. Project Muse users tend to be knowledgeable Internet searchers who have significant comfort with Web browsers, graphical presentations of information, and constructing searches in textual files.

-- The location of use, that is in-library, on-campus in faculty office and student residence hall, or off-campus. Preliminary data indicates that the searching of Project Muse is taking place predominantly on library-based equipment. This can be explained by the inadequate network infrastructure that persists at many campuses or the general lack of awareness of Project Muse until a user is informed by library staff about its availability during a reference exchange.

-- The browsers used to search the Project Muse database. An analysis of searches over an 18-month period confirms that Netscape browsers are used now in over 98 percent of the database activity, with a declining percentage of Lynx and other non-graphical options.

Project Muse enables searching by author, title, or keyword, in the table of contents or full-text of the journals, and across all the journals or just selected titles. All articles are indexed with Library of Congress subject headings. Hypertext links in table of contents, articles, citations, endnotes, author bibliographies, and illustrations allow efficient navigation of the database. User searching behavior is an important area for investigation, and some preliminary trends can be identified:

-- The predominate search strategy is keyword, with author and title inquiries occurring much less frequently. This can be partially explained by the heavy undergraduate student use of the database and the rich results enabled by keyword strategies.

-- Use of the database is equally distributed across the primary content elements: table of contents, article abstracts, images linked to text, and the articles. An issue for future analysis is the movement of users among these files.

-- Given the substantial investment in the creation of LC subject headings and the maintenance of a structured thesaurus to enhance access to articles, their value to search results and user success is being monitored carefully.

-- With the expansion of both internal and external hypertext links, the power of the Web searching environment is being observed, the user productivity gains are being monitored, and the willingness to navigate in an electronic journal database is being tested.

-- Users are directed to the Project Muse database through several channels. Libraries are providing links from the bibliographic record for titles in the online catalog. Library Web sites highlight Project Muse or collections of electronic journals. Subject pages list the Project Muse titles that cluster in a particular discipline.

-- Users are made aware of Project Muse through a variety of promotional and educational strategies. Brochures and point-of-use information are being prepared. In some cases, campus media have included descriptive articles. Library instructional efforts have focused on Project Muse and its structure and searching capabilities.

-- Printing and downloading to disk are important services linked to the effective use of Project Muse, given the general unwillingness of users to read articles online. Libraries have an interest in maximizing turnover on limited computer equipment, and are focused on implementing cost-recovery printing programs.

-- Project Muse is increasingly enabling users to communicate with publishers, journal editors, and the authors of articles through e-mail links embedded in the database. Correspondence has been at a very low level, but is projected to expand as graduate student and faculty use increases and familiarity and comfort with this feature expands.

With other 400 subscribing libraries and over three million potential users of Project Muse in the communities served, it is possible to document global use trends and the changing intensity of searching activity:

PROJECT MUSE
GLOBAL USE TRENDS
4th Quarter 1996                  1st Quarter 1997

1,833,692  Requests                 2,618,069  (+42.8%)
   19,922  Per Day                     29,090  (+46.0%)
      199  Subscribers                    322  (+61.8%)
    9,214  Per Subscriber               8,112  (-12.0%)

The progression of use over time as a library introduces access to Project Muse is being monitored. Early analysis suggests that the first two quarters of availability produce low levels of use, while third quarter use expands significantly.

User satisfaction with the quality and effectiveness of Project Muse will be the central factor in its long-term success. Interactions with users seek to understand expectations, response to system design and performance, and satisfaction with results. The degree to which individuals and libraries are taking advantage of expansive fair use capabilities should also be gauged.

Project Muse has focused on various technical considerations to maximize the dependability and efficiency of user searching. Detailed information on platforms and browsers is collected, for example, and access denials and other server responses which might indicate errors are automatically logged and routed for staff investigation.

Expectations for technology are generally consistent: more content, expanded access, greater convenience, new capabilities, cost reduction, and enhanced productivity. It will be important to monitor the impact of Project Muse in the subscribing communities and to assess whether it is delivering a positive and effective experience for users.

Over the now thirty-plus years of library automation activities, we have learned about those conditions which improve the positive impact of technology, and Project Muse and its implementation must respond to these needs:

-- if the computer package is more decentralized so that is in the hands of users (control);

-- if more developed computing capacity is available (power);

-- if users have greater competency and experience with computing (training);

-- if the developers are more responsive to the expressed needs of the user regarding design and operation of systems (support); and

-- if users routinely rather than selectively use computers and networked-based information systems (opportunity).

A recent study carried out jointly by the Association of Research Libraries and the Association of American Universities on electronic scholarly publishing identified a series of critical performance attributes for technology: ease of use, timeliness, responsiveness, accuracy, authenticity, predictability, adaptability, relevance, eligibility, cost, recovery, innovation and extensibility. These qualitative characteristics are essential benchmarks for evaluating Project Muse.

Charles Hildreth, an early investigator of online library catalog systems, established a series of analyses which serve as well our review of full-text databases. Hildreth cited five components for understanding the user interface:

-- physical, including the input-output equipment, and the structure and location of the workstation;

-- organizational, including the institutional setting, the availability of staff assistance, and the provision of user aids;

-- personal, including the abilities, experience, objectives and needs of the user;

-- communications, including the language and techniques of interaction; and

-- functional, including control of operations, search formulation and output.

Hildreth views user support in terms of five system qualities: easy to use, friendly and cordial, protective and forgiving, reliable and responsive, and adaptive and flexible. These various elements can be summarized in terms of three general characteristics: audience suitability or the degree to which effective use of the system is self-explanatory; metaphorical consistency or the extent to which a logical framework is provided; and display legibility. Systems which strive to support the user reflect a concern for these elements and emphasize both simplicity of design and searching power.

It is also important to maximize the core advantages of using information in digital formats:

-- accessibility, that is delivery to locations wherever users can obtain network connections

-- searchability, that is the range of strategies that can be used to draw relevant information out of the database

-- currency, that is the ability to make publications available much earlier than is possible for print versions

-- researchability, that is the posing of questions in the digital environment that could not even be conceived with print materials

-- interdisciplinarity, that is the ability to conduct inquiries across publications in a range of diverse disciplines and discover new but related information

-- multimedia, that is access to text, sound, images, video in an integrated presentation

-- linkability, that is the hypertext connections that can be established among diverse and remote information sources

-- interactive, that is the enhancement of user control and influence over the flow of information and the communication that can be integrated into the searching activity

Project Muse will be evaluated against these quantitative and qualitative models. It's success will ultimately be determined by its support for the electronic scholarly publishing objectives outlined in the ARL/AAU work:

-- foster a competitive market for scholarly publishing by providing realistic alternatives to prevailing commercial publishing options

-- develop policies for intellectual property management emphasizing broad and easy distribution and reuse of material

-- encourage innovative applications of information technology to enrich and expand the means for distributing research and scholarship

-- assure that new channels of scholarly communication sustain quality requirements and contribute to promotion and tenure processes

-- enable the permanent archiving of research publications and scholarly communication in digital formats

---

Copyright © of the papers on this site are held by the individual authors or The Andrew W. Mellon Foundation. Permission is granted to reproduce and distribute copies of these works for nonprofit educational or library purposes, provided that the author, source, and copyright notice are included on each copy. For commercial use, please contact Richard Ekman at the The Andrew W. Mellon Foundation.