Association of Research Libraries (ARL®)

6. THE ONLINE SURVEY

6.1 The Online Interview Instrument

The online instrument is mounted as an HTML form. The key questions are presented here along with an example of the pull down list that accompanies one of the questions.>Exhibit 5. Online Survey Instrument: Non-Reference Books

We initially launched the Web questionnaire in two parts. The reader was given the initial part, which asked questions that could be answered before he used the book, e.g., about his reason for using the book, timing of need for the material he was seeking, and his status, when he clicked on the title of the book. He was not required to complete it in order to move on to the book, but it was easy to respond at least in part. The scholar was asked to click on the button taking him to the second part when he finished his session with the book; it asked various questions about how he felt about the online format. We could not force the user to go to the second questionnaire and hardly anyone did. At the same time, the online book designers found working with two questionnaires to be difficult.

In preparation for the Fall 1996 semester, we switched to a single questionnaire format in which the scholar must choose to go to the questionnaire after he uses the book. Response rates have been poor with fewer than ten questionnaires submitted in any week and many of those responses incomplete.

Data captured from the questionnaires are processed (using Unix utilities) to produce a standard data file for input into SAS or SPSS. Findings from the most recent data are summarized below.

6.2 Early Results from CWeb Survey

6.2.1 CWeb Survey Responses by Online Text Used

From late September 1996 through early June 1997, we received 85 responses to the CWeb questionnaire.

The OED is both the most used of the online books and the one for which the most survey responses were returned.

Given The OED's overwhelming presence in the responses, this analysis is largely one of reactions to the online OED. In a few cases the analysis distinguishes between The OED and all of the other texts.

6.2.2 CWeb Survey: Primary Project for Using Book

The questionnaire asked the scholar to select the best description of that work or project for which you are using this book and gave a choice of five options. The distribution of responses was:

Research projects are the major purpose for using the online books.

6.2.3 CWeb Survey: Ways of Using Book

The questionnaire asks What did you do with this book on this occasion? (Select all applicable uses.) It offers different reasons for the monographs and for the various reference books.

For the OED responses, the distribution of book uses was:

For all the other books, the distribution of uses was:

Those who reported that they used the online book by reading part of it were asked how much they read. Responses were distributed as follows:

The majority of these online book users read less than 10 percent, say one chapter, online.

6.2.4 CWeb Survey: Forms of This Book Ever Used

The questionnaire asks Which forms of this book have you ever used? and offers the scholar nine options. Responses were distributed as follows:

For The OED, paper copy in the library received the most mentions with online copy elsewhere coming in a close second. For the other books, online copy elsewhere was the dominant response.

6.2.5 CWeb Survey: Preferred Form of This Book

The questionnaire asked If you have used this book in more than one way, which one do you prefer overall? The same choices were offered as above. Responses (56 for The OED and 17 for the other books) were distributed as follows:

Online copy used outside the library is far the preferred book form for The OED with more than twice the votes as the next most preferred form, online copy used in the library. Printout from online copy ranked third. The various forms of using the online OED received over 80 percent of the preferences votes.

The responses for the other books are also revealing. Just over a third of respondents preferred my own paper copy. Given the attributes ranked as important - always available and easily annotated, this is a logical top runner for non-reference books. However, various forms of using the online book received all the other votes of this small sample of users of the online book collection.

6.2.6 CWeb Survey: Reasons for Preference

The questionnaire asked Referring to the way of using this book that you prefer, why do you like it best? (Select all that apply.) Responses were distributed among the options offered as follows:

Easy to get to, which had no mentions in the in-class survey, was the most popular response given in this survey. In part this reflects the heavy presence of The OED in this survey, but this reason for the preference also tied with always available for the other responses involving books other than The OED.

6.2.7 CWeb Survey: Preferred Format and Reasons for Preference

Looking at all the responses, the top reasons for each format being preferred were:

The popularity of easy to get to is consistent with the preference for online copy used outside the library as long as the respondent has easy access to a computer with a Web browser.

6.2.8 CWeb Survey: Frequency of Use in Past Three Months

The questionnaire asked On how many occasions (including this one) have you used this book in any format during the last 3 months? The 79 responses were distributed as:

Those responding 'zero' were not following the directions to the question and presumably meant that this was their first occasion to use this book in this period. The mean was 8.7 occasions and the median 3.0 occasions - or an average of about three occasions per month based on the mean or once a month based on the median. It may be that heavy users of online books are more likely to notice our questionnaire and ultimately to respond and, hence, to be over-represented in this sample. However, the question asks about use in all formats.

6.2.9 CWeb Survey: Total Usage in Minutes in Past Three Months

The questionnaire asked For approximately how many minutes in total have you used this book during the last 3 months? The 79 responses were distributed as:

Again, those responding 'zero' were not following the directions to the question and presumably meant that they had not spent any time with this book previously in this period. The mean was 22 minutes and the median 15 minutes. These are not great amounts of time for using a monograph but they are substantial for using a dictionary.

6.2.10 CWeb Survey: Frequency of Use of Any Online Book in Past Year

The questionnaire asked About how many times in the past 12 months have you used an online book, i.e., a monograph or reference book available on CNet or another computer network? The 75 responses were distributed as:

The mean was 15 uses in the past year and the median five uses. This sample is most likely not representative of all users of the online books, let alone of the Columbia community.

6.2.11 CWeb Survey: Effect of Online Books on Scholarly Work

Two key questions asked on all of our questionnaires, other than those distributed in class, seek to determine the effect of online books on scholarly work.

• In doing the type of work for which you used this book, do paper books or online books help you be more productive?

• Do you find that you are able to do work of higher quality when you use paper books or online books?

The questionnaire offers a range of seven responses from Much greater productivity (quality) with paper through No Difference to Much greater productivity (quality) with online plus Cannot Say. 6.2.11.1 CWeb Online Survey: Productivity, Book Type and Format

As the following table shows, many OED users felt that they are more productive using the online OED works while only a modest number of the users of the other online books felt that they are more productive using online books.

Of the group of 64 users of The OED, 50 percent believed that they were more productive with online books and 36 percent believed that they were more productive with print books. Only one respondent thought there was no difference and eight responded cannot say. The 21 users of the other books did not share this feeling. Only 19 percent believed that they were more productive with online books and 48 percent believed that they were more productive with print books. However, another 19 percent noted no difference in productivity and ten percent responded cannot say.

6.2.11.2 CWeb Online Survey: Work Quality, Book Format and Type

As the following table shows, the distribution of responses to the second question about the quality of work when using print and online books supports the print format in general, although many respondents found no difference in their work quality with the two formats.

For The OED, 37 percent supported print books, 16 percent backed online books, and 31 percent perceived no difference in work quality. For all other books, 52 percent voted for print books, 14 percent for online books, and 29 percent perceived no difference in quality.

These responses are somewhat puzzling as the reference book most used online is The OED and the features of the CWeb version provide as much utility if not more than the print version (with the exception of being able to view neighboring entries at a glance).

Cross-tabulation of these two questions finds considerable correlation in the responses - those who supported the paper version for productivity tended to support it for quality as well.

Almost a third of the 85 respondents ranked paper books as yielding both greater productivity and greater quality, while only one person ranked paper books better for productivity and online books better for quality. About 14 percent ranked online books better on both scores, while about ten percent ranked online books better for productivity but paper books better for quality.

6.2.12 CWeb Online Survey: Columbia Cohort of Respondents

The questionnaire asked a respondent to select one of several statuses offered as that which represented his present primary relationship to Columbia University. The responses were distributed as follows.

Of the 85 respondents whose questionnaires were analyzed above, 58 percent were undergraduates and 24 percent graduate students. This is consistent with the server data on OED user status, which identified 58 percent of users and 55 percent of hits with undergraduates, and six percent of users and 11 percent of hits with graduate students.

6.2.13 CWeb Online Survey: Discipline of Respondents

The questionnaire asked a respondent to select one of 16 disciplines (including Other) as that which defined his scholarly focus. The 85 responses were distributed as follows.

As the table shows, as might be expected, many of the undergraduate respondents have not yet selected a discipline. There were no representatives of seven possible disciplines, including major ones such as Social Work, Social Sciences, and Natural and Physical Sciences, in the responses.

6.2.14 Online Survey: Place in Project

We will need to explore the responses to this survey closely now and as we track it in the future and utilize our findings in structuring the interviews we undertake in the months ahead. It would be surprising if we do not see a shift in responses as our collection grows and as users have an opportunity for continuing use. Of course, we may have difficulty eliciting repeat responses to our questionnaire from the same individuals. However, perhaps some repeat users who have not completed the questionnaire will do so in the future. If necessary, we will be more aggressive in seeking feedback from users, e.g., by sending them questionnaires or interview requests in email or by telephone.

We are exploring various methods to increase our response rate. From March 15 to May 31, 1997, there were 42 hits on the survey button; 14 on the OED survey and 28 on the monograph survey. In this period, 280 people used the online book collection; thus, only 15 percent of them went to the survey during any of their sessions with the collection. During this period, 22 completed surveys were submitted, for a 52 percent return on surveys viewed. We are hopeful that introducing a frames design to our books, with the survey button on the frame along with navigational and search buttons, will remind users about the survey and encourage them to go to it and complete it. Clearly, getting that initial interest is critical to getting users to assist with our research by completing the questionnaire.

Other options we are exploring include breaking up the online questionnaire so that users confront only a screen full of questions (i.e., each respondent would answer only a subset of our questions), however, the non-response problem is one of getting the users of the online books to click on the questionnaire button much more than one of getting them to complete the questionnaire once they have done that. We are exploring changing our incentives, such as by instituting an improved lottery, but changes to date have not had a notable impact.

6.3 User Comments

We are gathering more contextual feedback from users through follow-up questions on email and through personal interviews. We have been using this feedback in making design decisions and we will be pulling it together more systematically over the course of this semester and early next summer.

Comments on questionnaires help us keep grounded in our work. The following example, quoted in full, shows remarkable insight into the complexities of assessing impact in a rapidly changing environment. It was anonymous.