Association of Research Libraries (ARL®)

Kenneth R. Smith
Eller Distinguished Service Professor of Economics
And Faculty Associate to the Provost
The University of Arizona

I would like to acknowledge the advice and counsel of Dean Carla Stoffle of the University of Arizona Library. Members of her staff and a number of members of ARL helped me think about learning outcomes from the perspective of the library and I thank them for their support and encouragement. I also learned much of what I know about outcomes assessment from working on the University of Arizona pilot initiative with Elena Berman and colleagues from a number of academic departments across the University. I retain sole responsibility for the limitations of the final product.

The well-ordered world of the university library is giving way to a new era of promise and uncertainty, of technology, of new forms of engagement and redefined roles in learning. The assumptions and rationales that have been relied on in the past are inadequate as a guide to future developments. To be successful in this new environment those responsible for academic libraries need to understand:

o the changing expectations faced by universities,
o how universities are responding to these new expectations,
o how this affects the library’s mission,
o how libraries can develop a strategy to be a central part of the university’s response, and
o how the Association of Research Libraries can assist its members to take on new roles.

The purpose of this paper is to answer these questions in order to provide a fuller understanding of student learning outcomes at the university level and the relationship to emerging library roles and contributions. An important conclusion of this paper is that "shared need creates opportunity" for the library to become an even more central part of the University learning community.

Changing Expectations

Today’s debate is a dialogue about vision and how that vision is to be achieved. It is a discussion about the nature of the educational experience. The central idea is to use the concept of learning to redefine the mission of the University.

The relevance of learning as a central concept is that it requires us to focus attention on the student’s experience. It requires that we rethink the curriculum, moving from a model in which we package knowledge around the expertise of the faculty to a model based on the learning outcomes realized by students. These outcomes include not only what students know, but also the skills they develop, what they are able to do and the attitudes of mind that characterize the way they will approach their work over a lifetime of change.

This concept of learning requires a shift in focus from the teacher’s knowledge to the student’s understandings and capabilities. This shift in focus leads to a new perspective on the development of quality in the academic enterprise. More than anything, it requires the faculty to bring the strength of the research paradigm into the learning process. The high quality of research in American universities is, in part, the result of the central role of assessment in the research process. The best evidence of this value is the fact that, in research, faculty put their assessment activities (peer review, participation on peer panels) on their resumes.

In viewing our mission from the student’s perspective, we must constantly ask whether student learning is enhanced by the way we teach, by the organization of the university, by the structure of the academic program, and by the activities of faculty and other professionals. The assessment of student outcomes is a means of focusing our collective attention, examining our assumptions and creating a shared academic culture dedicated to understanding what we are doing and how well we are doing it and to improving the quality of learning that results.

What has become clear is that there is a broader view of the learning outcomes that is necessary for success. It is recognized that universities provide their graduates with an excellent base of knowledge. It is a measure of our success that their knowledge, to a significant extent, does not differentiate among our graduates. Their ability to apply knowledge in new situations, their skills (communication, teamwork, information and technical literacy), and the values and attitudes that affect how they work have become more critical factors in determining how effective graduates are as they apply themselves throughout their careers.

How Universities Are Responding

To respond to these new expectations involves developing the scholarship of teaching and learning. With an understanding of student learning objectives, the scholarship of teaching and learning identifies critical issues, uses research methods, and applies results to understand and improve learning outcomes.

For over a decade, institutional and professional accreditation bodies have been shifting their attention from input measures (faculty, courses, books) to outcomes measures (what students learn). Universities and colleges are required to develop and implement a student outcomes assessment program. Assessment requires academic organizations (departments, colleges, universities) to:

o make expectations and standards for quality explicit and public
o systematically gather evidence on how well performance matches those expectations and standards
o analyze and interpret the evidence
o use the resulting information to document, explain and improve performance.

More than anything, assessment is a means for organizing a conversation among the faculty and other professionals responsible for an academic program. The objectives of this conversation are to:

Understand our students

o What are they like when they arrive?
o What are their educational and personal backgrounds, their needs, attitudes, goals and values?
o What are the challenges they face now and in the future?
o What becomes of our students after they graduate?
o What opportunities are available to them, which ones do they pursue and what determines their ultimate success in navigating the future?

Determine learning outcomes required for student success

o What are the societal and professional expectations for our graduates; which external constituencies have a stake in how well our students are prepared?
o Based on a shared understanding of university and societal expectations and our disciplinary knowledge, what are the outcomes we desire from the learning experience?
o What do we want them to know and what capabilities or competencies do we intend to develop?
o What are the dimensions of learning on which we must focus our efforts?

Identify how the academic program achieves desired learning outcomes

o What is the relationship between our academic program, curricular and co-curricular, and the achievement of particular learning outcomes?

Measure the extent to which outcomes are achieved

o What cost effective assessment strategies will determine whether, or to what extent, students are achieving specific learning outcomes?
o How can we identify areas in which the program is not achieving acceptable results?

Use the knowledge to improve academic programs

o Can we develop a shared understanding by the department faculty of the reasons for the shortfall or the issues to be addressed?
o What changes in the academic program will address these reasons?
o After implementing these changes, do we get the desired improvements?

The focus on learning involves looking at the academic program not from the perspective of its subject matter content but from the perspective of the competencies to be developed by students. These competencies include knowledge, skills and abilities, and attitudes and values that are important to a graduate’s future success. It is easy to treat these new expectations as one more example of a demand for accountability in higher education. A closer examination leads me to conclude that they represent a recognition of the changing needs of graduates to be successful in an increasingly complex knowledge economy.

Faculty have always assessed the performance of individual students within their individual courses. The focus of outcomes assessment is on the collective success of the program in developing the competencies of the students in the program. The faculty is being asked to accept responsibility for a broader set of outcomes. To a significant extent this represents a new challenge because while faculty are knowledge experts, they are not necessarily learning experts.

Students will also be required to approach the learning experience differently. They are being asked to become more actively involved in the learning process. While they have mastered the passive accumulation of knowledge, they are less comfortable with the responsibility to demonstrate their ability to use knowledge in unstructured situations. They must become proficient as members of teams, in communicating their solutions, and effectively taking advantage of access to information and the use of technology. Finally, they must begin to develop and demonstrate maturity in how they approach their work.

Employers believe we do an excellent job at providing our graduates with a strong base of knowledge. Because of that, what our graduates know is not a major concern. What employers are most concerned about is performance in the other dimensions of learning - skills, conceptual abilities and attitudes of mind - and especially the ability to learn itself.

The focus on learning outcomes leads to a consideration of the learning process and the learning community. Consider Figure 1. The learning outcomes represent a set of competencies of the graduate. From the University’s perspective, they are achieved as a result of a total experience over a four (or more) year period. Each element of the educational program contributes, directly or indirectly, to their achievement. Looking at the learning process allows us to recognize the various activities that contribute to learning. On the far left of the figure we see how foundational courses (math, composition, etc), general education courses and special prerequisite service courses prepare the student for the major. The requirements for the major are designed to produce the learning outcomes necessary for the graduate to be successful. Across the bottom of the figure we see how the program offerings of the library, student life and technical services can contribute to the learning outcomes of the graduate.

Looking at the learning community allows us to consider how faculty, students and other learning professionals can contribute to learning outcomes. The faculty responsible for the major is in the best position to develop the complete set of learning outcomes, since those outcomes will depend on the specific objectives for the degree program. In doing so they will need to incorporate the outcomes that the faculty of the University have concluded are important for all students. They will also recognize that the department can take advantage of the contributions of colleagues throughout the university.

The American Association for Higher Education’s Principles of Good Practice for Assessing Student Learning recognize that "student learning is a campus-wide responsibility, and assessment is a way of enacting that responsibility. Faculty play an especially important role, but assessment questions can’t be fully addressed without participation by student affairs educators, librarians, administrators, and students. …assessment is not a task for a small group of experts but a collaborative activity; its aim is wider, better informed attention to student learning by all parties with a stake in its improvement."

Figure 1

The University is made up of various communities, each of which has a direct and/or indirect view of student learning outcomes. For example, faculty with responsibility for designing and delivering a general education program must define outcomes that assure students across the University will have a common foundation and also contribute to the learning outcomes that are important in many degree programs. An academic department has a direct interest in the learning outcomes for students in their majors. A department with an important course that serves a critical role in preparing students in another department’s degree program has an indirect interest in the learning outcomes for students in that major. For example, physics is an important service course for students in an engineering program. The Physics faculty can not be effective unless they are working with the Engineering faculty to produce a clear set of learning outcomes for their service courses that will help achieve the outcomes desired for the engineering students.

The Library's Mission

How does the focus on learning outcomes affect the mission of the Library? Like other communities at the University, the library must move from a content view (books, subject knowledge) to a competency view (what students will be able to do). Within the new environment, we need to measure the ways in which the library is contributing to the learning that the University values. Like the general education program, the library has a direct and an indirect interest in the learning outcomes for all the students at the University. Like the Physics Department, the Library should be able to contribute to the achievement of learning outcomes for various academic programs across the University.

The Library and Student Learning Outcomes

It is useful to begin by asking, within their own expertise and their understanding of what will make students successful, what do library professionals consider key learning outcomes. One potential answer to this question is provided by the Information Literacy Competency Standards for Higher Education, approved by the Association of College and Research Libraries on January 18, 2000. As an alternative, I asked two groups of librarians to help me define a set of learning outcomes. My goal was not to achieve a definitive answer but rather to provide an example that would help me discuss how academic libraries might begin to participate in this campus wide activity. The following table is illustrative of what might be produced in such an exercise.

Student Learning Outcomes

Become self reliant (comfortable and confident) in information literacy skills including:

• Identifying information needs
  
• Finding/locating information
  
• Selecting relevant information
  
• Assessing and evaluating information
  
• Synthesizing
  
• Using information effectively
  
• Presenting information
  
• Students understand and use the information search process (eg. Kuhlthan model)
  
• Understand different formats of information and deal with them effectively
  
• Be aware (have an accurate mental model) of the structured nature of information
  
• Understand how to evaluate bias and the credibility of information
  
• Appreciate the way the quality of information varies along an historical continuum
  
• Understand the social/ethical/political/economic implications of information and intellectual property
  
• Understand the research process through which new knowledge is created
  
• Understand the scholarly communications cycle and its application to scholarly research
  
• Become self confident and comfortable in information rich environments
  
• Develop attitudes of openness, flexibility, curiosity, creativity, and an appreciation of the value of a broad perspective.
  

Developing a set of learning outcomes will allow libraries to determine the extent to which their interests are aligned with the expectations of other academic communities in the University. They will find that faculty responsible for the general education program as well as those responsible for many of the academic degree programs also are interested in critical thinking, the effective use of information and technology, the search process and collaborative reasoning.

We have described above how current expectations require consideration of a broader set of student learning outcomes, not simply the subject material of a particular program. We discover that some of these outcomes are common to programs across the University. What students need to be able to do (critical thinking and creative ability), their ability to manage technology and implement an efficient information search, and their skills in communicating and collaborative reasoning are fundamental across many subject domains.

The library can build on a shared view of what are important student learning outcomes. All the individual communities are being asked to prepare students in ways that go beyond their expertise in their fields. It is this shared need to go beyond our traditional focus on what students need to know that creates an opportunity for the library.

Consider for a moment the way in which a department faculty might look at the learning outcomes for their degree program and how they are achieved through the course requirements. Having agreed on what outcomes they believe are important to their graduate’s future success, they can ask the faculty responsible for each course to identify the extent to which each outcome is a focus of the course. Collectively, across the curriculum, they can determine which outcomes are covered to a major, moderate or minor extent. At this point, while they haven’t yet assessed how well their students have developed on each learning outcome, they can evaluate whether enough attention is being paid to individual outcomes.

Table 1 presents a matrix that shows which courses in the accounting curriculum contribute to each of the learning outcomes. Let me draw two conclusions from this matrix. First, observe that many of the learning outcomes are similar to those identified for the library. Second, while the Accounting Department has a well-defined set of course offerings through which they attempt to develop these outcomes, they recognize that in some areas they have been unable to give an outcome adequate attention. Those outcomes that receive the most attention are, for the most part, those that the faculty have felt best prepared to address.

Table 1. Learning Outcome Matrix for Undergraduate Accounting Courses

These conclusions suggest that departments may be very receptive to including in their courses, "offerings" developed and delivered by the library to increase the emphasis on a number of shared outcomes, especially where the expertise of the library complements the expertise of those in the academic programs. By "offerings," we mean units of learning materials designed to develop competency in specific learning outcomes that are considered important by the library and by other academic programs. They are a way to give the library a curriculum (its own set of course segments) and an opportunity to connect this curriculum to other academic programs.

To be effective, these "offerings" must be incorporated into required courses. Thus, there is a need for the library to engage in a dialogue with departmental faculty in order to identify ways in which they can contribute to the learning outcomes of the academic program. The library must take the initiative in determining what the library has to offer that will help the department achieve greater success in achieving their learning outcomes. It is unlikely that the Department on its own will identify the library as a place to turn for help.

As an example, the University of Arizona Library is offering instruction sessions based on the ACRL Information Literacy Competencies, using the Kuhlthau "Information Search Process" model. The instruction will be integrated into a general education natural sciences course offered by the Astronomy department. The library team has developed learning modules that focus on entry-level information literacy competencies customized for use in Astronomy related studies. They will be a required part of the course but offered at a variety of times outside the normal class period. The aim is for students to become confident and skillful information users.

To pursue this strategy also requires that the library create new roles for its learning practitioners. To some extent and in some libraries this process of change has begun. Libraries have developed organizational strategies to serve the various academic communities. But the focus to date is primarily on making information more and more accessible rather that addressing specifically the learning outcomes important to student success. The library needs to ask what kind of expertise is required to be actively engaged in the learning process and an effective partner in achieving learning outcomes. It then will be in a position to adapt roles and responsibilities of its professionals to take full advantage of the opportunity.

How Can ARL Help Its Member Institutions?

Esther Dyson, a well known commentator of the information technology scene has observed that "we are entering a new economic environment – where a new physical set of rules will govern. Chief among the new rules is that ‘content is free.’ While not all content will be free, the new economic dynamic will operate as if it were." Value will increasingly be defined not in terms of collections or even access to information but by creating solutions that help to achieve student learning.

It is timely to think about the ways in which ARL can help its member institutions become involved in the assessment of student outcomes and demonstrate the value of their contribution to the University. What is clear is that we are at the beginning of a period of learning how to do this in an effective and efficient manner.

What works best at this stage is experimentation. At the University of Arizona, we have developed a pilot initiative to learn how to help academic departments respond to expectations for assessment of student outcomes. The strategy was to begin with volunteer departments interested in the assessment of student outcomes, provide them with information and support and make their experiences available to others. Participants agreed that stories and examples are helpful. These stories and examples are being shared through periodic meetings and through the organization of a "tool kit." A description of our tool kit is included in the appendix. It is important for libraries to understand the processes that are used to define learning outcomes, to select measures, to collaborate with other academic departments, and to use the results to improves their programs. In time, a tool kit will include a composite of best practice ideas that can be adopted by other departments.

Even with a group of departments that have volunteered to participate, progress is uneven. Each department has gotten started in its own way. However, we have observed that those who are achieving results have a significant impact on other departments in the pilot initiative and on those who are ready to become involved. ARL, similarly, will have members whose early engagement and resulting accomplishments will attract others to become involved.

As more and more major research universities are successful in using outcomes assessment to improve student learning and to demonstrate the way they are preparing students, it will be important that libraries are an effective part of their campus assessment program. Within the community of research universities, there are a number who are already leaders in the assessment of student outcomes. The University of Colorado, Boulder has almost a decade of experience, a testament to the impact of a mandate by the Governor. Others who are significantly engaged are the University of Wisconsin, Madison and the University of Illinois at Urbana-Champaign.1

What is clear is that expectations are accelerating and all members of ARL will have to respond. Within this new environment, it will be important to the Library to be recognized by their colleagues across the campus as a full contributor. Those who become involved in a pilot will play a critical role in demonstrating how this can be a valuable activity. Their leadership will help their colleagues to understand the opportunity and participate in their campus progress.

A pilot process for ARL would involve a number of libraries working through a sequence of activities and sharing experiences in periodic meeting and, more importantly, in a best practice tool kit. The activities would include:

o Develop learning outcomes from the library’s perspective
o Develop curriculum segments or "offerings" through which the library would achieve the outcomes
o Understand the learning outcomes of academic degree programs
o Consider how library offerings can be integrated into academic courses to achieve shared outcomes
o Identify ways to measure how well outcomes are being achieved
o Collect data and use information to modify curriculum strategies

To be successful in this new era, the library must contribute to student learning. This represents an expanded responsibility and a more active role in the learning process. The focus has moved beyond access to content or to tools. What is important is how the library’s capabilities can provide solutions that measurably impact the quality of learning. It will require a significant period of learning new ways to participate and new roles for the library professionals. To make this period of learning effective, ARL needs to organize a pilot initiative and share creative solutions with all its members. In this way their member schools will be better able to turn student outcomes assessment into an important opportunity to make the library an even more central part of the University.

Appendix

Tool Kit

This Tool Kit is based upon brainstorming sessions engaged in by representatives of 1999-2000 pilot departments. Broad interest was expressed in hearing about the process other departments are using as well as the products. Participants strongly agreed that stories and examples are helpful and should be part of the Tool Kit. There was also consensus that the organization should follow the temporal sequence of activities. The headings below are intended to capture not only products – results, instruments, etc. – but also the activities and human climate that resulted in the products. The Tool Kit may be accessed both by department and by heading.

[1] See http://www.Colorado.EDU/pba/outcomes/ , http://www.wisc.edu/provost/assess/manual.html, and http://www.oir.uiuc.edu/assessment/