Putting Knowledge to Work:
Holistic Collection
Assessment
Andrea Hudson, Reference Librarian
University of St. Thomas, Minnesota
Collection assessment is something that all librarians do. We count usage statistics on books and journals; we look at the number of hits to our websites; we track the number of questions we answer at the reference desk. Analyzing these numbers is important, but to get a comprehensive view of how useful a collection is to students and faculty, it is important to look at the numbers within the context of other measures. Librarians need to know the reasons behind shifts in usage patterns, and numbers alone will not give us these answers. A survey of ARL Libraries documented a 21% decrease in reference desk transactions and a 9% decrease in total circulation from 1991 – 2001. (Association of Research Libraries, 2002) Many factors can potentially explain these decreases; a major reason cited in this and other studies is the increased use of the web by students conducting research. This explanation is true to a point, and yet there are other factors that affect usage patterns--and other measures that libraries can enact to reverse negative trends.
In times of shrinking academic budgets, detailed assessment projects are an important tool. One area in which librarians maintain control is in choosing a mix of library resources that best reflects the needs of the intended users. This is where traditional collection assessment falls apart and new models need to be explored. Holistic collection assessment is a more comprehensive model; it not only pulls together many disparate quantitative measures, but also uses qualitative measures and narrative to place the data in context.
The University of St. Thomas (UST) is an urban Catholic university of 10,000 students with undergraduate liberal arts programs and graduate programs aimed at working professionals. The UST libraries initiated an experiment that goes beyond traditional methods of collection assessment in order to achieve a more comprehensive analysis of our resources. While the collection managers who approved the collection assessment project were primarily interested in the effective use of existing data, the taskforce had a slightly different agenda. Those who volunteered for the Collection Assessment Task Force (CATF) were interested in exploring qualitative as well as quantitative measures in order to give the numbers we already collected more meaning.
In 1997 a similar group at St. Thomas attempted an in-depth analysis; but due to the broad scope of the project, imprecise data gathering mechanisms, and the wide range of formats for library materials, the process proved too cumbersome and very few assessments were completed. The experience from five years ago is borne out in the professional literature concluding that as the range of materials analyzed increases, the traditional methods of assessment prove to be inadequate. (Grover, 1999, p.5) Despite these challenges, the current CATF members were committed to finding methods to analyze our collection in terms of how it supports our primary users. Taking cues from the previous group, the new assessment group started first by establishing a narrower scope of the collection to work with. We knew that to undertake this project in detail we could not analyze an entire discipline such as Business or Education; but instead should focus on concentrations within disciplines such as Human Resources Management and Educational Leadership.
The taskforce goals are twofold. A primary goal of the CATF is to facilitate access to all necessary data for ease of analysis. Since 1999, the librarians have worked on assessment projects such as annual database reviews and periodical evaluations that rely on hit counts and usage statistics. Therefore procedures in routine data gathering have been established and UST librarians are more accustomed to gathering data. In addition, library system vendors have greatly improved their data management software; consequently a majority of the vendor generated data needed for assessment is easier to collect. Even though data is now easier to collect, different departments within the library are responsible for data, so it is imperative for the librarians to bring the disparate data together for analysis.
Because this process relies heavily on the support of technical services, the CATF works closely with departments within the library to share with them our goals and gauge from them the best times of year to generate usage reports. We want to make sure that multiple librarians doing assessments will not simultaneously descend on a department for reports during a time of the year that is already very busy. Using shared workspaces such as intranets and institutional e-mail folders for the storage of annual reports and statistics whenever possible will also encourage the sharing of data.
The second goal for the group is to encourage librarians to think in terms of narrative. A common mistake in data collection is to assume that the data that is easy to collect is the correct data to collect. (Biblarz et al, 2001, p. 50) We want to collect the information that answers the fundamental question of whether or not our collection fits the research needs of our primary audiences. With shrinking budgets libraries must prove that they are using their money effectively. If the mix of library resources does not fit the research culture of a given department, then the focus of collection development for that department needs to change.
Anecdotal information can be gleaned from informal conversations with students, staff and faculty, but the UST librarians wanted to formalize the gathering of this information, record it, and use it to make better decisions regarding future purchases. To obtain quality data we want to conduct interviews and focus groups and perform assessments with faculty and students, preferably at the beginning and at the end of the semester to compare what students know at the beginning of the semester with what they know at the end of a semester in terms of research methodology. We also ask professors for bibliographies, and if possible, the grades associated with the bibliographies in order to analyze the sources used for an excellent paper vs. an average paper. Not only will these measures help us determine the resources that the students are using regardless of whether the resources are in our library, in another library, or free on the web.
We achieved our two goals by formalizing data collection methods, improving the communication of our goals to the entire library staff, and ultimately by creating the Departmental Resources Review Form (DDRF) that all librarians will use for the collection assessment process. The form clearly spells out which measures should be utilized, (circulation of books, in-house use of periodicals, ILL usage) and how to procure the data, including the direct links from the web-based DRRF to the appropriate data depository. We anticipated some reluctance on the part of liaison librarians as to the time commitment that this project would entail, so we worked hard to identify ahead of time which department holds specific data.
The concluding section of the DDRF is for analysis and recommendations. The librarian doing the assessment is required to synthesize all of the disparate information they collected – quantitative and qualitative – into an executive summary. Lessons learned from this kind of analysis are not just the number of time that materials have been used in a year, but the preferred format in which members of a department do research. Departments even within the same discipline can approach research very differently, some preferring current journals to books, others preferring to use only electronic databases to find articles. There are other disciplines where professional organizations and their literature are the primary resources students rely upon. Knowing this background from usage counts, bibliographies and interviews together gives the librarians a better idea of what to focus on with future collection development.
One area for improvement of this and other assessment models is that it does not fully address the needs of the unknown or under-represented groups. (Biblarz et al, 2001, p.52) This includes users who are rarely or never observed using the library or do not use library instruction. The DRRF depends on active participation by students and faculty, and the challenge in these cases is for librarians to acknowledge low levels of usage, particularly for departments that do not have a strong research component.
It is one thing for the librarians to collect and analyze the data and to complete the Departmental Resources Review Form, but unless the information is tailored and presented to library management for action, the analysis is of little use. As valuable as it is to collect all the data, it is not as important to present the entire report to the library collection managers. While more in-depth data such as circulation and in-house use statistics will be of interest to subject specialty librarians, the library managers and academic department heads may not be interested in that level of detail about a collection. For them the final recommendations are all they really need to know, and those recommendations may not even have to be more than one paragraph long. This is not to discount the work that goes into gathering data. It is useful for subject specialty librarians to know how much of the collection is getting use, and especially to compare usage counts over many years. However traditional data assessment looks at data as the end result, this model of assessment uses the data as a means to the end.
The group of librarians working on the beta project has been meeting every six weeks to compare progress and share experiences. One thing we have learned already is that the support mechanisms of regular meetings are invaluable in keeping the project on track. Even after the beta group has finished, there would be a permanent steering committee to help other librarians though the process of collection assessment.
Even after the evaluations are completed we may not be able to reverse downward trends in library use by our primary users, but it still is worthwhile to act on what we can control as opposed to doing nothing. The analysis of trends in library use by our patrons can help the library evolve its selection of materials and services that best serve the needs of our primary users. While this concept is nothing new, completing the assessment in a holistic manner – one that takes into account data and narrative from many sources gathered together in one place - provides a more comprehensive view of a collection.
The timing of this project could not be better since the University of St. Thomas is in the midst of undertaking a similar review process of its academic programs. The goal of the Program Review Process Design Committee is to establish a systematic review of courses in terms of meeting the goals of the program’s mission, evaluating resources to accomplish that mission, and assessing student outcomes (University of St. Thomas, 2003, 1) Libraries and academic programs alike are facing budget cuts, necessitating closer scrutiny of how money is used to maximize service to primary populations. The work of the Program Review Process Design Committee is evaluating programs based on the goals of the University of St. Thomas just as we are evaluating our resources in terms of meeting the needs of our curricula primary users. The UST librarians are not just doing assessment for assessment sake, but ensuring that our resources fulfill the university’s mission.
Association of Research Libraries. (2002). Service Trends in ARL Libraries, 1991 – 2001. Retrieved March 6, 2003, from http://www.arl.org/stats/arlstat/graphs/2001/2001t1.html.
Biblarz, Dora and Bosch, Stephen and Sugnet, Chris. (2001). Guide to Library User Needs Assessment for Integrated Information Resource Management and Collection Development. Maryland: The Scarecrow Press, Inc.
Grover, M.L. (1999). Large scale collection assessment. Collection Building, (18)2, 5-7.
University of St. Thomas. (2003). Introduction: Self-Study for the Higher learning Commission of the North Central Association [draft]. Retrieved March 11, 2002, from http://insideust.stthomas.edu/nca_selfstudy/Self-Study-Introduction.pdf.