Scott Schuetze, Ex Libris
At a time when research libraries are competing with Google for users’ attention, an effective web-scale discovery service can improve the use of your library collections and bolster your library’s role as critical to the research process. When students, faculty, and staff have a good experience using your discovery service—if it’s simple to use, delivers fast and relevant results, and provides accurate linking —they are more likely to use it again and again, making your library an integral part of the research process.
Evaluating your discovery service is important, because it allows you to make sure this service is meeting the needs of users successfully. It can also help you demonstrate the value of your library to your institution.
Many research libraries have completed successful evaluations of their discovery services using a variety of approaches. 
The Challenge
But evaluating your discovery service can be challenging. For instance, data must be collected from different sources, and it can be hard to tell whether the use of your library content stems from your discovery service or from another source. Also, while many papers and presentations have focused on resource discovery, there is no consensus on how libraries should evaluate the use of their discovery services or measure their effectiveness.
Despite these challenges, many research libraries have completed successful evaluations of their discovery services using a variety of approaches. A review of their methods reveals key insights that can help you evaluate the effectiveness of your own discovery service. Here are three suggestions.
Evaluating Discovery Services: Three Important Tips
- Use the resources you have available for quantitative analysis.
Quantitative data available to you can include figures such as the number of searches, page views, session logins or requests, full-text downloads, and link resolver usage. These figures are typically available through the extensive reporting and analytic capabilities that discovery service providers offer as part of their solution. What’s more, your library can compare this information to journal, e-book, and database usage, as determined through COUNTER reports, to see if there’s a correlation.
- Look beyond the number of search queries.
Search query data can tell you how frequently students and faculty are using your discovery service, which is a valuable statistic in its own right. But this data point doesn’t tell the whole story. You also want to determine how easily users are able to find the resources they’re looking for—and how relevant these search results are to their information needs. For instance: Do searches direct users to the very best resources from all publishers and databases you subscribe to, or are search results weighted more heavily in favor of certain resources over others?
- Consider qualitative analysis.
Qualitative data can come from user surveys and usability studies. While they are often more time-consuming and labor intensive, these measures can help you evaluate the behavior of your library users, identify their needs, and figure out how well your discovery service is meeting those needs.
Evaluating your discovery service on an ongoing basis helps you make sure you’re fulfilling your users’ expectations for fast, reliable access to information, which will position your library for future success.
August 16, 2018
