Class Presentation Notes:
Selected Bibliography on Measurement
and Evaluation
Administration of Information Resources -- LIS 391D.3 -- Fall 1998
This is a very large topic, so we will narrow quickly to measuring and evaluating in the arena of library and information science. Because my particular focus is in the municipal government area, your readings address government and I hope that some of our discussion will as well. Each of us is a taxpayer and feels government should be held accountable for the use of our money. We feel we own a part of the library and other services provided by government.
Measurement involves accountability to the taxpayer in the government setting and in the library setting involving the user of the library in counting up something.
Evaluation has to do with the use of the information and those uses can be to improve or take action of some sort.
Three components
define the goal
measure, and
report out to those who can act on the information
1. Measure--the act of determining the extent, dimension, or quantity of something
The concepts of measurement validity and measurement reliability come from recognition that measurement is inexact. Validity captures the notion of to what extent the measure gets at what it was aiming to measure.
Reliability is higher if repetitions result in the same measures time after time. There is a consistency of data measurement. There is a concept of "face validity" which is the appearance of validity which leads to the acceptance of the information because it seems reasonable or attractive.
Measurement is not an end in itself--rather, it is a systematic means for getting where you want to go.
Measurement can be dangerous--as the AAS article from last week tells us and as standardized test scores tell us. Factors that influence cognitive skills such as self-esteem many be areas of more importance than we have recognized in the past, but measuring these softer areas is difficult.
Saracevic and Paul B. Kantor have expressed the view that value of information can be measured. They believe it is important to study value because the social role of information is changing and information is assuming a larger role in every aspect of life. They also feel that libraries and information centers are shifting from a "just in case" model of collection completeness to the "just in time" model of providing access to information resources located anywhere at all. They also sense a heightened need for justification and valuation. They define value by distinguishing the shades of value and then determining that what users have to say about value is most important.
Kantor and Saracevic feel that the people's desire and comment is so important that an entire taxonomy should be based on user assessments. Their model is about the qualities of interactive with the service and the worth or benefits of the results of the interaction as related to the reasons for using the service--all from the user viewpoint.
Economists talk about value in exchange or value in use. Saracevic and Kantor are advocating a 3 step process which begins with the Taxonomy and user assessment, moves to pragmatic use of the Taxonomy, and a connection to economic indicators.
In this age of increased demand for service and reduced desire to provide more taxes the decision makers are asking for results that can be demonstrated. These are sometimes called outcome measures or outcome effectiveness measures. The decision makers are not the only ones asking for results, the clients and taxpayer is asking for results also.
Baseline data consists of a collection of measurements about an activity or process and these data provide a context for understanding measurements about a work process.
Benchmarking involves observation and data collection for evaluation products, services, and work processes. It involves studying the best practices and operations of other libraries or organizations of a similar size, budget, or location. Benchmarking affords the librarian an opportunity to pinpoint and close performance gaps, learn new approaches, and assess performance. Internal benchmarking, competitive benchmarking, and functional benchmarking. All are ongoing rather than one shot measurements.
There are many models for measurement. There is the goal achievement model which ties goals to results. Means achievement model is another where the decision makers want to know which of the decisions contribute to success. Another model focuses on people employed in the organization and their happiness and satisfaction.
1977 and 1991: The Measurement and Evaluation of Library Services.
1988 and 1993: If You Want to Evaluate Your Library… From University of Illinois, Urbana Champaign.
What is evaluation?--Assessing the value; using the scientific method; decision-making and assessment of alternatives; and a component of management which helps allocate resources.
His view is that evaluation has a practical component and that it is useful in problem-solving and decision making activities. He views library evaluation as an effort to determine to what extent the library successfully fulfills the interface role of getting information resources to a community of users. He advocates that the evaluation should be a diagnostic tool and should determine how well a particular library performs in a given situation and to identify as many major sources of weakness, failure, and inefficiency as possible so that improvements can be made.
Lancaster and others have contributed to our evolving understanding of measurement and evaluation. Other names include Robert Orr who conducted systematic and meaningful evaluations in the health care arena and wrote an article in 1973 which is often cited called "Measuring the Goodness of Library Services: A General Framework for Considering Quantitative Measures."
Other names with impact: Peter Hernon, Charles McClure, Kim Cameron, and many others.
In the Ammons article from Public Administration Review, we are introduced to a number of ideas: 1) that government purports to use measurement techniques, but often the hard data shows that there is big talk and little action in measurement, 2) that measurement can be made interesting to the public and appeal to a sense of civic pride, and 3) standards. The article gives a brief history of standards in the public library arena indication that the long standing practice in libraries of forming and promulgating standards has now shifted. He traces the history as moving from quantitative statements defining such things as per capita expenditures to imprecise phrases such as "reasonably adequate service." From there, the pendulum swung again to standards stated in quantitative terms such as "minimum annual additions and replacements." By the late 1980s this gave way in the public library arena to processes such as the public library planning process. This brings us to Thomas Childers, Nancy Van House, Douglas Zweizig, and a host of others who have tackled evaluating library performance and the meaning of library effectiveness. Manuals as assistive devices such as the Tell It! Manual, the What's Good book, and the Public Library Effectiveness Study go into great detail. The approach being used today advocates that a universal national standard is not as useful as the approach that deals with the specific universe or population. And even within an institution, you will find that distinctions and differences for measurement exist. Different organizational participants will have different priorities; institutions of higher education that are very similar in size or location or community can have totally different domains of interest and profiles. Most indicators of effectiveness have to be specific to an organization. So, we have a vast array of articles and tools to help us with evaluation.
Advocate for developing bridges and a strong relationship between user centered or oriented evaluation and system centered or oriented evaluation. He dips into both areas. Tague-Sutcliffe has recently worked on deriving a measure for the evaluation of records or sources of information and in that process considered the ability of the sources of information to provide information to the use; the informativeness of the items. The Saracevic study focuses on the individual and the specific task or reason that user is trying to use the library service, the specific value in the context of the information/service's use, and he uses the language of the user.
Hatry has worked at the Urban Institute for sometime and this institution is close to serving as a clearinghouse of how public policy is being formed and public policy processes as we have in government. They have produced a number of milestone documents in past 30 years which have been used as texts and have educated our legislators and government officials about program evaluation and program analysis. Program analysis is a systematic examination of alternative ways to reach public objectives which takes into account future costs, effectiveness, and impacts of each alternative. Costs consist of the expenditures of money and resources needed to carry out a program and effectiveness is the extent to which a program meets specific goals or objectives. Harry Hatry has recently authored a book on customer surveys which is an easy to use step-by-step guide for managers. Hatry is a researcher who is skilled in descriptive statistical techniques and he is very conscious of the danger that can result, not to mention the waste of expenditures, from inappropriate use of popular tools. He is often quoted because he has the ability to conduct research, write so that people can understand research, and because he is in the position to travel and speak to large groups of managers and government workers. When Harry Hatry speaks or writes, you will be able to follow along with ease and because he loves libraries, he frequently uses libraries as an example in his work.
"A value premise should not be chosen arbitrarily: It must be relevant and significant in relation to the society in which we live. It can therefore, only be ascertained by an examination of what people actually desire."
Source: Myrdal, Gunnar. 1958. Value in Social Theory: A Selection of Essays and Methodology. London: Rutlege.
| Return to Table of Contents |
This page is
created and maintained by Sue Soy ssoy@ischool.utexas.edu
Last Updated 11/11/98
© Copyright 1996 Susan K. Soy
Please feel free
to copy and distribute freely for academic purposes with this notice and attribution.
All other rights reserved