The quality of any evaluation is dependent on many factors; of these considerations, one crucial factor that often affects the credibility and usefulness of an evaluation's findings is the criteria used. For example, when judging the merit and worth of a restaurant, you may base your assessment on the quality of its food, or you may determine its value based on the cost of its food. The criteria used to determine merit and worth will significantly affect your conclusions. You will inevitably arrive at very different recommendations based on the criteria you use and the importance you place on specific criteria.
Because of this, any credible evaluation will clearly articulate the criteria by which the evaluators based their evaluation findings. The expectation is that formal evaluations will use defensible criteria or clearly defined standards. Essentially, standards and criteria are the same things; the main difference is that standards represent a set of agreed-upon criteria an evaluator can use when conducting an evaluation. It should not surprise you to know that while evaluations judge the merit and worth of an evaluand, the focus of an evaluation might be an evaluation. We call the evaluation of an evaluation a meta-evaluation; and, we have standards for evaluating the quality of our evaluations.
A meta-evaluation is an evaluation of an evaluation
As the field of evaluation became a recognizable movement, no agreed-upon criteria existed to judge the quality of an evaluation. Many evaluators and consumers were concerned that the profession's credibility was at stake. Several evaluators drew considerable criticism from colleagues, clients, and stakeholders for their evaluation work. Standards were needed to establish the field of evaluation as a credible profession. In 1975, the Joint Committee on Standards for Educational Evaluation (JCSEE) was formed. Its mission was to develop and promote standards for conducting high-quality evaluations. The first set of program evaluation standards was published in 1981. They have been revised several times, most recently in 2011, and many evaluation communities worldwide have officially adopted these standards for their professional organizations.
|
The standards are arranged in five dimensions that describe criteria to judge the quality of an evaluation. In no particular order of importance, they include:
|
|
Utility – the degree to which consumers find the evaluation results useful.
|
|
|
Was the evaluation conducted competently and rigorously? Was the evaluator responsive to the needs of stakeholders? Were the results and recommendations beneficial? Was it completed in a timely manner?
|
|
Feasibility – the extent to which the methods and procedures were viable, effective, and efficient.
|
|
|
Did the evaluators use resources wisely? Did they balance the interests and needs of those interested in the evaluation? Were the methods and procedures reasonable?
|
|
Propriety – whether the evaluation was conducted fairly, legally, correctly, and justly.
|
|
|
Was the evaluation conducted ethically, honestly, and professionally? Were participants treated fairly and their rights respected? Did the evaluator demonstrate cultural competence?
|
|
Accuracy – the degree to which the evaluation findings, interpretations, and judgments were dependable and trustworthy.
|
|
|
Was the evaluation design and methods technically sound, well reasoned, and justified? Does the data support the conclusions and recommendations? Did the evaluators guard against misconceptions, biases, distortions, and errors?
|
|
Accountability – the degree to which the evaluation was documented adequately and focused on improvement and accountability.
|
|
|
Did the evaluators carefully reflect on the quality of the evaluation? Was a meta-evaluation conducted?
|
|
Joint Committee for Standards on Educational Evaluation (JCSEE)
|
Reference
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.