Why Evaluate? Help me decide.

Many evaluation approaches have been developed. Those categorized as decision-orientated or utilization-focused approaches tend to work well in the field of instructional design. The instructional design process involves making a series of design decisions; evaluations are made to inform your design decisions and thus improve your designs.

The things we evaluate are called evaluands (or evaluees when evaluating people). In educational settings, we might evaluate people, programs, initiatives, policies, products, equipment, services, concepts, theories, or organizations. However, when we evaluate programs, services, and organizations, we evaluate the people who administer the program, provide the service, or run the organization. These people are what Stufflebeam (2014) calls – ghosts in the system. Regardless, the underlining purpose for conducting an evaluation is most often because we need to decide something about the evaluand (i.e., the thing we are evaluating). Rarely is an evaluation conducted just to obtain information. The information we collect is intended to facilitate our ability to make informed decisions.  

The decisions that need to be made can be varied. In formal evaluations, the reason for conducting an evaluation is explained in a purpose statement; this includes the decisions that need to be made and often the central evaluation questions that will inform the decision.

The motivation behind a decision-orientation question is a need to know what should be done. Decision-oriented questions describe an evaluation's purpose and justify the need for an evaluation. On the other hand, evaluation-oriented questions ask for an informed opinion or judgment. The answers to evaluation-oriented questions form the rationale for the recommendations provided. This usually includes considering the potential benefits and disadvantages of doing one thing instead of another.

Decision-oriented questions are usually somewhat dichotomous; they help us choose one course of action over another. Rarely are these questions "right vs. wrong" types of determinations. They often involve deciding between "good, better, and best" options. We are trying to decide how to proceed, which may mean deciding not to do anything at all.

On the other hand, evaluation questions are rarely dichotomous. They often describe the "degree to which" the evaluand meets a particular set of criteria and can be judged of value, merit, or worth. The need for a judgment may be implied or stated but will always be part of the question. Evaluation-oriented questions inform decisions by asking: what is the best, most valuable, most important, safest, satisfying, beneficial, rewarding, desirable, practical, helpful, ethical, moral, or smartest thing to do?

The answers to evaluation questions can inform a variety of decisions. Should we continue to develop/fund a product or initiative? Does the product need to be modified or completely redesigned? Should the product be discontinued (scrapped entirely)? Several decision-oriented and evaluation-oriented questions might be asked in one evaluation. Evaluation questions may ask: Which option would be best? Which features of this product are essential, and which are unimportant? Which designs are viable? Is this initiative working as intended? Are users satisfied with this product? To what degree can the producte be used, or the program be implemented as intended? To what degree was this training effective?

We assign merit based on specific criteria, making evaluation a very subjective process. People have different opinions because they value different things. Still, answering evaluation questions provides us with information that we need to make well-reasoned recommendations that are then used to inform decisions.

When conducting an evaluation, you will note that not all the questions we ask will require a decision to be made or a determination of merit and worth. Information queries are asked simply to get information, and descriptive research provides information about relationships, potential causes, and general conditions. Experimental research makes comparisons to establish statistically significant differences (i.e., differences that likely didn't happen due to chance). This information can be valuable to evaluators. In general, research-oriented questions are posed to better understand an issue or topic. Research is conducted to draw generalizable conclusions that add to the body of knowledge in a particular field; they do not, however, attempt to make evaluative judgments. Still, research and information queries can be beneficial and serve distinct purposes for evaluators, designers, and developers. Evaluation questions, information queries, and research questions can all be used to provide the information we need to make informed design decisions. 

A Basic Decision-Making Process.

Deciding involves choosing between a finite set of alternatives. Making an informed decision requires an analysis of the benefits and disadvantages of choosing one option over another. Making an informed decision involves making evaluative inquiries and collecting information.

A variety of decision-making models exist. To make an informed decision, you probably want to consider using a structured decision-making approach. To do this, you need to:

1. Define the problem and articulate the decisions that need to be made
2. List requirements and identify constraints.
3. Make a list of possible alternatives
4. Identify the information you will need to make an informed decision
5. Identify the criteria you will use to evaluate possible solutions/alternatives
6. Determine how important each criterion is
7. Evaluate each alternative (judge the merit, worth, and value)
8. Determine the best solution

Sometimes the decision process can be completed quickly (minutes or hours). However, for some decisions, a more comprehensive evaluation is required to make an informed decision. Depending on the scope of the evaluation, the information-gathering activities required could take several weeks or longer.  

Common Mistakes and Problems.

Sometimes we make good choices and other times not so much. Several common mistakes (biases) can affect the quality of a decision. Even knowing that they exist and that you may be vulnerable to them can help you make better decisions. We are all biased, but we can train ourselves to be less biased.

Many cognitive biases can affect decision-making. Here are a few mistakes (biases) that can cause poor decision-making.

Confirmation Bias. 

Confirmation bias means paying attention to evidence that confirms our personal beliefs while ignoring anything that doesn't. Confirmation bias causes us to resist novel (different) ideas and solutions. This can cause us to intentionally or unintentionally interpret or twist information to fit our preconceived notions. Because of confirmation bias, two individuals with different beliefs will often draw different conclusions from the same information. It can also cause us to seek out information that supports our existing views or preferred solutions.

Confirmation bias often happens unintentionally so it is something we need to be aware of when making decisions. To overcome a confirmation bias, seek out people and information that challenge your opinions even if you are confident that the evidence supports what you want to do. Attempt to fully understand opposing viewpoints and the evidence that supports a different solution. Consider alternative interpretations of the data you have. You should also consider the potential unintended consequences of each decision.

Availability Heuristic. 

The availability heuristic leads us to make decisions based on the first viable option or solution that comes to mind. This bias is reactionary and often emotional. Someone says something, and you act on your first inclination. If that inclination was to punch them in the nose, you might regret not taking time to consider other options. Impulse buying is another example of this bias. You identify a need (perceived or actual) and immediately go online and purchase the first solution (product) your search engine displays.

To overcome this bias, you need to be deliberate. Force yourself to consider alternative solutions. Delay deciding anything until the initial emotion of the situation has dissipated. Doing this will reduce the chances of experiencing buyer's (or decider's) remorse. Make sure you have all the information you need and that the evidence truly supports your making that decision. We tend to make better decisions when we pause, second-guess ourselves, and take time to obtain sufficient information to make an informed decision.

Survivorship Bias

The survivorship bias causes us to make decisions based on previous successes (or failures) without fully considering the context and circumstances that led to that success or failure. This bias causes us to resist change; it impedes us from adapting to shifting circumstances and conditions.  A common mantra you may hear from people with this bias is – this is the way we have always done it; it has worked perfectly well in the past; why would we want to give up (or change) something that works. Survivorship bias is often the result of fear – fear that change will be harmful or that we will be unable to make the changes that are needed. Remember, however, that fear of change may be warranted. Not all change is beneficial or needed. This bias restricts one's ability to make changes even when the evidence and circumstance warrant making a change.

Survivorship bias is mitigated by training yourself to be more introspective. Consider why you may be resisting a decision that requires change. Before deciding anything, consider the factors and conditions that resulted in previous successes or failures. Ask yourself whether the conditions that fostered those successes or failures still exist. Consider whether future success is likely given current trends and conditions. Force yourself to consider whether improvements could and should be made based on your evidence. Explore the possible benefits of making a change and the consequences of not doing so.

Anchoring bias and Priming

Anchoring bias causes us to fixate or rely heavily on the initial information we receive about a topic. When we reflect on that topic later, we interpret additional (possibly updated) information from the reference point of that anchor instead of seeing the new information objectively. This bias can happen regardless of whether the initial information is even related to the new information. When people are exposed to an idea or piece of information prior to making a decision, that idea or information influences their judgment and can skew one's judgment. Anchoring happens in these cases because the individual has been primed cognitively to consider the initial information as a baseline or reference.

What is black and white and read all over? This simple riddle is based on an anchoring bias. The individual is primed to think about colors. Then when asked to consider something that isn't a color (read vs. red), the individual has difficulty thinking about anything not associated with color. This riddle tends to work whether it is presented verbally or in writing.

Being aware of this bias and how vulnerable people are to this bias is important for evaluators and decision-makers. The best way to combat an anchoring bias is to slow down your decision-making process. Insist on taking time to consider the information carefully before deciding.

Halo Effects and Logical Associations Errors

First impressions are powerful. People have a hard time overcoming a first impression (positive or negative). We call it a halo effect when a first impression affects future decisions. A halo effect exists when your first impression (or experience) is negative and you still tend to think negatively about that person or product even when presented with contradictory information later.

Likewise, a logical association error can cause us to make poor decisions. If we like one thing about a person or product, we illogically assume we will like other characteristics of that person or product. Most product sales are not based on an objective evaluation of the product but rather on the charisma and likability of the salesman. Making a decision based on unrelated information is an example of a logical association error.

A healthy dose of skepticism can help counterbalance these biases. You should also consider evaluation criteria separately before making a holistic assessment. A person can excel in one area and fail in another. It is also possible for products to be beneficial for some and not for others. For this reason, it is crucial to conduct a negative case analysis when evaluating the effectiveness of a product or program.

Planning Fallacy

The planning fallacy describes our tendency to underestimate or overestimate costs and risks when we like or dislike a proposed solution. Providing or accepting incorrect information skews our assessment of the viability of taking a specific action. For example, we often underestimate the amount of time it will take to complete a task when deciding when to start an undesirable task; this is especially true if we must forgo a pleasant activity. A planning fallacy can happen even when our previous experience contradicts our estimations. The planning fallacy is problematic because we base our decision on inaccurate information.  In essence, we are fooling ourselves.

The best way to overcome this problem is to get accurate information. Get the information from reliable primary sources when possible. You will also need to be mindful about how you and others feel about a specific solution so you can gauge the accuracy of the information you are receiving.

Action Bias

Action bias describes a default tendency to favor action over inaction. We often feel the need to do something even when there's no compelling evidence to suggest that acting would lead to a better outcome. For example, sometimes updating a product produces no substantial benefit to users. The product functioned well enough as it was. But the product's producers feel compelled to include additional features, to do something. There may be an economic benefit to producing an "updated" product, but the action may not have been warranted based on functionality deficiencies and consumer demand (i.e., need).

The solution to this bias is to conduct a proper needs assessment and carefully considering the benefits of acting compared to the consequences of not acting.

Analysis Paralysis

Analysis paralysis is not a cognitive bias but rather a condition that stops us from making any decision at all. Paralysis occurs when Individuals or groups cannot make a decision because of the overwhelming amount of data available or the tendency to overanalyze the data. Working with large data sets can be a problem when we are unable to separate important data points from trivial information. This also can be a problem when the evaluator has a limited ability to organize and analyze the available data. For example, you may have thousands of open-ended survey responses to code and interpret with no one capable of or willing to conduct the analysis.

Overanalyzing is a condition that occurs when people get stuck reevaluating data for fear of making a poor decision. It also occurs when individuals go over the same data repeatedly, only to find additional questions rather than answers. One last reason for not being able to decide is because the decision-maker wants to eliminate all uncertainty, which is usually impossible.

Solving this problem can be a challenge because you need to permit yourself to make a mistake. Working with a representative sample of the available data can be a viable solution when working with big data sets. Setting parameters and limiting the time allocated to data analysis and discussion can also help.

Ambiguity Effect

The ambiguity effect is a cognitive bias that describes people's tendency to avoid ambiguity. This is a problem when a decision is avoided because the decision-makers feel they don't have all the information they need to decide. What can happen in these situations is our intolerance for uncertainty and ambiguity inclines us to select an option for which the probability of achieving a favorable outcome is known. In these cases, the decision-maker is not paralyzed; they simply tend to choose options where the probability of success is known over options where the chance of success is unclear.

This is a challenging bias to overcome. Some people are risk intolerant, while others are risk-takers. Neither of these personality characteristics is better than the other. Risk-takers often fail more often than they succeed, while individuals who avoid risk tend not to fail but may miss out on the rewards of taking a risk.  To overcome this bias, one must make sure the decision is reasoned and not unduly influenced by their tolerance for risk.

Base Rate Fallacy

A base rate fallacy is the tendency to place more importance on the exception than the norm. For example, 90% of people surveyed may feel a specific aspect of your course's design was tedious and irrelevant; however, the decision-maker with a base rate fallacy will place more importance on a few heartwarming anecdotal stories extolling the virtues of the design. This is a bias against objective statistics that may be hard to interpret, understand, or are unpopular.

The values of the decision-maker complicate this bias. For example, having an intuitionist-pluralist perspective leads a person to value individual impact, while having a utilitarian viewpoint favors solutions that promote the greater good. A base rate fallacy can be a logic vs. emotion challenge that is influenced by one's values. In addition, this fallacy can be based on the decision-maker's ability to understand statistics and whether they trust the statistics presented to them. When they do not understand or trust the statistics, they may lean towards something they can understand, personal experiences and individual stories.

Bandwagon Effect

The Bandwagon effect refers to the tendency of people to adopt certain behaviors or beliefs because lots of other people act or believe that way. This tendency can affect decision-making as well. For example, around the turn of the century (2000-2010), Educational Data Mining and Learning Analytics became a popular topic of discussion among those in the field of education. Many organizations decided they needed to use data analytics and data mining even though they had little idea what it was, how to do it, or in what way it might benefit them. It was a popular trend, and they needed to be seen as a leading-edge organization. It worked out for some and not so much for others.

Deciding to act on innovative ideas and practices is not necessarily bad. However, considerable time and effort can be spent implementing a potentially beneficial solution to little or no effect. In terms of decision-making, jumping on the bandwagon should be discouraged if the decision is reactionary or impulsive. Important decisions should be made carefully. They should be informed by evidence obtained through a careful evaluative thought process.