The following chapter presents research principles and ideas relevant to evaluation. You will not be surprised to find that people do not always share a common understanding of words and ideas. A shared understanding of terms is needed as many concepts are understood and used differently in various fields and disciplines. A brief discussion of a few crucial concepts is provided here as a foundation for understanding the evaluation principles, approaches, and methods presented later.  

Inquiry Methods & Paradigms

Research Paradigms are deeply held beliefs and assumptions that guide the way we do research. The reason people do research is to learn some truth about the world (i.e., make sense of reality). However, individual researchers often make their inquiries in very different ways. This is because they have different beliefs about the nature of reality and truth (ontology). They also hold different opinions about how best to gain understanding and obtain the truths they seek (epistemology). A person's ontological and epistemological perspectives form the research paradigms they hold. This affects how they explore the world, the information they seek (methodology), and what evidence they will trust when making decisions.

Ontology has to do with the nature of reality and truth. Three common ontological viewpoints include the belief that:

• there is one reality (absolute truth, one correct interpretation),
• there are multiple realities (individual truths, perceptions), or
• truth is relative (context-dependent, debatable, and interpreted).

Many theorists view realism (absolute truth) and relativism (contextual with multiple realities) as opposing extreme positions on an ontological continuum. They often consider the idea of multiple realities to be a subset of constructivism since both are personal interpretations of reality that will likely change after reflection and dialogue with others.

Relativists believe that a person's interpretation of reality is context-dependent. Individuals can perceive reality a certain way given a specific set of circumstances and have a completely different understanding of what is true in a different situation. Although different, the person's perception of reality would be accurate (true for that individual) in both situations. Therefore, truth is not absolute. In addition, two individuals can have very different interpretations of the same situation. These conflicting versions of the reality (what is true) are accurate for each person—they represent what each individual believes (how they feel), and that should be respected as their truth.

Realists accept that two conflicting views of reality can exist, but they consider multiple realities like they would competing hypotheses—an individual can believe two different things to be true, but the evidence will support one and not the other; both cannot be accurate; although, both may be inaccurate. For a realist, the goal of research is to obtain a correct and complete understanding of reality that thoroughly explains and can be used to predict outcomes and behaviors. 

In practice, realism (i.e., the belief that there is absolute truth) is prevalent in the hard sciences (e.g., physical sciences like biology, physics, chemistry). Relativism is more prominent in the soft sciences (e.g., social and political sciences). The physical sciences tend to have identifiable rules that govern things in the real world and can be used to predict behavior. However, the soft sciences suffer from the unpredictability of human beings (their personalities, dispositions, and free will). This is not to say there are no absolute truths in the social sciences, just that they may be difficult to identify and apply. Social contexts are complicated because people are complex and frequently unpredictable. In education, for example, we are often unable to control for confounding variables (e.g., agency), making the results difficult to generalize and often impossible to replicate. Social theories are often developed conceptually, under ideal conditions or controlled environments. In practice, classroom conditions are not always controllable, and students are unpredictable. A teacher may find that what worked today will not work tomorrow. Often the best researchers can do is identify promising best practices and hope for the best.

Epistemology deals with how knowledge is acquired and how we can come to know what we know. It describes how we find truth and understanding—how we make sense of the world we live in and our interactions with others. There are a few generally accepted ways an individual can come to know or believe something:

• Empiricism (Physical Senses),
• Rationalism (Cognition, Reason, Logic, Analysis),
• Intuition (Emotions, Feelings, Conscience), and
• Authoritarianism (experts sharing/communicating truths).

You will note that these ways of knowing are not exclusively or independently applied. We often empirically experience something through our physical senses and then use authoritarianism, reason, and intuition to build understanding. (see Piaget's schemas, accommodation and assimilation).

Figure 1: Methods Paradigm Foundations (source Kimmons, 2022)

Common Research Paradigms

There are three widely recognized research paradigms: positivism, pragmatism (or post-positivism), and constructivism. Other paradigms have been identified, but these tend to align with one of the three main paradigms and differ only in how they situate themselves within a specific research field. For example, a critical theory paradigm is a form of constructivism that situates its research in social justice issues and seeks to address political, social, and economic issues.

The Positivist Paradigm. Positivism describes an approach to doing research that relies heavily on the mathematical verification of observations. Positivists tend to hold a realist’s ontological perspective. They believe absolute truth exists and that reality can be understood objectively. They argue that perceived truth is not the actual truth regardless of how strongly a person believes their perspective is accurate. They rely primarily on empiricism as an epistemology but tend not to trust subjective interpretations of an observation preferring to use measurement to verify observations rationally using statistics. Fundamental positivism rejects the notion that intuition and personal values should be used to determine truth. They stick to what we can observe and measure, not what we think and feel. From a positivist viewpoint, the purpose of research is to establish generalizable cause and effect relationships. A few common ways positivists gather, process and analyze information include:

• Statistical Empiricism – this approach to knowing relies on mathematical verification of what we observe; it requires researchers to quantify observations so they can be analyzed statistically.
• Scientific method – the scientific method is based on rationalism and is used to test a theory or hypothesis in a structured, logical way. The scientific method is a deductive approach. Results of experimentation that use this process either support a proposed theory (understanding of reality) or not. It relies on replication.
• Authoritarianism and Scholarship – Learning from experts is acceptable in the positivist paradigm if the expert's knowledge was obtained using appropriate methods (e.g., see for example what works clearinghouse). Scholars gain an enhanced understanding of reality by reviewing research that, in their perspective, was of high quality.

Many of the methods used by positivists are quantitative.

• Experimentation – deductive hypothesis testing is a standard positivist method for building knowledge. There are many experimental forms, but the gold standard is a random controlled trial. From a positivist paradigm, it is best to verify results mathematically (i.e., statistical empiricism); thus, most experiments would include a measure of some dependent variable and measures of independent variables thought to influence the dependent variable.
• Correlational studies – establishing the existence of a correlation (relationship) between two variables is one of the first steps to determining causation.
• Structured data collection. – surveys, interviews, and observations that ask all participants identical questions. These bounded sets of questions are designed to produce accurate and reliable measures of a specific variable (e.g., behavior or attitude).
• Tests – assessment instruments designed to capture objective measures of knowledge and ability.

The Constructivist Paradigm. Unlike positivists, constructivists believe that reality (truth) cannot be determined statistically. Constructivism, or Interpretivism, is based on the idea that individuals must build (i.e., construct) knowledge personally and that knowledge is based on a personal interpretation of one's lived experience. This paradigm originates from a relativist ontology. It advances the idea that our understanding of reality (truth) does not exist as an absolute but is a personal perspective and is defined contextually by each individual independently. It involves an inductive process. However, fundamental constructivism rejects the idea that people can be totally objective. A person's effort to process information and gain understanding will always be subject to bias, personal values, goals, and preferences. While constructivists accept empirical evidence, they would not feel compelled to verify truths statistically. Instead, they would describe it qualitatively. They accept intuition and rationalism as valid ways of knowing and do not worry that individuals arrive at different conclusions. Multiple versions of reality (personal beliefs and feelings) are accurate for that individual and should be respected as their truth. The best way to establish a collective truth is through dialogue, negotiation, and persuasion. The goal of research isn't necessarily to predict outcomes and behavior but to understand them. Methods constructivist use often seem similar to positivist, but they are applied differently. Some common ways constructionists gather, process, and analyze information include:

• Scholarship – The type of scholarship used by constructivists is broader than that used by traditional positivists. It involves a reinterpretation of reality obtained by reviewing and synthesizing existing research, documents, records, and extant data.
• Authoritarianism – similar to positivism, this way of knowing involves learning from experts—those who have insights and understanding others do not. However, expert sources might include academic researchers, instructors, experienced practitioners, regular people, or God (information revealed by a superior being).

Many of the methods used by constructivists are qualitative.

• Naturalist Inquiry – is accomplished when a researcher uses a specific perspective or cultural context to observe, describe, and interpret the experiences and actions of people.
• Hermeneutic Phenomenology – an interpretation of people's lived experiences using the first-person point of view.
• Case studies– an in-depth study of a person, group, or event designed to explore every aspect of the subject's life in order to understand their behavior.
• Grounded theory–is an inductive process that gathers, synthesizes, analyzes, and conceptualizes qualitative observational data to develop theories (explanations of how the world works).
• Ethnography– involves a description of the customs of individual peoples and cultures. This often requires the researcher to completely immerse themselves in the culture and everyday life of the people the researcher is studying.

 The Pragmatist Paradigm. Pragmatism came about primarily as a result of the paradigm wars (or Quant-Qual wars) of the 1980s (Guba & Lincoln, 2005). Pragmatism, also referred to by some as post-positivism, began among scholars who argued that a mono-paradigmatic orientation of research was not adequate. Understanding (truth) could not be established entirely using statistics and random controlled trials, nor could it be altogether revealed using subjective descriptions of an individual's perceptions. The pragmatist paradigm promotes a practical pluralistic approach to research that uses the most appropriate methods for studying the phenomenon. This gave rise to a belief that advocates for mixed methods and an appreciation for the value-laden complexity of reality (axiology). Pragmatists use a variety of research methods depending on the purpose of the research and the questions being asked. However, their perspective of what constitutes quality research is not based on using a particular method but on the appropriateness, credibility, and trustworthiness of the method's application. In addition to any of the other previously mentioned methods used to gather, process, and analyze information, pragmatists might also use:

• Action Research – Action research is not a method but rather an approach. It differs from other research in that there is less concern for the universality of findings and more concerned about whether the findings can be used to solve a particular problem or answer a specific question. When a problem is identified, an action research protocol would consider various solutions and test the most promising to see if it solved the problem at hand.

Table 1: Common Paradigms

Paradigm	Typical Stance
Positivism	Absolute truth exists. Reality is objective, knowable, and generalizable. Truth is discovered by experimenting and testing theories (i.e., deductive, scientific, and statistically verifiable).
Pragmatism	Some absolute truths exist (e.g., in the physical sciences); others are individually or collectively perceived (e.g., social sciences). Not all explanations of reality can be understood perfectly or mathematically. The best way of knowing will depend on the question being asked (methodological pluralism, mixed methods).
Constructionism	Truth is context-dependent and varies (personal perceptions of truths, multiple realities). Collective reality (truth) is ever-changing, socially constructed, and represents a negotiated consensus. The best way of knowing involves qualitative interpretations of people's lived experiences (i.e., their perceptions, values, behaviors, and beliefs).

Quantitative and Qualitative data

Research is often described as either quantitative or qualitative. When people say this, they are describing the type of data being collected.

Quantitative research collects information that can be represented as a numerical value and analyzed statistically. Quantitative data (i.e., numbers) describes some attribute of an object or person (i.e., amount or magnitude). These numbers (i.e., measures) are used to communicate and compare qualities a researcher is interested in studying. For example, we could say someone is tall; this is a qualitative description of a personal attribute. We quantify this attribute by measuring the person's height (i.e., assign a precise numerical value that represents a person's height). This can then be used to communicate the precise height of an individual or compare the height of one person with another.

Many things can be quantified, but not all information can be easily reduced to a number. Information that cannot be quantified is referred to as qualitative data. Qualatative data is non-numerical information. It cannot be analyzed statistically—it is described, categorized, and interpreted. Examples of qualitative data include personal accounts, photographs, documents, and video recordings. Qualitative information may also represent a person's emotional state, reasoning, or beliefs. A particular attribute's frequency and magnitude (e.g., strength) can be quantified but not the various ideas they represent. For example, a researcher may ask people to explain why they behaved a certain way. A given reason cannot be quantified (e.g., I was angry, afraid, interested, determined, concerned, or felt responsible). These are nominal data points (i.e., labels without order or incremental value). Assigning a number to a nominal data point would have no meaning. However, the frequency of a reason being given can be calculated, and the magnitude (i.e., strength) of a particular feeling represented by a given reason can be quantified (e.g., how angry were you?).

Objective and Subjective interpretations

Positivists tend to value quantitative data because it is objective and can be statistically analyzed. While constructivists often collect qualitative data which needs to be interpreted subjectively (i.e., using values-based criteria). Researchers with a positivist paradigm may collect qualitative data but attempt to quantify it for analysis.

Many people mistakenly assume that objective analysis is better than subjective analysis. However, both are valid and often necessary ways of interpreting data. To be objective simply means that people tend to agree on the meaning of the data. A number has a specific meaning because people agree on that meaning. Using numbers to communicate can be more effective in conveying a precise meaning. For example, you may find that "several" people indicated they held a specific belief, but saying it in this way (i.e., qualitatively) does not communicate that finding very well. However, to say 43% of participants held a specific opinion (e.g., agreed or strongly agreed) would be a more precise (i.e., objective) way to communicate this finding. Still, people do not always agree on the interpretation of research findings. For example, a p-value is a number that represents the probability (i.e., likelihood) that an observed phenomenon happened by chance. A p-value is a quantitative piece of information. It is objective because people tend to agree on its meaning. However, a researcher must interpret a p-value to determine its statistical and practical significance, which requires a subjective analysis. Statistical significance requires we set an alpha value, usually .05 but not always; it depends on the context of the research. Even when a statistically significant result is determined (i.e., the p-value is less than the alpha value), the researcher must subjectively consider the finding's practical significance. This may be facilitated by calculating an effect size, but the practical significance of a finding is most often a subjective determination. For example, an evaluator might calculate the change in attendance rates attributed to a specific intervention. The resulting difference may be statistically significant (i.e., not likely to have happened by chance). The effect size may be adequate (i.e., accounts for a fair amount of the variance, and the difference in standard deviation units is acceptable). Still, those running the program may believe the magnitude of the change has no practical value (e.g., a statistically significant increase in attendance may represent an insignificant number of students).

Inductive and Deductive Reasoning

Inductive reasoning, or induction, involves making an inference based on observation. Induction is often used to develop theory. Deductive reasoning, or deduction, involves making inferences based on widely-accepted facts or theories (i.e., premises). Deduction is often used to test theories. Both these forms of reasoning involve logic, reflection, and interpretation. Both are used in evaluation and are aspects of evaluative thinking. Evaluators use an inductive process to describe situations and make recommendations. They might use a deductive process to judge the effectiveness and impact of an initiative. 

Figure 2: Theory Development using Inductive and Deductive Logic

Axiology

Axiology is the study of values and value judgments. Axiology is concerned with issues of ethics, morality, and aesthetics (personal preference and satisfaction), as well as other forms of value. Axiology is vital for evaluators because our values affect our judgments of what has merit, worth, or is valuable. The value (merit or worth) assigned by individuals to an object will differ depending on their values (morals, preferences, interests, goals, ethics).

The distinction between intrinsic and extrinsic value is central to axiology. Things with intrinsic value are valuable in and of themselves (e.g.,  they are life-sustaining or essential to our well-being, see Mazlow). Objects and people have extrinsic value because they are useful or desirable to someone for some reason. For evaluators, understanding our own values helps us to make good evaluations; understanding what others value helps us understand the decisions they make.

Personal benefit and the greater good. One particular axiological issue evaluators must consider is the tension between valuing what is best for the greater good and what is best for an individual (i.e., personal interest and benefit). For example, results from a negative-case analysis might find that most individuals participating in a program achieve the initiative's intended goals (i.e., they benefit from participating in the program). However, this may not be the case for all participants. This becomes a problem when the only way to increase the benefit for the few would mean diminishing the benefit for the many.  

Evaluators are encouraged to “strive to contribute to the common good and advancement of an equitable and just society” (AEA guiding principles, 2018). However, it would be incorrect to believe that there is one overriding value or belief that describes society's best interest (Davies, 2021; Patton, 1985; Schwandt, 2002). Thus, the issue remains regarding what constitutes promoting the public good—are we to do what is best for the majority, do what is best for individuals, or determine a compromise between the two.

References

American Evaluation Association. (2018). AEA-American Evaluation Association: Guiding Principles for Evaluators.[online] Eval. org. https://edtechbooks.org/-pSNL

Davies, R. (2021). Establishing and Developing Professional Evaluator Dispositions. Canadian Journal of Program Evaluation, 35 (3).

Guba, E. G., & Lincoln, Y. S. (2005). Paradigmatic Controversies, Contradictions, and Emerging Confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research (pp. 191–215). Sage Publications Ltd.

Patton, M. Q. ( 1985 ). Cross-cultural nongeneralizations. In M. Q. Patton (Ed.), Culture and evaluation: New directions for program evaluation (vol. 25 ). Jossey-Bass.

Schwandt, T. A. ( 2002 ). Evaluation practice reconsidered. Peter Lang.

An alternative resource for research basics can be found at Educational Research.