Every field of expertise has elements that need to be investigated, and regardless of the discipline, how you discover evidence for learning is very similar. In this chapter I will present common issues I have encountered in the field of instructional design when planning a research study. This is followed by discussion of when and how to use certain research methodologies. Finally, I will discuss how to conduct research and report results.

When and How to Use Research Methodologies

Novice researchers will often limit themselves as only being a qualitative or quantitative researcher. In practice, the nature of the question will dictate the most appropriate research method that should be used. In addition, using mixed methods often yields stronger results. For example, survey teachers about what to cut or reduce in a curriculum, then follow up with focus groups to yield stories and experiences that explain the survey numbers.

When to Use Qualitative and Quantitative Methods

Qualitative methods generally seek to answer questions that center around experiences that involve understanding values, beliefs, perceptions, emotions, culture, growth, paradigm shifts, processes, taboos, morality, reasoning, and acquiring learning. In short, the more a question centers around “the human experience,” the more it lends itself to qualitative means which gather rich experiences, stories, and examples. An example of qualitative research could be using grounded theory to develop a new theory of language acquisition by analyzing data from journals, portfolios, and focus group interviews of persons learning a new language.

Quantitative methods generally seek to answer questions that center around how much of something has been acquired such as knowledge, behaviors, or attitudes. Experiments, correlations, causes, and descriptions are often the reasons why how much of something is being measured. An example of experimental research is when a pilot program is compared with an existing instructional program to measure the effect of different instructional treatments on how much students learn. An example of correlational research would be the effect of parental involvement on GPA. The hypothesis of such a study may be that higher parental involvement will correlate with higher GPA. An example of a causal research study is the effect of dogmatic political preferences on macro-moral reasoning abilities. Perhaps the assumption is that dogmatic political party views may inhibit one’s ability to morally think through all aspects of social issues. An example of descriptive research is when a researcher describes how many hours of homework 7th grade math teachers give per week.

General Steps to Qualitative Research

Identify Your Bias

The first step in qualitative research is to identify your biases via reflective means. Reflexivity activities help a person go through a meta-cognitive process of identifying their preconceived notions, judgements, and values that may influence the research project. This can be as simple as journaling your thoughts about a research topic. This video clip explains some qualitative research methods with a few examples. There is a great description of reflexivity about eight minutes into the clip.

Plan the Research

The second step begins by writing a problem statement and research question. Once these and variables of interest are identified, an important part of the research plan is to design interview or observation guides. This may involve designing interview or observation guides. Guides take various forms such as a checklist, a standardized open-ended interview guide, or an informal guide. A few tips in designing the guide are to a) have colleagues review questions or guide, b) pilot test your questions or guide, c) check for leading questions or leading interviewers or observation bias, and d) be open to scrutiny.

During this phase, plan reliability and validity measures are

1. Credibility (internal validity). Credibility of the research can be established via triangulation, prolonged contact, member checks of data or analysis, saturation, reflexivity, and peer review.
2. Transferability (external validity). The generalizability of the research can be enhanced by using thick descriptions and variation in participant selection.
3. Dependability (reliability) is established by extensive audit trails and using triangulation. Subjectivity audits are used during the data collection process to evaluate how a researcher’s presence, questions, or biases may be impacting the research.
4. Confirmability is established with reflexivity and intra- or inter-coder reliability.

Gain Entry

The third step is to gain entry with the group or individuals the researcher is studying. This involves gaining trust, understanding the culture and environment, and helping the participants feel at ease. The amount of time and effort here depends on many factors such as the sensitivity of the topic and whether the researcher is taking a cultural native approach or acting in an apparent authority role. To further explore this concept of gaining entry, read this article.

Collect Data

The fourth step is to collect data. This can be done via observation, interviews, focus groups, open-ended surveys, and journals. During this phase, be prepared to adjust your thinking and be aware of your bias. Gather existing data and artifacts also. Because this is the step where the most mistakes are made, below are some additional practical tips and tools for qualitative data collection.

Observation

The most common ways to observe are live, virtual, recorded video or audio files, or a combination of the three. A rubric or observation guide can be designed to focus observation on certain concepts or phenomena. The benefit to live observations is the ability to view holistically all the nuances, expressions, non-verbal cues, and visual or auditory expressions that are not captured by other methods.

The benefit of recording video or audio files of observations is the ability to review them multiple times during transcription and analysis. One great tool for video observation is GOReact, where a pre-specified codebook can tag live or pre-recorded video files when coded phenomena occur.

Focus Groups

A focus group is one of the most common methods of qualitative research. Focus groups are group interviews with less than 10 persons. These persons are usually intentionally selected (e.g. students who meet certain demographic characteristics). Focus groups work well when participants feel non-threatened and permissive. They are effective because the interplay between participants can stimulate other’s thoughts, beliefs, perceptions, etc.

Because of the ease of gathering these small groups, some researchers mistakenly use focus groups when other methods would be more appropriate. For example, novice researchers may gather a focus group and proceed to ask the group survey-like questions rather than gather rich descriptive experiences, stories, and thoughts. Another common mistake is that researchers default to focus groups for topics best answered by individual interviews; for example, asking an online cohort about their group cohesion in a focus group setting presents a situation where participants may give socially desirable responses (individual, confidential interviews may yield more truthful responses).

Common Mistakes in Focus Groups: Communication Errors

The most common mistake for focus groups (and individual interviews) are communication errors from the interviewer which hinders responses such as:

• Restating too often. In teaching situations and normal friendly discussion, restating is a great communication skill. However, in interviewing, it’s not always a great method, especially with a novice researcher who can’t tell when they are putting words in participants' mouths. For example, if you are interviewing a group to find out opinions on a particular LMS platform and they say, “It feels so isolating,” a great follow-up to get more out that concept is to ask, “Can you tell me more about that?” or “Can you think of a story or experience that illustrates that isolated feeling?” In a focus group, it is best to seek clarification from the participants with additional questions, stories, experiences, or elaboration rather than assume you know what they mean.
• Leading the witness. Some interviewers inadvertently revert to their bias and ask leading questions.
• Overly emotional responses from the interviewer. Remember, you are an interviewer with a beating heart, but not a counselor or therapist. You don’t have to validate or comment on everything or nervously giggle at comments when humor is not intended.
• Not qualified to discuss. Interviewing about content for which you are not qualified (e.g. impacts of mental illness, causes of anxiety, marriage counseling issues, or any other topics that most often involve a trained professional).

Common Mistakes in Focus Groups: Failure to Control for Groupthink

Groupthink is when one person in the group shares an opinion or thought and the group finds it hard to break away into any other vein or divergent thinking. Sometimes, there is an actual group consensus. Other times, it’s groupthink. Discerning between the two takes experience. Some common indicators of groupthink are that a few people are not talking or give trite agreements (“what she said”). Discern if the persons not talking or tritely agreeing are just less articulate, or don’t want to share their contrasting views. You may have to curb bold or strongly opinionated persons who may seek to bully or monopolize the group. For example, “Let’s hear from someone who hasn’t shared yet” or “Hold your thought for a minute while we hear from…”.

To control for groupthink, first introduce the session with some ground rules for an open discussion (and repeat many of these introductory rules throughout):

• There are no right or wrong answers.
• In this group, it’s okay to disagree and still be friends. If everyone is saying their experience was great, but for you it was horrible, you need to speak up so all perspectives can be heard.
• It’s safe to be positive or negative, better or worse. Explain to participants that saying something positive does not make them a better teacher or student. Similarly, saying something negative does not make them a worse teacher or student.
• It’s okay to remove the filter between the brain and tongue, just for 45-60 minutes in this focus group. “Don’t worry if you are saying it right or wrong, just say what you experienced.”
• Distance yourself as the researcher from the product. For example, “I didn’t create this company or this training so however you feel about it won’t affect me.”

To control for groupthink during a focus group, use the following suggestions:

1. Have participants write out thoughts or stories before the interview begins. Refer back to these in the interview if groupthink emerges. Did anyone write something different?
2. Invite contrast throughout the interview by asking, “Did anyone have a different experience that I need to hear?” or “Have you or someone you know had a different experience?”
3. If the groupthink is very strong, use hypotheticals, such as “What type of person would have had a different experience?” or “Could someone have a different experience? Why or why not?” Follow up by asking, “Did any of you experience that at all?”
4. Employ indirect questioning. Indirect questioning seeks to control for socially desirable response sets. For example, if you were asking about a program designed to remove racial biases, there may be a strong socially desirable response set. To remove the social context, you could ask, “If you were to describe how well this program runs using an analogy of a car, what kind of car would it be? Why?” or “If this program were a TV show, what genre would it be? Why?” With indirect questioning, the goal is to remove persons from a contextual response set by asking about the program indirectly. This can control the parroting response set by making respondents think and respond in a new context about similar issues, impressions, or values, without having time to prepare a socially desirable response. The result is insights with some contextual limitations.
5. In conclusion, ask, “What do you want to make sure I heard from you?” (I often have participants to write this down). Or ask, “is there anything I should have asked but didn’t?”

Ethnography

Ethnography is a method of study that educational researchers adopted from anthropology. For example, a researcher may be a participant or observer in an online class and may conduct several interviews and focus groups. They may write about the setting, social implications, typical and best behavior, and seek a perspective from several groups (parents, teachers, students, administrators, etc.). The researcher could also report about the tensions between groups, or even explore power struggles within groups.

Analyze the Data

The fifth step is to analyze the data. After qualitative data is collected and cleaned (such as removing identifying names or using codes for participants instead of names), it is important to go back to your original questions and study plan. Although you can adapt and explore interesting concepts that emerge, it is also important to keep your focus on your original questions. Thoroughly explore the data and be open to new concepts, but do not be sidetracked by all of them. Word maps may help initially to see what phrases or concepts are prevalent.

In qualitative analysis, all artifacts are loaded into a software such as Delve (https://delvetool.com/), Quirkos (https://edtechbooks.org/-hVz), Dedoose (https://www.dedoose.com/), or MAXQDA (https://www.maxqda.com/) . The researcher then creates a codebook. A codebook is a list of concepts, behaviors, actions, thoughts, etc. that summarize themes in a group of qualitative data (such as responses to open-ended questions). My initial code book will also contain the original research questions. If you are working with a group of researchers you should make sure you all agree on the codebook and use it consistently when analyzing your data (often called “interrater reliability”—see this training for more information).

Common Mistakes in Analyzing Qualitative Data

• Failure to focus the codebook on the research questions.
• Failure to identify proximal relationships (e.g. math division, anxiety, and home support are all close together). For example, if less anxiety and parental involvement are often near each other in participant comments there may be a connection there.
• Myopic analysis—overly focusing on a powerful story that is not a generalizable trend.
• Failure to make your findings defensible by employing validity and reliability measures discussed in the second step (credibility, transferability, dependability, confirmability).

Report Findings

The final step is to report the findings. A good report involves the research question or problem statement; background, theory, and lit review; the study design; presentation of the data with rich descriptions; and an explanation of the data or findings. You can present findings in chronological order, or by theme, frequency, or rich narratives such as in a case study. To learn more, see this lecture: https://edtechbooks.org/-FNEs.

Common Mistake in Reporting Qualitative Data: Extrapolation Error

A common mistake is a tendency to want to extrapolate an ancillary finding into a generalizable trend. For example, a researcher may say, “I was talking to a student at lunch today and they mentioned… This is something I’ve heard many times.” When a researcher makes a generalizable error, they are often including a finding that was never asked for, planned to explore, or agreed upon. The anecdotes are often not related to the initial research purposes.

The danger of making such anecdotes into general findings is, in reality, it was only a few powerful stories from a few isolated interviews. These powerful stories should not be ignored, and can be explored further, but one should ask if the anecdotes alone are sufficient to represent a generalizable trend. For example, in one study, an educational administrator in Africa asked for a budget to buy a gun to scare the lions away from the school. Although this was an impressive story, it was not an administrative issue mentioned in any other program. However, all programs mentioned power or internet outages and lack of technological resources.

In summary, researchers who have planned, collected, analyzed, coded, and reported data from qualitative research understand that the benefits are rich descriptions and understanding concepts or phenomena in depth and context. However, a drawback is the significant time and resources spent in transcribing, reading, organizing, coding, analyzing, and reporting. In addition, the amount of information gathered is usually very focused and can be limited in scope. For this reason, many chose to use quantitative methods.

General Steps to Quantitative Methods

Quantitative research begins by developing an understanding of an instructional problem and possible theories to solve the problem. The key elements of this step involve knowing much of the background of a product or problem and identifying a client’s needs.

Write a Problem Statement and Research Question

The result of a good review of literature is to be able to write a good problem statement. A well-written problem statement will bring up past research or needs that lead to an instructional design question, then lead naturally into defining the scope of the questions that will be answered in the study. Here is an example of a problem statement for teachers that are simultaneously learning a foreign language and teaching skills:

Research shows that when second language learners seek to achieve multiple learning objectives within a second language, the acquisition of those multiple learning objectives may be impeded due to increased cognitive load and learning anxiety.

The problem statement is general enough to be read by top administrators and specific enough to narrow down the project scope. The research question builds on the problem statement to define the specific questions of the study plan. Here is a sample research question from the previous example:

Does separating instruction of teaching skill training from language acquisition training affect student’s: a) teaching skills, b) language acquisition, and c) foreign language learning anxiety when three pilot groups are compared with three control groups and baseline historical data?

Another approach to writing a research question is to operationalize one’s theories into a hypothesis. A hypothesis usually involves an if–then statement and defines the variables of interest. For example, if I use metacognitive strategies in my reading curriculum, then students will more efficiently learn to read at a 5th grade level. It is quite easy to turn a well-written hypothesis into a research question. For example, if I use metacognitive strategies in my reading curriculum, will students more efficiently learn to read at a 5th grade level? In this phase, there will be some operationalization of terms here such as efficiently, 5th grade level, and identifying some specific metacognitive strategies.

Develop a Study Plan

Once a research question is designed and variables are operationalized, it’s time to develop a research plan. A good research plan builds on the problem to be solved and further operationalizes variables to be studied and controlled for.

The Study Plan Matrix

At this point, the client and instructional designer can create a study plan matrix as shown in Table 1 (based on the prior example of language acquisition):

Table 1

Sample Spreadsheet

The study plan matrix operationalizes the study in a clear way. Instruments from the literature review are specifically identified. Analysis methods are clearly spelled out. In addition, the study plan will involve agreed-upon methods to control for extraneous variables and employ accepted reliability and validity measures to control for threats to validity. A good study plan can also control for scope-creep—when a research project is either ill-defined or a client attempts to pork barrel the project so as to make it much larger than it should be or was originally agreed upon. The study plan can be the basis for a business requirement document—a document that spells out timelines, cost, and deliverables for a client.

Data Collection

Sampling

Sampling is how one determines the selection of participants in a research study. Sampling methods result in a selection of a subset of a population. Random sampling methods (everyone in a population has an equal chance of being selected) aid in the ability to generalize one’s sample to be representative of an entire population. For example, all fourth graders in the district are put into a random sample generator and 400 of them are randomly selected to be representative of all 4th graders in the district. Non-random sampling is when decisions such as researcher judgment or convenience determine one’s sample. For example, selecting all fourth graders at the particular school at which one works. Differing methods have benefits and drawbacks but the ultimate goal in all sampling methods is to seek a representative sample of a population. For a simple explanation of types of sampling and their advantages and disadvantages see: https://edtechbooks.org/-hcm.

Experimental Research

The simplest type of experimental research is a single treatment and a single observation. Various designs seek to control for threats to validity (to learn more about how each design controls for threats to validity see https://edtechbooks.org/-Woh). The following chart shows various types of experimental designs.

Table 2

Experimental Designs (R = random selection, X = experimental treatment, O = observation)

Survey Research

Surveys are a very common method of data collection and a great tool to use when the question you are seeking to answer can be easily responded to in categorical selections or written comments. For example, asking about the frequency of a known behavior, how well students like a method of instruction, how well they agree or disagree with statements about an instructional treatment, or conscious perceptions potential learners have about their learning environment or teacher. Essentially, questions about what or how much of something are great candidates for survey research. For example, how much do you agree or disagree with the following statement(s) about your instruction, or, how often did your teacher follow up on your homework? An open-ended item might be as simple as: “Explain your rating.” Because surveys are so common, I will offer greater depth on this method of data collection.

General Survey Writing Tips

The following tips will help you design a better survey:

Pilot test. Test your survey out with 5-10 typical responders. Use a think-aloud protocol, where you ask people to say out loud what they are thinking when completing the survey. Conducting a pilot test will identify a majority of any usability or misinterpretation issues you will need to fix with your survey. In addition, asking 30 respondents to reply before sending it to all 1000 can give you a good idea of how your categories are performing. You may find a ceiling or floor effect for several items (all participants are selecting the highest or lowest category). This may be grounds for removing an item or revising it to be more discriminatory. However, it may also serve as a confirmatory item.

Review sample survey output files. If you respond to a few surveys and then review the output file, it can save you hours in data clean up later because you see how you can most effectively change the format for later analysis. For example, perhaps the output file has data from the same respondent on different rows. Or perhaps you notice where survey logic accidently skipped over a section that was not intended to be missed. Sometimes embedded data is not being properly gathered. Categorical responses could be in text format rather than numeric. In addition, you can test the import of your selected output file into your statistical software of choice. Part of that import may help to determine if the data is appropriate for your study plan. For example, linear regression may not be appropriate for nominal data. Nominal logistic regression may be a more appropriate analysis method (for a further discussion of regression, see: https://edtechbooks.org/-nWXt). Sometimes seeing the data can help inform whether your method of analysis is appropriate.

Do not gather what you do not need. Do not waste valuable survey response time in gathering already existing information. For example, many organizations, conferences, or workshops have participant information already gathered. If the organization has the appropriate data-sharing agreements in place, you can embed prior gathered demographic information and only verify its accuracy.

There are various types of items. Most survey software will offer multiple choice (a, b, c, d), Likert scales (1, 2, 3, 4, 5), semantic differential items (agree–disagree), ranking and ordering (place the highest on top), dichotomous (true–false), and open-ended items. For a discussion of when to use these differing items see https://edtechbooks.org/-yUXF. For an item writing workshop or lecture from the author, see: https://edtechbooks.org/-vgvS.

Establish criteria to measure and align your items to each criteria. For example, if you are using a survey for an implementation study, establish what teachers and students must do for a successful implementation. Then write items for each criteria identified. I use a spreadsheet for this. Here’s a simple sample of criteria and items in an implementation survey design.

Table 3

Sample Survey Criteria

Avoid leading items. For example, “How easy was it to use the app?” is a leading item because this assumes it was easy to use. A better way to word the item would be, “Rate your experience with the app” (then use categories like 1 = easy to use, 5 = difficult to use, etc.).

Avoid double-barreled items. Double barreled items ask about multiple elements. For example: Rate your experience regarding class preparation and knowledge of due dates. It is better to separate these into two distinct items.

Use a common stem to avoid repetition. When several items or options begin with the same words, use a single stem and put the items or options beneath it or in a survey matrix. Here’s an example of a stem with a multiple choice item

What determines a person’s eye color? Their parent’s genetic…

a. centrioles

b. chromosomes

c. organelles

Use the right categories. There is a tendency to default to the categories your software provides, but the software may not always give you the right options. For example, the default may be a 1-5 scale but a 1-10 scale may be more appropriate based on participant responses. Or the software may give a numerical scale or a preset categorical scale, but a unique categorical scale should be developed for the audience or topic. For example, in one survey I created for teenagers, I took several days interviewing teens and testing categories to get the right “teen-speak” categories that they could effectively use to differentiate their level of belief in certain topics.

Timing. For volunteers, survey gathering should only be a few minutes. For employees that are required or strongly encouraged to take the survey, you can create a survey somewhat longer (7-10 minutes), while still respecting people’s time. Test the time it takes to respond to your survey as part of your pilot. In addition, consider the time your survey will launch. When persons are busy and overloaded, they will not respond as well. For example, if you launch a survey at the same time HR requires a 60-minute online module, fewer people will respond.

The survey invitation. The survey invitation is just as important as the survey itself. Often, people will receive survey invitations in an email. Your email should include a brief description of the survey, the time it will take to respond, a deadline, who is asking for the information, how the information will be used, as well as any guarantees of confidentiality or anonymity. If there are any rewards for survey completion, how to collect the reward should be specified. The most obvious thing the invite should include is a working survey link with an option to cut and paste the link. During the data collection period, several follow-up invitations should be sent. Depending on the nature of the data collected and decisions to be made, it may be worth the effort to seek out non-responders via phone interviews, additional invites, or paper surveys to compare their responses with prior responders. Generally, your first responders are more positive than non-responders.

Sampling. Sampling methods should be employed to avoid oversampling or survey burnout. For example, if your population is 10,000 ready respondents, consider a random sample of 300-400 persons, especially if you are conducting several surveys throughout the year for this population. Survey sampling websites can help you determine the appropriate sample size. If a ready sample is not available, a researcher can pay persons to take the survey or use social media or snowball sampling methods to find their target audience (see prior section on sampling).

As efficient as surveys are, they are not the best method for questions that require complex explanations or for studying multiple overlapping concepts in developing fields. For example, a researcher may be seeking to design a user interface for an app or may be forming a theory where grounded theory methods would be more appropriate.

Statistical Analysis

There are four main types of quantitative analysis: descriptive, causal, experimental, and correlational (predictive). Descriptive analysis describes your data in a summary form. The mean score, a histogram, a standard deviation, frequencies, and skew are all examples of descriptive statistical data. Most often, descriptive data is used to determine the appropriate type of further analysis. For example, if your data is highly skewed, you would use a different correlation technique than simple correlation (r). This introduces the concept of a statistical decision tree. A statistical decision tree helps a researcher make the right decisions about using the appropriate methods for analysis. Click here to see an image of a statistical decision tree or here for a computerized model.

Analyzing Group Differences (Causality and Experimental Analysis)

T-tests are used to compare differences in two groups, most often the mean (average) difference of two groups. Essentially, all statistical tests are measuring whether differences are due to more than chance alone. The assumption behind comparing two differing tests are that some experimental treatment caused the differences; in instructional design, usually the designed curriculum or tool is assumed to be causal.

The most common t-test analysis is an independent samples t-test. This type of test compares two different samples on a common measure. For example, online and live student’s final test scores in a course are compared. A paired samples t-test analysis is used when you have two measurements on the same person (or thing). For example, a pre- or posttest where student one’s pretest score is compared with student one’s posttest score. A third type of t-test is a one-sample t-test. This compares the mean of a single group with a known group. For example, comparing a current cohort’s attitudes towards learning math with baseline historical data from prior years.

But how do you analyze differences in samples when there are three or more groups? You could perform several different t-tests, but this can become very complicated when there are several groups. Analysis of Variance (ANOVA) analyzes mean differences among several samples and yields similar statistics to t-tests to show differences are more than due to chance alone.

Analysis of Covariance is similar to ANOVA but seeks to remove the effects of known variables. ANCOVA can also be used when simultaneously analyzing categorical variables and continuous variables and how they affect a third variable. For example, suppose you are testing three different curricula, but you cannot assign students at random. To control for this lack of random sampling, you administer a pretest score. The pretest score is a very strong predictor of your posttest score. With ANCOVA, if the relationship between a pretest and posttest score can be statistically quantified, the effects of the pretest level can be controlled to examine the overall effect of the three curricula (essentially seeking to remove the impact of pre-existing knowledge).

Analyzing Relationships (Predictive Analysis)

Correlational research seeks to examine relationships between two variables, such as the relationship between the amount of books read and five paragraph essay scores. Often, the goal is to find predictive relationships (e.g. those who read 5 books a month are likely to have 3 times higher scores on 5 paragraph essay scores than those who read 1 book a month).

The simplest of relationships is a linear relationship, or a line. As one variable changes (increases or decreases) another variable changes in a consistent manner (increases or decreases). Pearson’s r is used with simple continuous data (foreign language anxiety scores relationship with language acquisition scores). Spearman’s rho is used with rank order data (rank in graduating class relationship with rank on the SAT). Phi coefficient is used with dichotomous categorical variables (Instagram account or not and retired or not).

Regression statistics measure the relationships between many variables. Linear, or simple regression produces a best fit line for prediction between two variables (e.g. score on one test and score on a second test). Multiple regression is when a dependent variable is predicted by many or multiple variables (amount of time studying, days in class, and pretest scores all predict final exam score). There are many specialized types of regression models used in predictive modeling and to exhaust them all would be a much larger paper, but linear and multiple regression are the most common types of regression models used in instructional design research. To learn more about correlational, predictive, descriptive, and experimental research analysis, enroll in a quantitative research methods course.

Common Mistakes in Quantitative Data Analysis

Following are the most common mistake in quantitative data analysis:

1. Inappropriate sampling. For example, too small of a sample, not using a random sample when the analysis method requires it, and over generalizing about all persons in a group when only a sub-population of the group was sampled (a.k.a. extrapolation error).
2. Inappropriate methods. For example, using Pearson’s r for correlation with rank order data. Or, using a simple t-test when the data is highly skewed and a Wilcoxon method is more appropriate. A statistical decision tree can help you avoid this mistake.
3. Failure to report descriptive statistics. Novice researchers may jump right into their analysis or report of their findings without explaining why a method was chosen or not chosen. A simple qualifier in your analysis such as, “A histogram showed the data was not normal. Therefore assumptions of normality were not met and XYZ method was chosen for analysis,” would suffice.
4. Failure to review and clean data. Your data must be in the right format for the statistical package you choose to use. Therefore, reviewing data, often in spreadsheet form, or reviewing the first few lines in your statistical package can help you see if the variables are all aligned with the data, if they are consistently coded, and if you are analyzing all or just some of your data.
5. Miscoding variables. Most statistical packages allow for some transformations or computing of variables. For example, strongly agree to strongly disagree could be recoded as 1 through 5. Or a total score can be computed with several or all of the variables. When recoding, be consistent. If 1 = strongly disagree on one variable, it should be consistent throughout the dataset. It’s always good to double check when transforming data or recoding data.
6. Failure to account for missing data. Novice researchers often forget to decide how to account or code missing data. Others mistakenly treat missing data as a 0, and the results can be inaccurate.

Reporting Research to Stakeholders

In academic research, you report to a committee, usually with differing opinions, preferences, and specialties. In business, industry, and government, reporting is not very different. Learning what your stakeholders are interested in and how they prefer reports is important. For example, if they prefer visuals in the forms of graphs, charts, and process flowcharts, create those. If they prefer quantitative data (such as means, standard deviations, correlations, regression lines) over qualitative data (stories, experiences, and personas), then it may be worth reporting such data. However, most stakeholder groups are diverse enough it is often best to include several differing types of reporting, visuals, tables, flowcharts, diagrams, and rich experiences and stories.

Often, a persona of a typical person in a group or subgroup illustrates poignant areas of findings. For example, one client wanted research to identify who applies to teach for them. After analyzing data from 1500 applications, we created the following persona:

This is Sophia. Sophia is a Latin American Studies Major, with a Spanish Minor. She currently does not have a job but is looking. One of her friends suggested a job at [your organization]. She is bilingual and has just returned home from abroad in a Latin American country. She loves [the organization’s] environment and mission and is always looking for opportunities to share what she has learned, especially with new language learners. Statistically, Sophia’s persona is the most likely to apply to [your organization].

After the reports are presented and, data is shared and reminded, do not take it personally if the findings are not immediately acted upon or if every recommendation from the committee or research team is regarded. It often takes time for organizations to act upon new findings, and organizations are often juggling many initiatives. Often, researchers are not involved in strategic planning, so if your research findings are not immediately acted upon, there may be a strategic plan for acting on them at a later date or there may be other more pressing needs to address for the organization.

In one office I worked for, we had a phrase we used to describe a concept of waiting to share findings: “Don’t share the wine, before the time.” A common mistake of researchers is to share their findings before they have been fully gathered or analyzed. For example, the first 100 of 300 replies are analyzed and the most common finding is that students are enjoying the new LMS. However, there are still 200 responses left to code and analyze. Suppose the other findings differ but the findings have already been shared or reported as positive. And no matter how many times you say “this is preliminary data,” all a stakeholder hears is a fact.

Conclusion

This chapter is an introduction to research that can be useful to inform design decisions. The tools of research in instructional design are similar as the tools of research in most other disciplines. Once the basic principles of research are mastered in one setting, it is easier to begin using them in others.

Even though there is a science filled with appropriate and inappropriate research decisions, there is also an art to research. With experience, a researcher sharpens their research skills and knowledge of when to use which method to a point where they see the art in the science, and the science in the art. As a researcher, I enjoy this process of aiding an organization or client in the art of discovery. Among my colleagues, I often joke about researchers being a special type of breed. We are curious by nature, so curious, that it leads us to almost crave discovery. And, in my opinion, that is why you will find the most curious minds are always engaged in research.