There are many different types of response scales; multiple-choice options are common, but Likert scales are used quite often as well. Each method used to obtain a response has limitations and strengths, as well as specific design principles that should be considered.

Multiple-Choice Response

In this situation, the item stem is phrased as a question followed by a list of potential answers. Each response option must be mutually exclusive or clearly different from the others. There are a few specific design decisions you need to consider when using the multiple-choice response method.

"Best Answer" or "Choose All That Apply"

Normally a respondent is expected to choose the answer that best represents their situation, beliefs, or condition. However, if the situation warrants, respondents may be asked to select all that apply to them personally. Only use the “all that apply” instruction if you have a good reason for doing so.

In the relationship status example, selecting the best answer is most appropriate. The options are mutually exclusive and expected to describe every relevant situation.

Opting Out

Obviously, we want respondents to answer every item on the survey. Unfortunately, ethical considerations may require you to allow individuals to choose not to answer a specific question. In reality, you can’t force individuals to answer any question whether you provide an opt-out option or not. If an individual selects the opt-out option or simply chooses not to answer the item, the result is the same. Most research protocols involving the protection of human subjects will require you to allow respondents to opt out at any time. However, failing to answer a key question may equate to refusing to take the survey, even if a respondent answers all the other questions. If the item was to be used as a disaggregation or branching variable, you should require respondents to answer the question (noting that respondents should still be allowed to opt out if they choose). If  this is the case, you might not be able to use any of the respondent's data if they don’t answer key questions.

Consider the relationship status item. This item most likely would be included in a survey to provide a basic description of respondent demographics. If this were the case, an individual’s failure to respond may not be a problem. Providing respondents with an opt-out option for this item protects the individual’s rights. However, if the results were intended to be disaggregated based on the relationship status, failure to answer the item would make the respondent's data unusable. This would be even more problematic if large numbers failed to answer or the response refusal had a systematic pattern. Similarly, if the information was to be used as a branching variable (so you could ask group-specific items), failure to capture this information would render this design feature inoperable.

Number of Options (Option Specificity)

In most cases when using a multiple-choice response method, you want to provide a list of all reasonable responses, but not always. In the relationship status example, the number of possible responses will depend on the various mutually exclusive relationship situations possible. However, the degree to which specific details are required should be determined by the research purposes. You may only need to know if the person is single (never married) or something else. If this were the case, and you had no plans to use the more detailed information, you may only need two response options. In a multiple response scale, the researcher either attempts to make the list of options exhaustive, or they will need to allow “other” responses. However, If the list gets too long, there is the potential for fatigue and option order affect bias.

Allowing "Other" Responses

When using the multiple-choice response method, the use of an option allowing respondents to write in another response has its advantages but also has limitations when it comes to reporting, interpreting, and drawing conclusions. The “other” option is used when the researcher is unsure whether or not they have an exhaustive list of possible answers. It should be noted that obtaining a comprehensive set of responses should have been obtained in the pilot testing phase of the instrument, which would reduce the need for allowing other responses.

To create a comprehensive list of possible answers, you might simply create a list using your own logic. For example, in the relationship status example it is very likely that you could logically determine each possible situation. However, this is not always the case. Consider a situation where you were looking for respondents’ reasons for doing something. In the pilot testing phase, you might include an open-ended question asking potential respondents to state their reasons. Responses would then be coded and categorized. A comprehensive list of the most common reasons might then be obtained. This would be the basis for the list of answers you use in the final version of the survey. In this situation, you may still wish to allow for other responses. However, you would need to clean the data afterwards by reviewing each response not already listed. This is done by reading each additional response and deciding whether the response is novel (something you hadn’t thought of), irrelevant/unrelated (the equivalent to a non-answer), or just a different way of saying what was already in the list of options (in which case the response should be recoded).

If the additional response is both new and meaningful, future versions of the survey should be updated to include this answer. However, the interpretation and reporting of the result you obtained with the incomplete set of response options would be difficult. If you report that one person provided an "other" response (a factual statement) you may be underreporting the importance or significance of the answer based on the frequency or prominence of the response. Had others been presented with the option, they may have also selected it as a response. This is an issue because we, as researchers, just don’t know.

Likert-Type Response Scales

A Likert response scale is most appropriate when the item stem is a statement. Likert response scales are typically used when designing a scale rather than a questionnaire but are also used in questionnaires. The Likert response scale is meant to capture the strength of an individual’s agreement or belief. An adaptation of the Likert scale is often used to group individuals based on some numerical amount (e.g., income or age grouping). Response options in a Likert scale need to be ordinal in nature and mutually exclusive; multiple-choice response options need only be mutually exclusive. There are a few design decisions you need to consider when using this specific type of response scale.

Ordinality and Mutual Exclusivity

This was mentioned previously but is important enough to explain further. A characteristic of ordinal data is that each point on the continuum represents a measurable increase in the amount or magnitude relative to the previous value on the scale and is some amount less than the next value on the scale. However, unlike interval-level data, the difference in the amount between each point on the scale is not neccesarily consistent (i.e., equidistant). For both ordinal and interval-level data, each point must be separate and distinct (mutually exclusive with no overlaps). In the texting example, if you provided the options “1 to 2” and “2 to 3,” the options would not meet the mutually exclusive requirement because respondents with exactly two hours could select both answers. Likewise, the following set of response options intended to capture the amount of time spent on an activity does not meet the ordinality requirement because “multiple times a week” and “as little as possible” are ambiguous. Respondents may interpret the amounts represented differently than what was intended (e.g., "as little as possible" may actually be a lot, not a little; "multiple times a week" might be less than twice a day). Cognitive load is also increased as respondents struggle with trying to determine the amount of time each option represents.

Option Specificity (Number of Response Options)

This is an issue for both multiple-choice responses and Likert response scales but for different reasons. With Likert response scales, the issue is one of specificity and, in some cases, ordinality. Typically, a Likert scale will have four to eight options. The specific number should be informed by the respondents’ ability to make clear differentiations between options on the scale. Having too few options (e.g., "agree" or "disagree") limits the ability to interpret the data regarding the magnitude of agreement or disagreement. In addition to not being needed, having too many options (e.g., more than eight) makes it difficult to give meaningful labels to each response (i.e., labels that makes them distinct and clearly ordinal in nature). Respondents most likely won't be able to make meaningful differentiations between points on the scale, and less granularity would likely serve the research purposes. 

Direct Entry Option

One way to alleviate the specificity and ordinality problem, in situations where amounts are used to determine group membership, would be to have participants enter the amount directly or derive the amount based on some other data (e.g., determining one’s age based on their birthday). Group cut points could then be established based on precise amounts rather than broad categories. Category cut points could be established in an a priori manner or based on natural divisions occurring in the data. If this approach is used, you will need to make it clear to respondents what units are being used (e.g., days, hours, or minutes) as well as whether partial amounts can be entered (e.g., decimals or fractions). With online surveys, application software often has features that allow researchers to set restrictions for data entry to ensure amounts are within a valid range and formatted properly.

Odds and Evens

Another often debated topic is the inclusion of a middle point. Having an odd number of options is sometimes used, which make it possible to have a middle point. Too often, however, it is used inappropriately. Having a middle point is acceptable when there is a true middle position. For example, when you are attempting to determine the amount of increase respondents believe occurred due to some phenomenon. If there was no increase and there was no decrease, a middle point representing no change is appropriate. Including the “neither agree nor disagree” option is thought by some to be appropriate in certain situations depending on the statement posed in the item stem; others feel it can lead to lazy responses and central tendency error when respondents don’t consider their choice carefully enough. Removing the middle point by having an even number of options forces respondents to decide whether they agree more than they disagree. It is never appropriate to place an opt-out option (e.g., “I prefer not to answer,” ”not applicable,” or “I don’t know”) in the middle of the response scale.

Item Stems as Statements

Some practitioners recommend using questions (with multiple-choice responses) rather than statements and Likert scales. Both can be used effectively and there are advantages and disadvantages to using either. However, for a statement to work properly with a Likert scale, the statement needs to represent a clear position rather than a middle of the road (ambiguous or noncommittal) declaration. For example, you might ask individuals whether they thought police salaries were OK; however, there is little value in knowing whether respondents strongly agree that salaries were OK because it is difficult to interpret differences between agreeing they were OK and strongly agreeing they were OK. In both cases the respondent feels police salaries are just OK. It's like saying, "I believe my hair is brown" and "I strongly believe my hair is brown." The reason for using a statement in the item stem is to determine the strength of an individual’s feeling towards the idea or concept. In addition to making sure the statement presents a clear position, the statement should not be something that everyone might agree with in general (e.g., "It would be nice to provide adequate salaries for police.").

In addition, information about respondents' feelings might require additional information to determine what people think should be done. You many need two questions because knowing how individuals feel about police salaries may not align completely with what they feel should be done about it.