Overview and Importance

The goal for any research is to get accurate and comprehensive information that can be used to answer the research questions. Survey research is no different. However, creating a survey that provides useful results takes time and effort. Unfortunately, many researchers fail to take adequate time to validate their survey instruments. One of the worst things for researchers is to get their survey data back and realize they should have created the survey differently. Without valid data, the study is ruined. And while it is unlikely that any survey you might create will be perfect, following a rigorous survey design process will increase the likelihood that the data you obtain will be useful.

Instrument Validation  

Unfortunately, the survey design process is not understood very well; many people do not know how to accomplish the task properly, and too often the process is not carried out thoroughly. More importantly, many do not understand that proper survey design involves instrument validation.

The term validity has a slightly different meaning when applied to assessment (i.e., testing) and surveys (i.e., self-report data collection). Validity in assessment generally refers to the degree to which a test measures what it was designed to measure. This definition can apply to surveys as well, however for survey research, validity also refers to the quality, completeness, and usefulness of the data. More specifically, validating a survey instrument is a rigorous process intended to verify that the data obtained from administering a survey will answer the research questions adequately. This does not mean the information obtained from the administration of a validated survey will be perfect. There are many ways in which the information obtained can be biased regardless of the quality of the survey (see Survey Error and Response Bias). Validation simply decreases the potential for response bias and helps us avoid mistakes that can result in survey error. Again, the validation process does not guarantee perfect data; rather, once a thorough validation process is completed, the researcher can be confident that the instrument has the potential to produce useful information. The validation process attempts to make sure the survey can be used to accurately gather all the data it was designed to obtain.

Phases in the Survey Design Process

The survey design (i.e., validation) process involves several steps or phases. They are outlined below. Subsequent chapters then deal with specific aspects of each phase in greater detail.

Conceptualization Phase. This is a theoretical endeavor that affects all the other steps in the process. The aim of this phase is to clearly identify the study’s purpose and research questions. Having a clear understanding of the purpose and research questions helps us figure out what data is needed.

Administration Planning. Although implementation is not technically a part of the design process, this phase involves planning the survey administration to ensure adequate sampling and sufficient response rates. Understanding how the survey will be administered will also affect item creation.

Data Analysis and Report Planning. This is a planning step that will guide how data is collected. Understanding how the data will be analyzed and reported improves the likelihood that you get all the data you need and that the information obtained will be adequate for the statistical analysis needed to answer the research questions.

Item Creation and Response Scale Design. This phase involves development. Both survey items and response scales are created. The goal is to make items that will produce the information required.

Refinement and Pilot Testing. The objective of this phase is to make sure mistakes have not been made before administering the final survey. Based on the results, changes to specific items can be made. Depending on the results, several iterations of the phase may be needed.

Section Contents