One important aspect of research on, and interventions for, students with LD is understanding the parameters within which findings hold. These parameters could be related to various characteristics, such as participants, the environment, and the interventions (Coyne et al., 2016). For example, a particular reading intervention may have evidence to be useful with students in K-2, but that does not mean it is equally effective for students in grades 3 to 5. Similarly, some patterns may be present in children in an urban environment, but not in those living in rural settings. To find out where these parameters lie, research should go through several phases, from piloting a study, to direct replication, and finally to conceptual replications (Coyne et al., 2016). Direct replications are a way to ensure effects found in a study are robust, and not due to “error, bias, or chance” (Coyne et al., 2016, p. 250). Direct replications are essentially duplicates of the original study and difficult to realize in applied educational research. In fact, in a comprehensive review of 36 special education journals, Makel and colleagues (2016) found only 90 direct replications in 45,490 articles. Conceptual replications help define the parameters of an effect (Nosek & Errington, 2019). These replications can be closely aligned to the original study, with only a low number of dimensions different from the original study (Coyne et al., 2016), for example, examining the effect of an intervention using the same intervention materials and training, grades, and population, but conducting the study in a different geographical area (e.g., Gersten et al., 2015), or extending the length of the intervention (e.g., Toste et al., 2019). When more dimensions change, a replication is considered distal and will speak to the generalizability of the effect, for example, when changing both the group size of an intervention and the geographical area (e.g., Doabler et al., 2019). Ideally, researchers move systematically through the different phases of replication.

The importance of this phase structure for defining these parameters is not limited to Open Science. In fact, they are reflected in both grant funding structures of the NSF and IES. Moreover, IES includes mention of these parameters in their mission and this commitment is reflected in their recent commitment to funding replications of previous positive interventions. Specifically, IES holds “finding out what works for whom under what conditions” (U.S. Department of Education, Institute of Education Sciences, 2018) as a central goal. To achieve this goal, IES funds projects under the themes of Exploration, Development and Innovation, and Initial Efficacy and Follow-Up. Similarly, NSF funds projects as Exploratory, Design and Development, Impact, and Implementation and Improvement. For conceptual replications, IES has added a different funding competition specifically focused on reading and mathematics interventions that have previously shown to be effective (CFDA 84.305R and CFDA 84.324R).

Besides funding agencies, replication is also prominent in the discussion of finding evidence-based practices in single case design research (Horner et al., 2005). The proposed criteria by Horner and colleagues (2005) for determining if an intervention has sufficient evidence base include five different direct or distal, high-quality replications across three different researchers in three different geographical areas.

To be able to execute direct or conceptual replications, researchers will need access to the original materials, including intervention materials, assessments, and data analysis plans. Traditionally, researchers would have to reach out to other researchers to request materials and more information. Requesting information from authors may not have a high success rate for various reasons (Manca et al., 2018). For example, researchers may have changed institutions and the listed contact email address may therefore not be valid anymore. By adhering to the principles of Open Science and having materials such as Analysis publicly available in a central repository, such problems could be avoided, and more replications may be performed increasing our knowledge on the boundaries of intervention effects.

A second benefit of Open Science methods is they can serve as a catalyst for research. While data are often collected with a particular hypothesis in mind, it is likely other research questions could be answered using different models with the same data. For example, a large number of researchers have answered a copious amount of questions using data freely available from the Early Childhood Longitudinal Studies (ECLS) and the Schools and Staffing Survey (SASS). These large-scale data sets may seem fundamentally different from research project data. However, data sets from multiple projects can be combined with each other. The combination of data sets in common repositories can be a powerful tool to foster creativity and stimulate novel research questions. As an example, the combination of about 230 individual data sets on children’s language in the Child Language Data Exchange System (CHILDES), an open data repository, has led to the publication of over 5,000 articles. While it is not impossible that the original researchers might have reached this number of publications, it is more likely the repository gave other researchers a chance to explore a new question or theory.

Besides opening up data to allow others to ask novel questions, combining data can help answer questions that researchers were not able to answer in their own sample due to low numbers of participants, or low numbers of behavioral occurrence (Bainter & Curran, 2015; Curran & Hussong, 2009). This is also especially important in research on LD. Students with LD often comprise the left tail of a normal distribution. Thus, within any given sample of students, the number of students with LD will be low. It is often either time consuming or extremely costly to collect enough data on our population of interest to be able to run analyses with sufficient power. Other research groups, however, may have very similar data. Combined, these data may generate a sample size of students with LD that provides enough power to answer novel questions.

For example, to examine if students’ scores on executive functioning measures predicted reading disabilities, Daucourt and colleagues (2018) combined eight different data sets. The original data sets included student achievement data from reading intervention studies. However, the original research did not include measures of executive functioning. Daucourt and colleagues sent out an additional parental questionnaire to all students that included items related to executive functioning. Their final sample included only those students from the original studies whose parents returned the additional questionnaire and the sample consisted of about 10% of the original sample (i.e., 420 students). Around 30% (i.e., 139) of these students were considered having a reading disability. The authors were then able to show that lower executive functioning was related to reading disability.

In this particular example, Daucourt and colleagues (2018) were able to capitalize on existing data to run a relatively low-cost study. By using the already collected reading achievement data from the original intervention studies, they avoided having to unnecessarily spend resources on assessments and were able to append these data with the parent questionnaire. In other cases, collecting additional data may not be necessary. With the same eight data sets, researchers could examine the impact of the original reading interventions on only the subset of students with reading disabilities. Whether or not additional data collection is needed, researchers will need to be able to find and access the data sets containing their variables and populations of interest.

In most publications, authors present summary data on the variables of interest. To increase transparency and spur reuse of these data, Open Science urges researchers to make all data, at the individual level, publicly available (Nosek et al., 2012; Nuijten, 2019). Providing Open Data involves uploading a raw, yet curated data set to an online, public repository. Many funding agencies consider publicly available data sets permanent products of the grant. Deposited in a public repository, data set will obtain a digital object identifier (DOI) and is therefore a citable, permanent product of a research project. This guarantees data sets remain accessible with the same identifier over longer periods of time. Researchers can include this product on their CVs and show the impact of their work beyond publications through the number of times their data set was used in secondary analyses. Recognizing the amount of time and resources that are associated with curating and archiving a data set, funding agencies such as the NIH allow part of a budget to be allocated specifically for this purpose.

There are several online repositories available for archiving data sets. These repositories contain data from a wide range of disciplines, such as the Open Science Framework (www.osf.io), Figshare (http://www.figshare.com), and the Inter-university Consortium for Political and Social Research (ICPSR) (http://www.icpsr.umich.edu/). Increasingly, discipline-specific repositories are being developed, such as LDbase (http://www.ldbase.org/), a repository for data specific to research focused on learning differences and LD, and Databrary (https://nyu.databrary.org/), a repository for developmental video data. In addition, grant funding agencies may have their own repository. For example, the NIH supports DASH, a repository for data and specimens collected by NICHD grantees (https://dash.nichd.nih.gov/) and the National Database for Autism Research (NDAR; https://nda.nih.gov/). The NIH is currently exploring other archiving options for data sets that are not fully aligned with domain-specific repositories. A first attempt to this end is a specific NIH instance of Figshare (https://nih.figshare.com/). Finally, there is a specific repository for qualitative social science research (https://qdr.syr.edu/). Each repository may have specific requirements for depositing and storage of data (e.g., allowing embargoes, allowing researchers to self-deposit, types of data files supported). Table 1 provides an overview of several of such requirements for the repositories listed above.

Table 1. Selected Features of Several Public Repositories Hosting Educational Data.

 

Table 1. Selected Features of Several Public Repositories Hosting Educational Data.

View larger version

Preparing data to be deposited in an online repository requires more than uploading files. Researchers should take several steps before data are ready to be shared. First, a raw data set needs to be cleaned and deidentified before making it available. Curating a data set involves removing all identifiable information about participants (such as birthdays, names), checking for out-of-range values, and ensuring consistency across variables (Klein et al., 2018). In line with the IES SEER principles, it is not necessary for a researcher to share all data that were collected, but at a minimum all data that were used in any publications. Finally, researchers should make decisions about access options. Besides making the data available immediately, many repositories can restrict access to a data set during an embargo period or release data only to researchers who requested access.

In addition to a raw data file, it is also imperative to include metadata. Metadata can be described as information that can support the “discovery, understanding, and stewardship” of other data’ (Day, 2005, p. 10). In other words, metadata can help researchers locate data sets that possibly contain information they are interested in and evaluate if the data can be used to answer their question. For educational research, the information provided in metadata will relate mostly to the context and procedures of data collection and storage and is often made available through a codebook. A codebook contains the names of the variables, their labels, specific text of a questionnaire, the values of the variable and their labels, and how missing data is indicated for each variable, and scoring rules. If variables have been transformed before analysis, additional information about the methods used to do so might be included (“What is a Codebook?,” n.d.). Several commercial software programs exist that can generate codebooks based on survey data (e.g., StatPac). In addition, the Document, Discover, and Interoperate alliance (DDI) provides an online tool to generate interactive codebooks that can handle postprocessing and ongoing data collection (https://ddialliance.org). For R-users, several packages exist that can add the metadata to data sets such as codebook (Arslan, 2018).

Adhering to Open Data may seem a daunting task. The Digital Curation Centre (DCC) has helpful resources on all aspects of data management and curation, including which data to share, best way to organize data, and how to write a good data management plan (www.dcc.ac.uk). Many academic librarians are also well versed in data curation and can be valuable resources when preparing data to be shared. Working collaboratively with experts in data management and curation will help researchers make their data findable, accessible, interoperable, and reusable.

With Open Analysis, researchers provide a detailed account of all steps taken in the statistical analysis, beginning at the raw data and ending with the final statistical results (Klein et al., 2018). Providing the complete steps of an analysis is important for several reasons. First, during any analysis, a researcher has the freedom to make choices on how to run the analysis. This is sometimes called the researcher degrees of freedom (Simmons et al., 2011). Using varying analytical decisions can lead to differences in analysis outcomes. Carefully annotating the decisions, in addition to commenting on the analysis itself, will provide the necessary details to understand the analysis for the study. In addition, this annotated workflow provides researchers who are new to a statistical analysis with an overview of the decisions that need to be made, and they may gain a deeper understanding for the specific statistical techniques. It is also possible that preparing documentation of data analysis can lead to the discovery of errors in code (Epskamp, 2019), giving authors the opportunity to rectify results.

Second, statistical software packages have different default settings for certain operations (Epskamp, 2019). This may seem problematic only for more complex and advanced statistical methods, such as structural equation models; however, even more commonly used statistical analyses are handled slightly different in different software. Running an unbalanced analysis of variance (ANOVA) model with the default options in SPSS can yield different estimates of parameters than the default in R, because each program calculates the differences between groups based on a different combination of components, called Type I, II, or III sums of squares (see Navarro, 2017, for a detailed explanation of the different types and how they influence parameter estimates). While these differences may be small, they could lead to erroneous decisions about irreproducibility. By providing details about the software, its version, and possible additional packages used in their workflow and write-up of the study, researchers can avoid confusion about their results.

Open Analysis documentation will look different for each project. Many (commercial) statistical software programs will allow the researcher to save the syntax (e.g., Mplus, SPSS, and SAS), sometimes including annotations. Open-source statistical software, such as R, Python, and JASP, always allow a researcher to save the complete workflow with comments. It is likely data analysis will not always be shared perfectly. Researchers may be unable to share a complete workflow, or the workflow may only work on certain systems, leaving users of other systems to evaluate the analysis based on code alone (Klein et al., 2018). Rather than to let this be a deterrent for sharing all together, it is preferred to share any part or version of a workflow available. If it is impossible to share syntax, for example, for data analysis performed in a spreadsheet program, researchers could share screenshots of the flow of menu options used to perform an analysis or a step-by-step description of decisions made (Epskamp, 2019; Klein et al., 2018). By sharing what is available, even if it seems scant, researchers can demystify their analysis and increase transparency.

In many journal articles, researchers include small sample items of a measure or limited examples of study protocols, such as intervention steps or implementation fidelity checklists. However, these materials are seldom sufficient for replication of a research project. Previous page restrictions in journals may influence the limited sharing of research materials, but with the advent of repositories and cloud-based storage, it is possible for researchers to share all details of their study with other researchers (Grahe, 2018). By adhering to Open Materials, researchers add to the overall transparency of their project and give independent researchers the opportunity to carefully control the differences between their project and the original project (Grahe, 2018; Klein et al., 2018).

When sharing research materials, it is best to be as exhaustive as possible. At a minimum, all study protocols, assessments, and stimuli needed to successfully run a replication study should be uploaded (Grahe, 2018). It is likely, however, that there is a need to add specific walk-throughs or instructions for parts of the project. In the case of intervention materials, for example, it will be helpful to note the degree of flexibility an interventionist has in going off script. Additional important materials include blank informed consent forms (Lewis, 2020). If sharing materials infringes on copyright, for example, for commercialized assessments and intervention materials, these materials do not have to be provided by the researcher, given that they are openly available already (Grahe, 2018).

Providing Open Materials, particularly the most essential materials, is likely the least complicated and time consuming of the open science practices. In many cases, materials have already been created and are likely stored in the project’s digital location. Most of the data repositories mentioned in the Open Data section allow researchers to add materials to their data sets for easy access. Similar to Open Data, repositories can assign DOIs to the materials, making them citable products of a project.

In a preregistration, researchers delineate the parameters of their study by clearly describing their hypotheses, methods for data collection, and data analysis plan in a study protocol before data analysis is conducted (van’t Veer & Giner-Sorolla, 2016). The ultimate goal of preregistration is to provide transparency on the research process. Transparency through preregistration does not imply a plan cannot be changed. On the contrary, preregistration can be an iterative process allowing researchers to specify how they responded to unforeseen challenges during the research design and collection analysis (Gehlbach & Robinson, 2018). For example, many researchers are currently forced to adapt research protocols due to COVID-19. In this case, an original preregistration protocol of a study examining the relation between independent reading, motivation, and LD may have included three waves of in-person data collection. Due to restrictions on face-to-face contact, researchers changed the setting for the last wave of data collection to video conferencing. The updated protocol should specify this change and address potential implications of interpreting the outcomes of the last wave given the change in setting.

In addition, some analyses may be difficult to list specifically. For example, researchers may have a set of predictor variables to include as random variables in a hierarchical linear model based on their substantive theory. During the model building process, however, some of these variables do not appear to vary in their slope across clusters and adding the random slope does not increase the fit of the model. The researcher decides to drop these variables. The final model depends on outcomes of intermediate tests of significance. In this case, the data analysis section should consist of a clear decision-making process for the inclusion or exclusion of variables. Researcher may also list contingencies to the original analysis plan (Gehlbach & Robinson, 2018). When uploaded to a registry, preregistrations are assigned an ID number and each iteration of a preregistration receives a specific time-stamp so that the history and appropriateness of the changes can be assessed by others.

Several of these registries exist, some with a wide range of topics, and others more specific. For systematic reviews and meta-analyses, for example, protocols are typically uploaded to Cochrane (https://us.cochrane.org/) or PROSPERO (https://edtechbooks.org/-vwSg). Both organizations provide extensive documentation on their sites guiding researchers through the protocol and registration process with specific templates to follow. Specifically for intervention research, registries are hosted by the Society for Research on Educational Effectiveness (SREE) (https://sreereg.icpsr.umich.edu/), OSF (www.osf.io), and AsPredicted (www.aspredicted.org). Most of these registries mainly support experimental and quasi-experimental group design studies and provide templates with guiding questions. Recently, the field of special education has also called for preregistration of single case research (A. H. Johnson & Cook, 2019) and it is certainly also possible to preregister qualitative studies.

For Registered Reports, a full introduction and methods section of a manuscript are submitted to a journal and then it goes through typical peer-review process. This process gives outside experts the opportunity to provide feedback on the design of the study, potentially signaling flaws or suggesting improvements and expansions. After this peer-review process, the journal may give a “provisional acceptance,” which means the journal will publish the study when executed according to plan regardless of the findings (Nosek et al., 2019; van’t Veer & Giner-Sorolla, 2016). With respect to education and research with LD populations in particular, several journals have specific guidelines for submitting registered reports including Exceptional Children and Scientific Studies of Reading. The Center for Open Science (COS) provides lists of other journals accepting registered reports and journals that have published special issues with registered reports (https://cos.io/rr/).

Preregistration is the most prominently featured aspect of Open Science in the SEER Principles. The SEER principles focus on the comparison between what was proposed and what was eventually done and reported. Besides promoting transparency, making protocols available before the start of a research projects helps to make a distinction between outcomes that were hypothesized before a study began (i.e., confirmatory results) and exploratory results that were the result of unexpected patterns in the data. The exploratory results might warrant subsequent confirmatory research especially designed to test the new hypothesis. This distinction between confirmatory and exploratory outcomes is the main benefit of preregistration and registered reports (Cook et al., 2018; van’t Veer & Giner-Sorolla, 2016). This does not mean that exploratory analyses are precluded from research. On the contrary, Open Science values exploratory analyses as a means to find unexpected results. These analyses and results should merely be noted as exploratory.

Open Access refers to making research reports publicly available without a subscription barrier (Klein et al., 2018; Norris et al., 2008). For many researchers, reading about a certain method, data set, or intervention in a paper is the stimulus to examine the issues more carefully and possibly to conduct direct or conceptual replications (Kraker et al., 2011). When research is presented Open Access, more researchers will have the opportunity to engage with the research.

Grant funding agencies already expect research articles to become available to the public and have their own outlets. In the case of research sponsored by IES, it is expected papers are made available to the public through ERIC; NIH grantees used PubMed Central, and NSF uses its own public access repository, NSF-PAR. Research in several different areas has shown that articles published Open Access (either through the journal or through self-archiving) get cited more often than articles behind a paywall (e.g., Eysenbach, 2006; Metcalfe, 2006; Norris et al., 2008). In general, there are two ways to share manuscripts with the public. Using the Green way, researchers post their work on preprint archives; using the Gold way, researchers either publish in a fully Open Access journal, or pay additional fees to the publishing journal to make the manuscript Open Access (Harnad et al., 2004). These fees differ per journal and can be as high as US$3,000, with an average cost of about US$900 (Solomon & Björk, 2012). To help researchers with the cost of making research open access, many universities now have grant programs specific to this purpose. In addition, costs for Gold access in journals can be written in budget justifications of major grants.

Many of the journals in which research on LD is published allow researchers to post preprints and postprints (i.e., Exceptional Children, Exceptionality, Learning Disability Quarterly, Journal of Learning Disabilities, The Journal of Special Education). The website hosted by SHERPA/ROMEO (https://edtechbooks.org/-DDfk) has information on the archiving policies for most journals related to education science and LD, as well as their access options. Preprints can take the form of near-final versions of a manuscript that has been submitted, or the final version, accepted for publication. Some journals require a preprint to be the unformatted version of the manuscript. Several archives exclusively hosting preprints exist. EdArXiv is a recently established archive for educational preprints and associated with the OSF repository and authors can link preprints hosted on EdArXiv to their OSF projects. There are several benefits of posting a preprint to an online archive. First, all papers that are archived receive a DOI and thus can be cited and referenced, prior to the lengthy peer-review process begins. This speeds up the impact our science can have. Relatedly, the archives will also track the number of downloads and citations of these papers. More importantly, the archives allow the researcher to protect their work legally by assigning it a license, such as a Creative Common license (https://edtechbooks.org/-sihS). Even if papers are theoretical, purely exploratory, or were not written from open science at the start, authors can make sure their work is accessible to all by posting preprints. See Fleming (2020) for a useful flowchart on the decisions on posting preprints.