By Lisa Carey, Education Consultant for the Center for Innovation and Leadership in Special Education at Kennedy Krieger Institute, and Alison Pritchard, Director of the Neuropsychology Research Lab at Kennedy Krieger Institute
February 7, 2017
Not all teachers are required to take research methods courses as part of their credentialing. However, laws and policies governing education frequently reference the requirement of using evidence-based practices and interventions, which are practices supported by educational research. The Every Student Succeeds Act (ESSA), signed in 2016, requires schools to use “evidence-based interventions” for the purposes of school improvement. The US Department of Education provided Non-Regulatory Guidance on Using Evidence to Strengthen Educational Investments. But, in examining the guide, it appeared that a crash course in research methods might be necessary to fully understand what types of practices qualify as evidence-based under the law. If you are new to concepts of research methods or would like a quick refresher, continue reading!
A Close Read
Let’s use our close reading strategies to examine how evidence-based is defined by federal legislation.
We will start by looking at the definitions of evidence-based practice from the Elementary and Secondary Education Act of 1965 (ESEA), which also applies to the 2016 Every Student Succeeds Act (ESSA).
According to section 8101(21)(A) of the ESEA
“…the term ‘evidence-based,’ when used with respect to a State, local educational agency, or school activity, means an activity, strategy, or intervention that:
(i) demonstrates a statistically significant effect on improving student outcomes or other relevant outcomes based on –
(I) strong evidence from at least one well-designed and well-implemented experimental study;
(II) moderate evidence from at least one well-designed and well-implemented quasiexperimental study; or
(III) promising evidence from at least one well-designed and well-implemented correlational study with statistical controls for selection bias; or
(ii) (I) demonstrates a rationale based on high-quality research findings or positive evaluation that such activity, strategy, or intervention is likely to improve student outcomes or other relevant outcomes; and
(II) includes ongoing efforts to examine the effects of such activity, strategy, or intervention.
If you are a research methods novice, many of the vocabulary terms used in the legislation quoted above may be unfamiliar to you. As an expert learner, you recognize how important a thorough understanding of vocabulary can be to instruction, so why don’t we unpack some of the language used within the legislation?
In looking at this definition of evidence-based, several key words or phrases stand out. But, before we dive into the terms used within this definition, it might be helpful to know that there are three main categories of research methods; qualitative, quantitative, and mixed-methods.
Quantitative research methods utilize measurement to ascertain the effects of an intervention. Most quantitative research will use statistics to analyze results of an educational intervention or practice. For example, a quantitative study might compare math facts knowledge differences between an experimental (or intervention) group and the control group (meaning the group that did not receive the intervention). By statistically comparing the number of math facts that the experimental group answered correctly (on average), with the number of math facts that the control group answered correctly, the researchers are able to ascertain if the intervention was associated with better math facts knowledge .
Qualitative research methods do not focus on measurement. Rather, this type of research uses data such as interviews, case studies, artifacts, and focus groups to determine the impact of an intervention or educational practice. For example, a qualitative study might conduct focus groups of teachers who participated in a school-wide intervention in order to look for trends about teacher beliefs and attitudes toward the intervention .
Mixed-methods research combines the two approaches into one study. A mixed-methods study might include measuring increases in student vocabulary knowledge after an intervention, as well as interviews with teachers who participated in the study to ascertain the feasibility of the intervention .
As you just learned, quantitative research methods utilize measurement and statistics to determine the impact or effect of the studied practice or intervention. Thus, the following terms used within the ESEA definition of evidence-based are related to quantitative studies:
Statistically significant: When statistics have been used to determine how likely an event is to have occurred due to the researched intervention rather than by chance. For example, a statistically significant effect or difference in the math fact intervention example would indicate that the researchers are at least 95% certain that the difference between the control and intervention groups’ math facts scores is due to the intervention. That leaves only a 5% probability that this difference between the groups’ scores is due to chance .
Effect: The extent to which an intervention impacts the outcome being studied, or the difference between the intervention and control group on the outcome variable. For instance, the statistically significant difference in the math fact intervention example above could also be referred to as a statistically significant effect – it’s the effect that the intervention presumably had on the outcome.
Experimental study: A quantitative approach to research in which one or more variables (known as independent variables) are manipulated in order to examine the effect on a dependent variable (the outcome you are interested in). For example, if researchers want to know if a reading intervention improves student comprehension skills, they would have one group of randomly assigned students use the intervention (the independent variable) while a second group, or control group, goes about business as usual and doesn’t use the intervention. If the intervention group has a better outcome and the results are statistically significant (i.e., not likely due to chance), then researchers can state that there was an effect of the reading intervention on comprehension scores (i.e., that the reading intervention improved comprehension scores).
Quasi-Experimental study: A quantitative approach to research that is very similar to an experimental study except that it does not use random samples. Random samples help ensure that the intervention group and the control group are equal on all variables except receipt of the intervention. Education is a difficult arena for experimental designs (randomly assigning students to classes is logistically difficult), so they use methods to try and make their sample groups as similar as possible, and then carry out the experiment as described above.
Correlational study: A correlation is a statistical approach to determining if there is a relationship between variables. A correlational study is one that does not use an experimental or longitudinal (following the same students over time) approach to estimate the relationship between variables. For example, a researcher may be interested in the relationship between disruptive student behavior and grades. She may find that as the number of disruptive episodes in class increase, the grades on assignments decrease. While there is a relationship between disruptive behavior and poor grades, this study cannot demonstrate causation, meaning the researcher does not know if students act out more because they are frustrated with classwork they cannot successfully complete, or if students who are disruptive miss class time and struggle academically as a result.
Statistical controls: Humans are highly variable. In order to determine if a correlation or quasi-experimental outcome was due to the variables being examined, other variables must be statistically controlled for. This means using statistical methods to consider the impact of variables such as demographics on the results of the study. You might come across statements like, “controlling for gender and race, student self-efficacy was found to have a positive correlation with math achievement.” This means the researchers used statistics to analyze the possible impact of gender and race on their results and found that the two variables of interest were still significantly related even when any impact of gender or race was removed.
Selection bias: Studies utilize participants. In the case of educational research, participants most typically include students and teachers. Selection bias is related to procedures researchers used to determine who would participate in their study. Selection bias can be a problem when a study’s participants might not be representative of the population as a whole, as a result of the way that the researchers chose who would be in the study. ESEA requires that researchers statistically control for selection bias by using techniques described above.
What Kinds of Research Should I be Looking For?
You may have noticed that the ESEA definition seems to emphasize elements of quantitative research methods within its definition. While we could host a long debate about the merits of each kind of research, we will just quickly point out that all three kinds of research listed above are considered evidence within the field of education and all three kinds of research appear in high-quality, peer-reviewed education research journals. Additionally, all three kinds of research offer benefits and draw backs in their examination of pedagogical practices. However, as a society, we tend to like our proof in number form, and thus there is a strong policy leaning toward qualitative research. It is important to note that while the Institute of Education Sciences (IES; the government agency that funds, evaluates, and disseminates education research) has a strong focus on quantitative research, it also values qualitative research.
In addition to describing evidence-based practices using aspects of quantitative research, the ESEA definition also uses the phrase “demonstrates a rationale based on high-quality research findings,” which can be interpreted as an argument for using an educational practice supported by qualitative studies. The definition also states that decisions may be based upon “positive evaluation,” but does not mandate that these evaluations of practices be quantitative or qualitative in nature.
Where do I find this research?
Now that we’ve conducted a close examination of the ESEA definition of the term evidence-based practice, you may be wondering where to find good research. If your school or school system has access to databases that include peer-reviewed journal articles, you might try using key terms to search for information. If you’re having trouble finding what you’re looking for, ask your friendly, local librarian for assistance! Don’t have access to online databases or journals? Try a Google Scholar search, but make sure the article comes from a peer-reviewed journal. Many researchers are making their papers “open access,” which is helpful for teachers looking for information - thank you, researchers! Finally, you can use the What Works Clearinghouse to find interventions and practice guides. This government funded resource ranks interventions and practices by the level of evidence existing to support them and provides graphs demonstrating how effective the intervention was in improving student outcomes.
No matter what sources you use to uncover an evidence-based practice, be sure to consider the students who participated in the study. As most researchers will tell you, context is crucial. For example, if you’re looking for evidence-based practices to support students with reading learning disabilities, research conducted with students with autism spectrum disorder (ASD) might not offer the information you are seeking. However, many interventions and practices have been studied using multiple populations.
Hopefully, your confidence as an educator has been boosted by learning what it means for a practice to be evidence-based. . You should be empowered to seek out and use evidence-based practices for the benefit of your students and take a critical look at practices you’ve been asked to implement. Time and again, researchers have demonstrated the strong impact teachers have on their students. In arming yourself with powerful instructional practices, you ensure that your positive impact is even greater.
 U.S. Department of Education, Institute of Education Sciences (2013) Guidelines for Education Research and Development. Retrieved from https://ies.ed.gov/pdf/CommonGuidelines.pdf
U.S. Department of Education, Institute of Education Science.(2015) Experts Discuss the Use of Mixed Methods in Education Research. Retrieved from https://ies.ed.gov/blogs/research/post/experts-discuss-the-use-of-mixed-methods-in-education-research
 Punch, K.F., Oancea, A. (2014) Introduction to research methods in education, Second edition. Thousand Oaks, California: SAGE Publications Ltd.