Join the conversation

Who can you trust about educational technology?

So asked Richard Rose in September’s issue of School Administrator. He argues that research on educational technology should be approached with skepticism for a number of reasons. For example, the role that money plays in this research: Interested parties, including technology providers, nonprofits and even unions, often directly or indirectly benefit from research showing results for a particular product. There is also a lack of consensus within the research – you can find research that supports almost any position. In addition, there is bias introduced by the “publication strainer” (publications prefer research that supports their platform of doctrines) and author timidity and pragmatism (not wanting to waste time on research that isn’t published, busy authors submit what they know publications want).

In reading this piece, it struck me that it could have been written about any aspect of education. While the motivation of technology companies may be different than the motivation of those pushing vouchers, charter schools, alternative certification programs, particular reading programs or any other educational products or policies, it is widely acknowledged that much education research – for the reasons Rose cites and others – is substandard. But as Rose acknowledges, not all educational technology research (which I would change to, not all education research) is “tainted by vested interest or too insipid to bother with.” There is good research out there – you (whether you are a practitioner, parent, policymaker or other) just need to know how to look for it.

In a recent article in American Educator based on his new book, Dan Willingham offers a four-step approach that those who haven’t taken years of statistics courses and/or don’t have the time to comprehensively review the research can use to determine whether a policy, program or other educational resource is evidence-based and worth adopting.

For each “scientific” claim – be it that a new curriculum, math program, school restructuring strategy or other will have results – that you are investigating, he recommends that you:

• Strip it and flip it. Examine the claim in its simplest form. Get clear on the change suggested, the outcome promised as a result of the change, and the probability that the promised outcome will occur if you make the change. Willingham suggests filling in the following statement: If I do X, then there is Y percent chance that Z will happen. For example: “If my child uses this reading software an hour each day for five weeks, there is a 50 percent chance she will double her reading speed.” (If you can’t figure out either what you are supposed to do – X – or what is supposed to happen after you do it – Z – there is a serious problem with the claim). Then flip the promised outcome: There is a 50 percent chance she won’t double her reading speed – is the risk that the program won’t work acceptable? And flip the claim in another way: What happens if I don’t do X? Once you’ve thoroughly examined the claim, you might decide it’s not worth your time to investigate further. (The American Educator article goes into great detail on this particular step of the process.)
• Trace it. Who is making the claim about a product or program? Pay attention to the qualifications and motivations of the person trying to persuade you, but do not rely too heavily on credentials alone. And do not discredit research simply because you tend to disagree with the person conducting it or institution funding it.
• Analyze it. Consider the claim in the context of your experience, but recognize that experience is not an infallible guide. Then apply some simple guidelines to evaluate research claims. You do not need to get too technical, but consider general principles of good practice. For example, does a study claiming program effectiveness include both a treatment group using the program and a control group?
• Decide whether to adopt it. While a lack of scientific support does not always mean you should avoid a program or position (as Willingham points out, most programs lack such support), decisions should only be made once you have all available, relevant information in front of you.

Willingham admits that this system is imperfect and “not a substitute for a thoughtful evaluation by a knowledgeable scientist.” Still, for those who are charged with making decisions in education – be they about technology, curriculum, governance or anything else – it is a starting point to ensure that education research is neither given too much weight nor ignored completely. After all, to truly advance education both for our nation and for each child, we have to look beyond the easy, politically popular claims that a particular strategy is "evidence-based" that we too often accept at face value.

Image by Ed Uthman from Houston, TX, USA (Histologic Slide Under Microscope) [CC-BY-SA-2.0], via Wikimedia Commons