Join the conversation

...about what is working in our public schools.

What Really Happens When We Pay People for Test Scores?

vonzastrowc's picture

Paying anyone--students or teachers--for test scores might be a bad idea. That's one of the big lessons I draw from Roland Fryer's now famous study of programs that pay students for good behavior, hard work or test results.

In fact, I think the implications of Fryer's study reach farther than that. The study offers a glimpse of how dangerous it could be to attach any big consequences--good or bad--to test scores alone. Here are some of the things I took away from Fryer's report:

We ignore inputs at our peril. It has become received wisdom that outcomes--and that usually means test scores--are all that really matter in school reform. But Fryer's study suggests that people in schools who call for more attention to inputs and the processes of arriving at outcomes aren't just whiners after all. The study found that cash rewards for certain behaviors--like reading more books--were more effective than cash for test scores. In fact, cash for scores seemed to have no effect on student achievement. Why? Incentives to do the right things, the things that promote learning, might well work better than incentives to do well on a test.

Getting kids too focused on their test scores may do them little good--and may even harm them--in the long run. Fryer's team noted that students getting cash for scores naturally grasped at test-taking strategies rather than, say, better study skills or deeper engagement in class materials:

Students [who were asked what they could do to earn more money on the next test] stated [sic.] thinking about test-taking strategies rather than salient inputs into the education production function or improving their general understanding of a subject area.... Not a single student mentioned reading the textbook, studying harder, completing their homework, or asking teachers or other adults about confusing topics.

That lends some credence to those who call schools under NCLB "test prep factories." The students Fryer's team observed seem to be the natural products of a culture that focuses on "outcomes" at the expense of everything else.

Tying cash to test outcomes may blunt higher-order thinking skills. Fryer's team found evidence that students did not want tests to move them outside their comfort zones. The study describes some students' responses to New York's "incentive tests":

These exams can quickly move students outside their comfort zone and into material that was not covered in class-especially if they are answering questions correctly. The qualitative team noted several instances in which students complained to their teachers when they were taken aback by questions asked on the exams or surprised by their test results.

The students may have felt that higher-order thinking skills just weren't part of their bargain.

Forces outside of schools may drag down students' test results and challenge even the best incentive programs. Fryer's study notes that poorer students didn't get as much out of even the best programs. They got less bang for the school's buck. Fryer speculates about this finding:

The best evidence in favor of the importance of complements in production are the differences between students who were not eligible for free lunch (and likely have intact, more educated famliies who are more engaged in their schooling) and those who are eligible for free lunch, though the differences are not statistically significant. The social ills that are correlated with eligibility for free lunch may be important limitations in production of achievement.

These comments might send a chill down the spines of teachers whose evaluations and pay will hinge mostly on their students' test scores. Just one less reason to teach low-income children, I guess.

Tying teachers' pay to test scores might not be such a great idea after all. Fryer comes right out and says as much:

It might be less effective to give teachers incentive pay based on outputs (test scores of their students) relative to inputs (staying after school to tutor their students.)

Coercing or coaxing teachers to raise test scores may have little or no impact. It may even cause harm. But giving them more support to do the kinds of things that are likely to improve student learning might be a much better approach.

In its recent review of Fryer's study, Time magazine puts the issue this way: We tend to assume that kids (and adults) know how to achieve success. If they don't get there, it's for lack of effort--or talent. Sometimes that's true. But a lot of the time, people are just flying blind." Fryer suggests that pay tied to outcomes does little to remove the blindfold.

 

Despite what some have said about him, Fryer is no ideologue. He is cautious about his conclusions, and he would never present any of the speculations I've elaborated on as ironclad. The evidence is still thin. His interpretations need more research.

But some of the things he's hinting at are anathema to the hardest of hard-core people in the merit pay camp. As states like Florida prepare to bet the farm on teacher pay and evaluation programs tied mostly to test scores, they should pay close attention to what Fryer is writing between the lines.

(For another response to Fryer's study, see Larry Ferlazzo. Ferlazzo is very concerned about the long-term impact of cash rewards for students, and he offers a number of other ways he and his colleagues have been inspiring higher performance.)


Great post, Claus. We should

Great post, Claus. We should think of test scores as one of several possible outputs, rather than the be-all and end-all of teaching. Stop and think about all the private schools that don't bother with these shenanigans. Are they therefore without evidence of student achievement? As for public schools, I teach at one that has very high test scores. We never talk about them, never do "test-prep" - we just focus on good "inputs". Students read, write, and engage with their curriculum at a high level and for more authentically educational purposes. And I've seen the same thing at a successful charter school in South Central L.A. - View Park Prep. The test scores will follow when students have a challenging and stimulating curriculum, good support systems, and are comfortable at school.

There are lots of people who

There are lots of people who opine on the dangers of performance pay, but actually very few who have any real experience with it. Let me make some statements from the perspective of someone who actually works in a performance pay district.

1) You can use bad metrics to pay people. American schools been doing it with higher education credits and years experience for decades.

2) We should be focusing on both inputs and outputs in the systems. Those main foci should be adult behaviors (primarily teaching quality) and student achievement (primarily test scores). Having more and better measures of both the inputs and outputs creates more powerful and accurate systems.

3) We should not misuse test scores. Tests are one important indicator (of many) of how well schools and kids are doing. Tests, as any other measure, have bias and error and the analysis of test data should acknowledge that. The analysis of the data should be longitudinal, and not simple attainment based. Using a method like value added accounts for the outside factors and creates a quality estimate of educator effectiveness and student growth.

4) We should be continually making better evaluation systems to measure quality teaching. The concept of "good teaching" continues to evolve and so must our measures of it. Evaluators must be trained and tested in reliability to insure that evaluations are quality measures.

5) We should stop painting "worst case" scenarios in performance pay. Most of the criticisms I read of performance pay take some element of some district's program and extrapolate it to some Armageddon-like hypothetical "what if" end. Yes performance pay systems can be done badly. They can also be done very well - and there are all kinds of shades of gray in between.

While the research emerging on performance pay is still early as there just aren't (really) that many districts doing it, the studies increasingly show a positive effect. It is a tremendous and nearly ridiculous reach to say that performance pay systems "blunt" higher thinking skills because they may use student achievement results as one metric of performance.

In contrast there is a massive and established research paradigm showing that education credits and years experience are terrible metrics on which to pay teachers. It is clear these are not strategic ways to pay people.

Performance pay is about an organization using its finite resources in a strategic way to advance its goals.

There are also significant differences between the concepts of performance pay for employees versus performance pay for students. While critics like to lump all this together for the convenience of their argument, they really should be considered separately and on their own merits and shortcomings.

Jason Glass
Eagle, CO

Thanks for your thoughtful

Thanks for your thoughtful and spirited comment, Jason. You're right to recommend caution in comparing the effects of student incentives to those of teacher merit pay. Fryer makes the explicit comparison, as I note above, but he does so with all the caution you would expect from a top-flight researcher. Still, I do think his study demonstrates what can happen when incentives are tied too tightly to outcomes measured by unreliable or narrow assessments.

By no means does Fryer offer a blanket indictment of all merit pay programs. But his comments on incentives tied to narrow outputs are very suggestive. In my opinion, they give us a sense of the damage ill-designed merit pay programs can do. And programs like the one that raced through the FL legislature--which does away with salary scales and ties the bulk of a given teacher's evaluation and additional pay to test scores without offering any funds to develop the missing tests--have a very high risk of being ill designed.

As for blunting higher-order thinking skills.... A poorly-designed or narrow test tied to big consequences for students doesn't really create much incentive to get at those skills. Make teachers face big consequences tied to one narrow outcome, and I can imagine that higher-order thinking skills may fall by the wayside.

It seems we agree on the problem (and I hope I'm not being presumptuous here): "Tests are important, but we've got some work to do on knowing their limitations, analyzing the data appropriately, and using the information from them as one piece of the puzzle in how we help kids learn." I'm not sure everyone is willing to proceed with your insight and caution.

Thanks much for the courtesy

Thanks much for the courtesy of your reply Claus - our field is better through these kinds of debates.

I think we do have agreement that pay systems can be designed poorly and we can create systems to incentivize outcomes we'd rather not see in schools.

To muddy the waters a bit with another frequently debated concept, I am currently in the process of nonrenewing the contracts of several non-tenured teachers in our district who have demonstrably better skills than many of those we are forced to keep on due to tenure laws. I also see teachers in their third year (just before tenured status in my state of CO) get pushed out and viewed with an extremely higher level of criticism than they would otherwise have if they were not one year away from a practical lifetime guarantee of employment.

My point being, performance pay can be set up to lead to stupid outcomes and value distorted behavior. But the "old" systems that so many rush to defend have many of the same problems, just in different ways ... like the billions educators annually spend on chasing advanced degree credits that show no relationship with quality instruction.

My plea is for us to put our heads together and come up with better ways. I look forward to your next post Claus - I'll jump off this one to let you get to work!

The truth is that in the "big

The truth is that in the "big picture," teacher pay -- and the student's future income -- IS dependent on good test scores no matter what. Good districts can offer good pay because voters want to support *good* schools. When I went to high school, they had an "analogy of the day" on the loudspeaker to prepare us for the SAT. They offered special classes and coaches. The curriculum was "rigorous" or whatever you want to call it, but it was geared with the mindset that the children who would graduate from the institution would go on to the better universities. Yayy for me, because I'm not very bright but was able to do quite well and get into a decent university (the admissions office must have liked the fact that Daddy could pay cash for my education. Why else would they admit me?)

Do you know how much pressure my friends were under? It wasn't unusual (in the days before Ritalin was commonly prescribed) to hear of kids taking cocaine so that they could do well on exams, or attempting suicide if they did not do well.

The tests are everything!

Even in the not-so-great district I live in now, the kids have been called into assemblies and told that how well they do on the tests could mean the difference between people voting for bond issues and not. Do you want a new playground? Do badly, and you can forget it. Indoor recess and old computers and books. :(

Other school districts are even worse and tie the scores in to which teacher a child is assigned the following year. (Or maybe ours does and the others are just open about it.) One of my friends has the hope that her daughter will do well and get into the class she wants. If not, it's homeschool time.

Imagine the pressure on this SEVEN-year-old child who wants to stay in school with her friends. Hey, I love homeschooling, but I would be so bummed if the reason I were pulled is because I didn't ace ONE STUPID TEST.

It really makes me sad.

Respectfully Mrs. C., I must

Respectfully Mrs. C., I must disagree.

If we can't get a kid to handle the relatively minuscule pressure of a "fill in the bubble" standardized test, do we really think we are preparing them well for the world?

These kids are capable of much more than we give them credit for. I think they can handle a standardized test.

Where I think we might agree, Mrs. C., is that perhaps we put too much emphasis on standardized tests. Assessments are a valuable piece of information and can tell us much about how much kids are learning and how we can help them learn more. But they aren't perfect and they don't capture the whole child's successes.

Tests are important, but we've got some work to do on knowing their limitations, analyzing the data appropriately, and using the information from them as one piece of the puzzle in how we help kids learn.

Jason Glass
Eagle, CO

To clarify: *I* could care

To clarify: *I* could care less about the tests, but college admissions people do care. School districts care. Most parents care. Do you think the pressure to do well on the SAT is "miniscule" when teens attempt suicide over their scores? I think people in general care way too much, and it starts way too early. It used to be kids could be kids until about third grade or so. Now they start on 'em at a year old with the flashcards and smartie-kid gym programs; it's crazy.

I honestly think it's time to just instruct the children to get the zeroes on purpose, at least in the younger years. I wish kids could be just KIDS until high school. Maybe even junior year.

All this competition between schools - ELEMENTARY schools, even - is just stinkin' nuts.

I would agree that

I would agree that competition in the extreme, to the point of obsession, is unhealthy and damaging. But certainly this is not the case for all kinds of competition. Can we agree that there are lots of examples of healthy competition that spurs creativity, innovation, and growth? Our entire nation's economic system is premised on this idea.

I would assert in response to your question, Mrs. C., that a kid who commits suicide over a test score probably has a lot more going on in their life than the test. Implicit in your statement is the argument that tests cause suicide. I just think that's a stretch. There are certainly lots of flaws and things to debate about standardized tests. Them being suicide inducing is far down on my list.

I wish kids could just be kids too, and many of them can. But for the generations of kids in poverty our schools have failed for decades, we need a sense of urgency that just isn't and hasn't been present in the "good old days" view of education.

I do appreciate the response and thanks to Claus for writing a thought provoking piece.

Thanks for your respectful

Thanks for your respectful discourse, too, Jason. Tests don't cause suicide any more than bullying does. Some of the news stories "blaming" bullies for other people's suicides of late have been absolutely ridiculous. I hope you don't think I feel that way about test results... that's just silly. But that's not to say that there isn't a cultural backdrop in which all this occurs. :)

This topic will have far more

This topic will have far more meaning if you watch the TED video that "covers" this topic:

http://video.ted.com/talks/podcast/DanielPink_2009G.mp4

In short, RESEARCH-BASED knowledge shows that paying for performance hurts performance in complex situations. This isn't opinion; although, judgment is needed.

Yes Dennis, I've seen Pink's

Yes Dennis, I've seen Pink's videos and read his work. I'm a fan, though I don't agree with everything he says nor do I follow it blindly (not that I'm saying that you do!).

Again, my point as someone who actually works in a performance pay district, is that if these systems are designed with care they allow organizations to line up their finite resources with their organizational goals. In the case of schools, I do not care if people gain another year of experience or get more college credits. I care if kids are learning and teachers are providing great instruction. When those things happen I want to higher compensate those that make that happen.

In my opinion, the most professionally insulting thing we do to exceptional educators is to treat them like everyone else.

I want to use the dollars of the organization toward that end. The traditional "lock step" salary system just doesn't get us to that end.

Jason Glass
Eagle, CO

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options