Join the conversation

For merit pay, I'd agree these systems are not amply able to distinguish between "bad" and "typical". But for the purpose of removing policies like LIFO or tenure, with a few years of data identifying the bottom 5-8% of teachers is actually not so bleak. With less than 10% error, we're looking at a pretty good, relatively objective measure of the academic teaching quality of teachers after three years. In five years, we're doing even better.

When many districts are looking at seniority layoffs that may tap into teachers on the job for nearly a decade, it seems very fair to me that we should look into these numbers based right on the IES report. The effects on students could be staggering (via Ed Week's Teacher Beat, an interesting analysis of seniority v. effectiveness layoffs http://is.gd/dKjis).

Another suggestion I may take into account based on the IES report is extending the time it takes to earn tenure. Instead of pursuing policies to allow us to remove lower performing teachers with this job protection, let's be much more strict about who gets in. Let's wait 5 years, get stable value-added scores and reject the lowest of the low teachers (bottom 5-10%) and put off the tenure decision a year. Then reevaluate and see where you are every two years. The decision is a big deal for the district and the teachers-- so let's get it right, use the data we have, and certainly not offer tenure to someone who's got only a 10% chance of being better than bottom of the barrel.

For ranking schools relative to one another, these value-added measures appear quite strong, and here I start to think about all the problems that go away when we start to think about value-added as a potential replacement for something like AYP which is fraught with issues. With errors in the 2 and 3 percent range after just a few years of data, it seems that identifying the "persistently low achieving schools" as required by SIG could have benefited quite a bit from value-added measures.

And of course, like you, I don't want any of these measures to be used in isolation for any decision. But despite the curmudgeon about this report, there's a lot in here we already know and a lot in there to be encouraged by, if your uses of value-added are not nefarious.

My final thought is this-- with all the issues value-added measures have, what are we going to use in its place? Do we currently have information that works better than value-added (or better than the potential of value-added) to make decisions on? I am defining information broadly, as anything that any person in the system would use to develop their personal rationale for making a choice.

I get the sense that the "error rates of perception" may be higher than anything we produce in value-added, but we still have to make decisions.

Here's how I look at it: Say you want to fire the lowest 5% of teachers every year, which is what some people are calling for. Say value-added scores count for 50% of teacher evaluation, which seems to be the gold standard for reformers. So maybe 10% of average teachers will get marked unsatisfactory every year and feel like they could be on the chopping block.

Maybe you'd have to look at many more years of data to have confidence. The 26% error rate is for 3 years of data. If you use only 1 year, the report sees a 35% error rate.

So is there any way to smooth out the data over time and make teachers feel like there's less chance involved in big decisions that could determine their fate?

"The 26% error rate is for 3 years of data."

It's also for identifying the bottom 18th percentile. If you go down to the 8th percentile (which is in the IES study; comparing teacher to school average has 16% error in 3 years and 10% with 5 years; comparing teacher to district average is 17% and 11% for 3 and 5 years respectively) the errors are far more acceptable.

So if we go down to the 5th percentile and use 5 years of data, I would say you have a pretty solid piece of data to make a up a large portion of these decisions.

As a policy-person, one idea I may have is develop a great observation protocol like the one in Cincinnati. Make sure everyone is evaluated 2 or 3 times a year for PD purposes. After three years, anyone who comes up as a "negative" on value-added should have additional observation and PD. If after 5 years that person is still identified as a "negative" in the value-added model, and the observation reports have some corroborating evidence, the teacher should be exited.

We could also imagine using a similar system that catches a few additional folks for making tenure. Award only preliminary tenure in 3 years at best, and only confirming it after 5 years of data.

Perfect? No. But the doom and gloom, fire and brimstone reporting I've read on this seems to assume the worst of people using and designing these systems and are probably hoping they can be used for something more sophisticated than simply separating out the bottom 5% from everyone else.

The authors really should have been a bit more nuanced about describing their results.

J Becker--Of course, there's a lot of gloom-and-doom reporting to go around. That was the bigger point of my somewhat rambling posting: The PR excesses could drown out thoughtful discussion. And I think there have been some PR excesses supporting reforms like like value-add.

Too many in the media or among the pundits have made far more of value-added measures than you have. The assumption in some quarters that "real" reform begins with evaluation systems based mostly on value-added scores  seems rather less cautious than your remarks do. The over-heated rhetoric has run far ahead of the reality.

And too often, those who raise questions in good faith are lumped together with mere obstructionists. That also stifles constructive debate.

It takes a lot of words to make people move. Aggressive PR seems right for the traditional people but still I find it unethical. People need to be convince from truths. They need true information unless you want people to go out as soon as they got in.