Learning First Alliance

Strengthening public schools for every child

Learning from the Leaders: A Conversation with International Assessment Guru Andreas Schleicher

vonzastrowc's picture

SchleicherWEB.jpgAmericans often hear about the United States' lackluster showing in international comparisons of student performance.  They hear less about education policies and practices in countries that top the international lists.  As it turns out, U.S. education policies--particularly our accountability policies--are often out of step with policies in the most successful nations.

This is one conclusion we draw from our recent discussion with Andreas Schleicher, who heads the OECD's Education Indicators and Analysis Division in Paris.  Schleicher oversees the Programme for International Student Assessment (PISA), a test often cited in reports about American students' decline in international rankings.

During our interview, Schleicher delivers some familiar bad news:  U.S. performance on PISA is below average for the OECD.  Socio-economic status has a larger impact on student achievement in the U.S. than in countries that top the PISA rankings.

The differences between the United States and its high-performing peers do not end there.  Schleicher suggests that the U.S. diverges from its peers by relying largely on "external accountability"--tests and consequences for poor performance--to improve schools.  Other countries, by contrast, do more build schools' capacity for success, and they use a variety measures to gauge their progress.  Standardized tests are important to the high performers, Schleicher tells us, but so are other evaluation strategies such as "school inspection systems, peer reviews, and self-evaluations." 

High-performing nations also tend to favor what Schleicher calls "lateral accountability" by "creating a knowledge-rich environment for the actors in education themselves rather than just comparing [schools]".  In other words, other countries build educators capacity

How do you build various ways in which networks of schools stimulate and spread innovation, connect to themselves, collaborate among themselves?  How do you make sure that teachers don't work in isolation, schools don't work in isolation, but really collaborate as a knowledge-rich profession?

Among the other important points Schleicher makes in the interview:

  • The top-performing countries generally combine universal academic content standards with an "appropriate, good environment for the standards to become a reality in the classroom."
  • Top performing nations "personalize" education by eliminating tracking policies and offering very early interventions for struggling students.
  • Top performers "start very early on.  They have virtually universal early childhood provisions [and] very strong diagnostic systems to identify disadvantages and gaps in learning outcomes."
  • Private schools perform no better than public schools in the United States--when you adjust for socio-economic background

Our Schleicher interview offers many more insights than we can report here, so be sure to listen to the interview.  You can download the entire recording here.

A transcript of highlights appears below. 

You can also hear any of the following interview excerpts:

How do we stack up?  Overall performance and education equity. (2:18)

How do high-performing countries serve students in greatest need? (2:52)

How have countries raised their international standing? (3:34)

What are the ingredients of success?  Learning from high performers. (4:54)

What about school capacity?  A broader view of school accountability. (2:39)

How do public schools compare to private schools? (2:01)

Final thoughts:  Equity and excellence need not be mutually exclusive. (1:44)


Transcript of interview highlights:
PUBLIC SCHOOL INSIGHTS:  Describe the Program for International Student Assessment [PISA] and its goals.

SCHLEICHER:  What we're trying to do is to look at the results that education systems achieve in terms of learning outcomes.  The interest really is to see to what extent students can extrapolate from what they have learned and apply their knowledge in novel settings.  So creative problem-solving is a very important and central aspect of this.  Of course, all of this is to gain some insights into some of the factors that explain the success of students, schools, and nations in education.

PUBLIC SCHOOL INSIGHTS:  How has the U.S. done on this assessment, relative to other countries in the OECD?

SCHLEICHER:  Actually, the U.S. performs slightly below the average of the principal industrialized countries.  We also see that socioeconomic background has somewhat more distinct impact on the success of individuals, which raises some equity-related issues.

PUBLIC SCHOOL INSIGHTS:  Have you been able to determine what some common practices have been in the OECD countries [that succeed] on PISA?

SCHLEICHER:  First of all, the comparisons show that outcomes cannot simply be tied to money.  So we do see a broadly positive relationship between investment and outcomes, but it's not as clear.  When you look beyond money, one of the things that you see is that in the top-performing education systems there are two things that go hand in hand.

First of all, they set ambitious, rigorous, and universal standards for all students.  That's only one side of the coin.  The other is that those countries tend to create an appropriate, good environment for those standards to actually become a reality in classrooms--to provide access to best practices and professional development in schools. 

For example, [successful countries] get the right people to become teachers.  That's, I think, one of the key attributes here, because the quality of an education system cannot exceed the quality of the teachers. For example, you see countries like Finland and Korea--two of the top performers--that recruit their teachers from the top 10 percent of graduates.  They develop those teachers into effective instructors.  That involves things like coaching classroom practice, moving teacher training to the classroom, developing strong school leaders... All of those are attributes that we tend to see [in the most successful countries].

There's another trend where we see [the most successful countries] having an appropriate balance with the enabling conditions for local responsibility in schools.  You see a lot of room for creativity and responsibility at the level of schools in those countries.  Decisions are made at the level of those most able to implement them.  That's sort of the second overall dimension, the balance between local responsibility and accountability.

And the third is about how do teachers--how do schools--succeed in personalizing the education system?  We do see that some of the successful education systems don't make extensive use of tracking and training; rather, [they] give teachers and schools more means, more options, to actually support students in their differences.  They have good [professional] support systems, so that individual teachers diagnose and become aware of the specific weakness of their own practices.

On external accountability: test-based systems are often part of the accountability systems in other countries, too, but we do often see greater emphasis on, for example, school inspection systems, on peer reviews, on self-evaluation...sort of on a broader mix of evaluation and accountability systems, with the primary purpose of not just public accountability and sort of the contestability of public service, but often greater emphasis on school improvement.  How do teachers get access to the information they need to improve their teaching?  How do students and parents get access to the information to see sort of where they stand, how do schools understand what the neighboring schools are doing?  The emphasis is often more on creating a more knowledge-rich environment for the actors in education themselves, [rather] than just comparing. 

But, overall, test-based systems are certainly a very important part [of accountability systems] in most of the countries.  You see, the point is often not just measuring where schools are and having vertical systems of accountability, but one of the key terms used in, for example, Belgium, Canada, Finland...is that "lateral accountability"-basically, how do you build various ways in which networks of schools stimulate and spread innovation, connect with themselves, collaborate among themselves?  So, how do you make sure that teachers don't work in isolation, schools don't work in isolation, but really collaborate as a knowledge-rich profession?

Some of the countries that are very successful in educational outcomes as measured by PISA, they start very early on.  They have virtually universal early-childhood provisions, very strong diagnostic systems to identify disadvantage and gaps in learning outcomes...That is certainly part of the policies in terms of the successful education systems.

PUBLIC SCHOOL INSIGHTS:  I wonder whether you have any information about the relative success of private versus public schools in OECD countries.

SCHLEICHER:  Yeah, we do measure this.  The results sort of are not very startling.  We do see, in most countries, a clear advantage of private schools over public schools, but once you account for socioeconomic conditions, actually that difference completely disappears.

PUBLIC SCHOOL INSIGHTS:  Have you had similar results in your findings about U.S. schools?

SCHLEICHER:  Yes, yes.  The result is quite consistent.  Basically, once you take into account socioeconomic conditions, public and private schools have identical results. 


Photo from Schleicher's OECD biography

Skepticism about PISA

Finland scored #1 on the PISA exam.  However, there is a published letter signed by 207 Finnish university, polytechnic (applied sciences) and technical university mathematics teachers warning about the totally unsatisfactory and, indeed, worsening performance of Finnish students when they get to the universities.  Here is a key quote:

The results of the PISA survey (http://www.jyu.fi/ktl/pisa/) have brought about satisfaction and pride in Finland. Newspapers and media have advertised that Finnish compulsory school leavers are top experts in mathematics.

However, mathematics teachers in universities and polytechnics are worried, as in fact the mathematical knowledge of new students has declined dramatically. As an example of this one could take the extensive TIMSS 1999 survey, in which Finnish students were below the average in geometry and algebra. As another example, in order not to fail an unreasonably large amount of students in the matriculation exams, recently the board has been forced to lower the cut-off point alarmingly. Some years, 6 points out of 60 have been enough for passing.

This conflict can be explained by pointing out that the PISA survey measured only everyday mathematical knowledge, something which could be - and in the English version of the survey report explicitly is - called "mathematical literacy"; the kind of mathematics which is needed in high-school or vocational studies was not part of the survey. No doubt, everyday mathematical skills are valuable, but by no means enough.

Out of the 85 assignments in the survey about 20 have been published. The assignments are simple numerical calculations, minor problems or deductions, interpretation of statistical graphics and evaluation of situations where text comprehension is an essential part. However, hardly any algebra or geometry is included. Nevertheless, the assignments are well in agreement with the goals of the survey; in fact, the goal was to study everyday mathematical knowledge.

Two of the authors expanded on these remarks as follows:

There are positive aspects in mathematical knowledge and teaching in Finland. The success of basic school pupils in the practice-oriented numerical problems of the PISA study is fine. A contributory factor to this success is basic school mathematics books, which include excellent examples of everyday life. In addition to the compulsory courses, upper secondary school students have the possibility to deepen their knowledge in good optional mathematics courses. In Finland, the teachers are known to be motivated and they have
obtained a good education.

However, it is undeniable that new students in universities and in polytechnics have poor mathematical skills on the average.

In August of this, year three professional mathematicians evaluated the items on the PISA Mathematics test for a study funded by the National Center for Education Statistics in the U.S. Department of Education.  All three agreed that the types of questions in this item set did not test mathematical knowledge and problem solving ability as it relates to the kinds of situations that come up in the workplace or in our universities, but rather the more mundane and low level problems that come up in everyday situations.

In short their analysis of the entire set of PISA test items in mathematics confirms the observations quoted above by the Finnish university mathematicians.

I would suggest that all students should do better than our students on the kinds of problems that PISA tests just for everyday survival.  However, our greatest need is for high school graduates capable of high level mathematical reasoning both in the workplace and as the single most important factor in predicting success at our universities.  (See The Toolbox Revisited, Clifford Adelman, www.ed.gov/rschstat/research/pubs/toolboxrevisit/toolbox.pdf)

PISA does not measure the kind of reasoning that is crucial to our economic well-being, and it is very, very important that we all understand this distinction.  Indeed, as poorly as our students perform on PISA, they perform as badly or even worse on the kinds of questions that actually tend to measure the problem solving abilities crucial in these non-routine areas.

The three mathematicians also evaluated the TIMSS 12th grade mathematics item set for the same study.  Their view was that a significant number of the TIMSS questions were good measures of the more important training mentioned above.  

The three mathematicians also indicated that even these TIMSS questions were at a disturbingly low level.  They get at no more than the beginnings of the the aspects of education that matter in the workplace and at the universities. As a result our students dismal showing on the 12th grade TIMSS exam becomes even more of a concern.  Indeed, I am far more worried by our TIMSS results than our below average performance on PISA.

R. James Milgram
Professor of Mathematics
Stanford University