6th December 2011
Claims that England has been plummeting down international pupil performance tables cannot be justified, according to a new study from the Institute of Education, University of London.
The coalition government has embarked on radical reform of the English school system, and has pointed to the country’s deteriorating performance in tests conducted by the Organisation for Economic Co-operation and Development (OECD).
However, an analysis of English pupils’ scores in international maths tests over the past 12 years has found that there is, in fact, no hard evidence of any decline in comparative performance over time. A clearer picture is unlikely to emerge until all the international survey data are compared with national test scores, the IOE study says.
The Prime Minister, David Cameron, and his deputy, Nick Clegg, highlighted England’s apparently dramatic slide down the PISA (Programme for International Student Assessment) rankings in their foreword to last year’s schools White Paper. The OECD’s three-yearly PISA surveys of 15-year-olds suggest that England fell from 8th to 27th in maths between 2000 and 2009. The falls in literacy (7th to 25th) and science (4th to 16th) have also been marked.
The Education Secretary, Michael Gove, has acknowledged that the White Paper was designed partly in response to the PISA findings, which the government sees as a damning assessment of Labour education policies. He told the House of Commons earlier this year: “The PISA figures … show that … the standard of education … offered to young people in this country declined relative to our international competitors. Literacy, down; numeracy, down; science, down.”
But the new analysis by Dr John Jerrim concludes that England’s changing position in international performance tables neither supports nor refutes policymakers’ calls for reform. He points out that while PISA indicates that England is slipping down the tables, another major study, TIMSS (Trends in International Maths and Science Study) reports that the maths scores of the country’s 13 and 14-year-olds rose in comparison with other nations between 1999 and 2007. He has calculated that the disagreement between PISA and TIMSS over the change in English pupils’ maths performance is much bigger than for any other country.
Why should two respected international surveys produce such contrasting findings? Dr Jerrim identifies several possible reasons.
Changing samples of pupils: The 2000 and 2003 PISA studies surveyed English 15-year-olds in two year groups (one third were in Year 10 and two thirds in Year 11). However, Dr Jerrim has established that almost all the English pupils who took part in PISA in 2006 and 2009 were in Year 11. Such a change might be expected to boost England’s PISA score but it is almost impossible to quantify its actual impact because the 2006 and 2009 PISA papers were taken much earlier in the school year.
Earlier test dates: The two most recent PISA tests were set in November or December rather than in March, April or May, as in 2000 and 2003. The Year 11 pupils who took the PISA tests in 2000 and 2003 consequently had about five months more schooling than Year 11 pupils tested in 2006 and 2009. This very significant date switch, which only occurred in England, Wales and Northern Ireland to prevent PISA tests clashing with GCSE exam preparation, has not been recognised in previous analyses.
Problems with the sample: Children and schools can refuse to take part in PISA and TIMSS. This was a particularly significant problem for England in the 2000 and 2003 PISA studies as many children with poor maths skills did not participate. This may have artificially raised England’s position in the international rankings in PISA 2000 and 2003 relative to that in 2006 and 2009. Dr Jerrim calculates that this problem alone could explain as much as half of the apparent fall in PISA maths scores over this period. Changes in the refusal rate could also help to explain England’s improving TIMSS results, he says, as the proportion of first choice schools taking part in this survey has increased dramatically.
More countries taking part: Dr Jerrim also notes that the number of countries covered by PISA rose from 43 in 2000 to 65 in 2009. This could also have contributed to England’s worsening PISA rankings as “it is easier to come tenth in a league of 43 than it is in a league of 65”, he says. Another complication is that several countries have only taken part in some TIMSS surveys. To overcome this problem Dr Jerrim examined the maths performance of 10 countries, including England, that have taken part in every PISA and TIMSS survey since 1999. This exercise showed that Japan, Hong Kong and South Korea, were consistently ahead of the other countries, while both surveys agreed that Indonesia was at the bottom of the scale. All that could be said about the other six countries -- Australia, Italy, the United States, Hungary, Russia and England – was that they were bunched together. Their exact positions were unclear, Dr Jerrim says.
He concludes that if we are to gain a better understanding of England’s comparative performance over time it will be necessary to link international survey data to records of pupils’ performance in national tests. “England is somewhat unusual in having national measures of performance collected at ages 7, 11, 14 and 16 and this could potentially be exploited to investigate at least some of the issues I have raised,” he says.
----------------------------------------------------------------------------------------------------------------
John Jerrim’s paper, “England’s ‘plummeting’ PISA test scores between 2000 and 2009: Is the performance of our secondary school pupils really in relative decline?”, will be published on the IOE website later today.
|
