Nic Spaull comment on pirls 2016 results “We underestimated how deep the crisis was”

5 Dec 2017
Nic Spaull comment on PIRLS 2016 results
“We underestimated how deep the crisis was”

“I think the thing that was most striking for me was that we had previously underestimated the number of South African children that couldn’t read for meaning. Previously we thought the number was 58% but it turns out that it is 78%. Basically we were using the wrong benchmark in the past. This is the first time that the easier PIRLS test (which used to be called prePIRLS and is now called PIRLS Literacy) was put on the PIRLS scale. “

“boys and girls”

“I think it’s really concerning that boys seem to be doing worse over time. Although we can’t be 100% sure of the decline there does seem to be some evidence that girls are doing roughly the same as they always have been doing and boys are actually doing worse”

“There is also evidence that the gap between boys and girls is growing over time. The gap in 2016 is bigger than it was in 2011 with girls scoring much higher than boys. In fact South Africa has the second largest gender gap of all countries that were assessed. Only Saudi Arabia has a larger gender gap (note these are all pro-girl gender gaps). In SA the average Grade 4 girl in South Africa is a full year of learning ahead of the average boy in Grade 4.”
“Mixed signals from the system”

“Something that is definitely confusing is that we seem to be getting mixed messages from the syste,. The Trends in International Mathematics and Science Study (TIMSS) released last year showed improvements in maths and science at the Grade 9 level. I think most of us were expecting an improvement in the PIRLS scores between 2011 and 2016 so it’s very disappointing to see now improvement between 2011 and 2016”

Silver lining

“I think the only piece of good news coming out of the PIRLS results is that it looks like there was an improvement in reading outcomes between 2006 and 2011. Because all the test scores are on the same scale (i.e. in the same metric) we can compare trends over time. It looks as if there was quite a big improvement between 2006 and 2011 but no improvement between 2011 and 2016. For example, the SA Grade 4 students in 2011 scored higher than the Grade 5 students in 2006.

“Declining number of high achievers in SA”

“ I think something that is also concerning is the decline in the number of high achieving students in South Africa. There are only a very very small number of students achieving the High International Benchmark and it seems to be declining. In 2011 there were 3% of SA Grade 4 students reaching the High Benchmark while in 2016 there were only 2% reaching this benchmark. In contrast, in England 57% of students reach the High International Benchmark and in Chile this is 25%.

Dr Nic Spaull is a Senior Researcher in the Research on Socioeconomic policy Group in the Economics Department at Stellenbosch University
You are welcome to quote any of the parts below and ascribe them to me as oral quotes:
If you do not already have an embargoed copy of the PIRLS 2016 report (450 pages) some of the highlights and most striking findings are included below. Please note all of these results are strictly embargoed until 11am SA time on 5 Dec 2017.
Background: PIRLS is implemented by the Centre for Evaluation and Assessment (CEA) at Pretoria University headed by Prof Sarah Howie. In 2016 it tested 12,810 Gr4 students from 293 schools across the country (PIRLS report page 309). The sample is nationally representative and can be generalized to the entire country. Students were tested in whatever language was used in that school in Grades 1-3, i.e. all 11 official languages were tested and children were generally tested in the langauge with . which they were most familiar. The results will be released by Minister Motshekga today (5 Dec) in Pretoria at 11am.
Main findings:

1. 
   8 of 10 SA children cannot read: 78% of SA Grade 4 students cannot read for meaning. That is to say that they could not reach the Low International PIRLS Benchmark in reading. They could not locate and retrieve explicitly stated information or make straightforward inferences about events and reasons for actions (PIRLS report page 55)
   
2. 
   SA scores last in reading of 50 countries: South African Grade 4 children have scored the lowest mark in the latest 2016 round of the Progress in International Reading and Literacy Study released today. The study included mostly High Income Countries but there were a number of middle-income countries such as Iran, Chile, Morocco, and Oman.
   
3. 
   SA lags far behind other countries: While 78% of SA Grade 4 kids cannot read, in America this is only 4% and in England just 3% cannot read. However the study also included middle-income countries. In Iran only 35% of Grade 4 students could not read for meaning and in Chile it was only 13% (PIRLS report page 55).
   
4. 
   Reading crisis deeper than previously thought: When South Africa participated in prePIRLS 2011 (an easier version of PIRLS) we thought that 58% of SA Gr4 children could not read for meaning. However this was on a separate test and not on the PIRLS scale score (i.e. not the same metric). 2016 was the first time that prePIRLS (now called PIRLS Literacy) was put on the same scale score as PIRLS. The true figure for children that cannot read for meaning is 78% - revealed today. Note this does NOT mean that reading outcomes have gotten worse between 2011 and 2015. In 2011 77% could not read for meaning and in 2016 78% cannot read for meaning (this difference is not statistically significant, i.e. the difference is negligible).
   
5. 
   Some evidence of improvement in reading 2006 to 2011 but stagnant since 2011: The only good news coming out of PIRLS 2016 is that there may have been significant improvements in reading between 2006 and 2011. Because the scale scores are now comparable we can compare the performance of Gr4's in 2006 and Gr4's in 2011 and 2016. This comparison seems to suggest quite a significant increase in reading scores between 2006 and 2011. Notably the Gr4 students in 2011 achieved higher scores than Gr5 students in 2006. Further analysis is needed but there does seem to be legitimate evidence of improvement between 2006 and 2011. Unfortunately no evidence of improvement between 2011 and 2016. 
   
6. 
   SA reading scores stagnant since 2011: There has been no improvement in reading scores over the last five years (i.e. 2011 to 2016). Note that although the average scored declined from 323 to 320 this can NOT be interpreted as a decline. The standard errors overlap here so there is no certainty that there was any decline whatsoever (this is like taking your sitting heart rate 10 times and getting very tiny differences each time - they are not statistically significantly different) (PIRLS report page 29)
   
7. 
   SA gender gap in reading 2nd highest in the world: Girls score much higher than boys in reading across the board. In Grade 4 girls are a full year of learning ahead of boys. This gender gap is the second largest among all 50 countries that participated. Only Saudi Arabia's is higher. (PIRLS report page 36). The gap between boys and girls is also growing over time. The gap between boys and girls was larger in 2016 than in 2011 (PIRLS report page 43).
   
8. 
   SA boys scores seem to have declined between 2011 and 2016: The average Grade 4 girl in SA scored 341 in 2011 and 347 in 2016 (unlikely to be statistically significant). The average Grade 4 boy in SA scored 307 points in 2011 and 295 points in 2016 (this is likely to be statistically significant but we cannot tell until the SA report is released  (PIRLS report page 43).
   
9. 
   Declining number of SA students reaching high levels of reading achievement: In 2011 3% of SA Gr4 students reached the High International Benchmark. In 2016 only 2% reached this same benchmark  (PIRLS report page 58).
   

The full report is available on the PIRLS website from 11am today: - http://pirls2016.org/pirls/summary/ I will also link to the full report on my blog and Twitter account at 11:00am. 
If you would like additional comment you can email me on this email address. (Comment ONLY via email. Please do not phone).