It was great to read in Ofsted’s March update that “Ofsted does not expect any prediction by schools of a progress score, as they are aware that this information will not be possible to produce due to the way progress measures at both KS2 and KS4 are calculated.” Sean Harford went even further in his blog, describing the process of predicting progress as a ‘mug’s game’. This is welcome guidance from Ofsted and shows that they understand the complexities of value added measures in comparison to the old levels of progress measure.
However, whilst Ofsted won’t be asking for such data, I recognise that schools still like to have an idea of progress scores before they pack up for the summer, especially with floor standards link to these measures, and that’s why I produce the VA calculator. It’s a free tool and I’m happy for all primary schools to use and share it. It’s available in two formats: old school excel, and new-fangled web tool. Feel free to have a play around with both.
.
But, if you do use it, it’s important that you understand its limitations, which I’ve outlined below in order of impact and likelihood of change.
1) Estimates
Value Added measures involve comparing each pupil’s actual result against an estimated result. The whole school progress score is the average of the differences between the actual results and estimates. An estimate is the national average score for pupils with similar prior attainment in that particular year. For example, let’s take a year 6 pupil that was 2c in reading, writing and maths at KS1. They have a KS1 APS of 13 and are in prior attainment group 9. The DfE have to identify all pupils nationally with the same KS1 APS score, and they calculate the average scaled score for this prior attainment group in order to generate the estimates . In 2016, pupils in this prior attainment group son average scored 97.26, 96.69 and 98.33 in reading, writing and maths respectively (if you want to know more about the crazy world of writing progress, read this). Our 2c pupil’s KS2 scores are therefore compared against these benchmarks – if they score higher then they get a positive score; if they score lower they get a negative score. The issue with trying to work out progress for the current year 6 cohort is that we are comparing them against last year’s estimates, whereas we should be comparing them against the national average score for pupils with similar prior attainment in the same cohort. We don’t have this data yet and won’t have it until September. Judging by the overall improvement in results nationally, we can safely assume that the estimates for each prior attainment group will change and will no doubt rise in most if not all cases (they may drop for lowest PA groups because DfE intend to introduce data from special schools to mitigate this problem).
Verdict: definite change
2) Standard deviations
Standard deviations change every year, and these form part of the calculation of confidence intervals on which statistical significance depends. This means your data might not be significantly above or below average on the VA calculator but might be when we get the proper data in September. Or vice versa.
Verdict: definite change
3) Pupils that are entered for tests but do not score will be assigned a nominal score
This was covered in the progress loophole of despair post here: pupils assessed as HNM or EXS in reading and maths that sat the test but did not score enough marks to achieve a scaled score were excluded from progress measures last year. It looks like this particular loophole will close this year, which is a good thing, but we have no idea what nominal score these pupils will be assigned. I’m voting for 79 (1 point below the lowest scaled score of 80). The loophole was a particular issue in reading last year, when over 3000 pupils were entered for tests but did not achieve enough marks to gain a scaled score (compared with around 300 in maths). It is likely to be a much smaller number this year, but it will still affect quite a few schools.
Please note: this specifically relates to pupils assessed as HNM or EXS that did not manage to achieve a scaled score. Pre-key stage pupils that were entered for tests (for whatever reason) and did not manage to achieve a scaled score had a fall back, nominal score and therefore were included in progress measures (see No. 4 below).
Verdict: definite change
4) Floor standards & Coasting
Not really related to the VA calculator but something schools will have their eye on. Last year the progress floor thresholds were set at -5, -7 and -5 for reading, writing and maths respectively. These were incredibly low and reflected the fact that attainment was low (they just want the right amount of schools below floor after all). These threshold will go up this year, but by how much is anyone’s guess. Same applies to coasting – they just halved the floor standards, remember? – unless they scrap the coasting measure. Please let them scrap the coasting measure. I assume we’ll get the new floor and coasting thresholds at some point in the autum term but for some reason they don’t apply them in the Ofsted dashboard until validated data is released.
Verdict: almost certain change
5) Nominal scores for writing
Currently, pupils are assigned the following nominal scores according to their writing teacher assessment:
WTS: 91
EXS: 103
GDS: 113
It is likely that these will change, which will have an impact on progress scores. If they go up, then that will mitigate the inevitable increase in the estimates; if they stay the same then there will be a negative impact on overall progress scores. For what it’s worth, I’d like to see the writing progress measure scrapped because it’s frankly ridiculous.
Verdict: change likely
6) Nominal scores for pre-key stage pupils
Last year, pre-key stage pupils were assigned nominal scores according to their specific teacher assessment, which caused a huge amount of damage to the progress scores of those schools that had pre-key stage pupils. These scores, used for progress measures only, are as follows:
BLW: 70
PKF: 73
PKE: 76
PKG: 79
We have no idea if these nominal scores will change but I suspect they won’t because increasing them will encroach upon the actual scale score range, which doesn’t make sense, and decreasing them will further penalise schools with low attaining, SEN pupils. Obviously, any change to these scores will have an impact on progress calculations.
Verdict: change unlikely
NB: If you want to recalculate progress with pre-key stage pupils removed, this is something the VA calculator can help with.
7) Prior Attainment Groups
Currently there are 21 prior attainment groups, as detailed on p17-18 of the Primary Accountability Guidance. These primary attainment groups (PAGs) are integral to the progress measures and are, like the other factors detailed above, a vital part of progress calculations and the VA calculator. They are the bolts onto which the estimate nuts fix. If these change then it will fundamentally alter structure of the progress measures, and make latest data incomparable with last year. I suspect they will remain the same.
Verdict: change unlikely
We will have to wait for the release of progress data in September to get answers to these questions. In the interim, the VA calculator can be used as a guide to indicate whether progress is well above or below average, or broadly in line. We will of course update the VA calculator with 2017 estimates in September so you can use it to accurately recalculate progress with specific pupils removed. I will have more confidence in the forecasts next year, when the estimates are based on this year’s more reliable results, and many of the issues listed above have been resolved.
So use it and share it; just be aware that data is likely to change.