Last summer, when the news trickled out that the floor standard would be kept at 65%, rather than being hiked up to to a mountainous 85% as originally proposed, the sense of relief was palpable. We already knew that the expected standard was going to be tougher to achieve than a level 4, and to not only raise the academic expectation on individual pupils but also the accountability threshold for schools seemed monumentally unjust. So, the DfE did the right thing and left the floor standard threshold where it was. After all, they wouldn’t want the majority of schools to be below floor, would they? We all cheered.
But now that sense of relief is ebbing away. First came the interim teacher assessment frameworks, then the primary school accountability technical document, and most recently the key stage 2 exemplification guides. Suddenly, for many schools, that 65% threshold looks like another country. And for some it’s simply off the map.
And so, many schools will be pinning their hopes on the progress measures, and it’s these that worry me. The first major distinction between the new and existing progress floor standards is that the new ones are VA-based, whereas the existing ones relate to the percentage of pupils making two levels of progress – so called expected progress. But unlike in previous years where a school that fell below the 65% L4 threshold only had to match one of the expected progress medians to be above floor, in 2016 any school that falls below the 65% expected standard threshold will have to match or exceed all three measures of ‘sufficient progress’, to be above floor. Now, it’s important to point out that because it’s a VA measure we currently have no idea what constitutes ‘sufficient progress’ (the DfE can’t calculate VA until all results are in) but we know the thresholds are likely to be negative. Note that, in future, a VA score of 0 will indicate average progress – replacing 100, which indicates average progress now – and so approximately half of schools will have negative VA scores, just as around half of schools have VA scores below 100 now. This is why they can’t set the ‘sufficient progress’ threshold at 0: there would be too many schools with negative VA scores in all three subjects. So, it’ll be negative – how negative we don’t yet know – and maybe statistical significance will have a part to play too. Furthermore, the DfE have suggested that in future they will set the sufficient progress threshold in advance, which will be helpful (sort of).
I don’t have a problem with VA per se. It’s about as fair as progress measures get because it inherently recognises that pupils make different amounts of progress from different start points in different subjects, and is therefore a lot more realistic than the ridiculously simplistic expected and better than expected progress measures that it’s replacing. VA, by its nature, does not involve a universal expected rate of progress.
I do however have a problem with the way VA is calculated for writing and have had since its inception. Writing VA is clumsy, and clunky and blunt. Unlike in reading and maths, where pupils achieve fine grades from their tests, in writing pupils are awarded a broad teacher assessment. This teacher assessment (usually L3, L4 or L5) is then converted to a point score (21, 27, 33). The weird thing about writing VA though is that pupils’ point scores from their teacher assessments are compared against an unachievable fine grade estimate, which represents the average KS2 result for pupils with the same KS1 prior attainment. The result is that there are wild swings in writing VA that you don’t see in the other subjects. This is demonstrated by the following table:
The table shows the actual results (whole levels) for pupils in writing, and the fine grade estimates against which their results are compared. The estimate represents the average result for pupils nationally with the same prior attainment at KS1. Pupils are almost without exception a long way above or below their individual estimates, resulting in big positive or negative VA scores that are nowhere near as common in reading and maths where pupils’ fine graded test scores ensure they can get close to their estimates. There are, for example, many pupils in the above table with estimates of 28-29 points (4a) that are awarded a L4 teacher assessment, which attracts 27 points, and who therefore have VA scores of -1 to -2. The only way these pupils can achieve a positive VA score is if they are awarded a L5. Their VA then shoots up to +4 or +5. Because the estimates are based on national average outcomes and more and more Level 5s are awarded each year, there has been a corresponding increase in the estimates to the point where L4 in most cases is worthless. Consequently, schools that err on the side of caution and award Level 4s are finding themselves with very low – sometimes significantly low – writing VA scores. I often see RAISE reports with positive VA in reading and maths and negative VA in writing, and others that have negative VA in reading and maths and positive VA in writing. No prizes for guessing which one makes me frown the most.
So, currently, writing VA is prone to big shifts into negative and positive territory as a result of how many L5s the school gives out and the 6 point swings this generates. Often, one more L5 can make the difference between a blue box and safety. And, having modelled various scenarios, it’s feasible (although somewhat unlikely) that a few more L5s can even turn a blue box green.
And what about this year? Will there be a change for the better? Probably not. We don’t know the details yet, but p15-16 of the primary accountability guidance provides the following with regards writing at key stage 2:
“Pupils will be allocated nominal points, based on their key stage 2 writing teacher assessment, which will be used in the progress calculation. This is done only in order to calculate a school’s progress scores. Pupils will still receive their teacher assessment as their key stage 2 outcome and no pupil will receive our nominal point score as their key stage 2 outcome. We will confirm the exact numbers that will be assigned to teacher assessment categories after the first set of assessments have been completed in the 2016 summer term.”
We have no idea what ‘nominal’ scores will be allocated to each of the 3 main teacher assessments or to the pre-key stage assessments, for that matter, but lets assume the nominal scoring is something like this:
working towards the expected standard = 90
working at the expected standard = 100
working at greater depth within the expected standard = 110
If this were the case – yes, I’ve made these up but it doesn’t seem too far fetched – then I can’t see how it will result in anything other than a continuation of the exaggerated gaps and skewed results we have now. Pupils with a particular start point, say a KS1 APS of 13, may have a KS2 writing estimate of 93. This represents the national average score for pupils with the same prior attainment and is calculated – if the nominal scores listed above were adopted – by adding up all the 90, 100, 110 scores for all pupils that have a KS1 APS of 13 and dividing by the number of pupils (there might be 50,000 of them). This unachievable estimate of 93 is below the expected standard but is above 90 so, in my example, if a pupil is assessed as working towards the expected standard they’d have a VA score of -3; if they are assessed as working at the expected standard they’d have a VA score of +7. Then imagine that happening across an entire cohort. Even with the new moderation arrangements there is clearly a temptation here to game it, to try to beat the system. If this happens then the following year the situation will get worse as averages, and the corresponding estimates, increase. Same as it ever was. Obviously, they could opt for scores on a separate scale, e.g. 1, 2, 3 for each of the three main assessments but a) a close scale would not adequately differentiate between pupils or do justice to the progress they make, and b) it’s still arbitrary.
So, writing, which many schools already fear will be the subject that pulls them below the 65% threshold, will continue to have a flawed VA methodology, which benefits those that push pupils up and punishes those that are perhaps more realistic or cautious. Considering how high the stakes are, I believe it’s time progress measures in writing were scrapped – VA should only be calculated using fine graded test data. The removal of writing from the baseline for secondary progress 8 measure next year speaks volumes about how much the DfE values teacher assessment so why are they continuing to use it in this critical accountability measure?
Either we make do with VA in just reading and maths or, if we have to have three progress measures, then perhaps it’s time the DfE did what they probably intend to do anyway: use the EGPS test scores instead.
Until that happens, writing VA is junk and the entire system of floor standards is a sham.