The Progress Loophole of Despair
In 2016 – the first year of new national curriculum assessments in primary schools – the DfE made a bit of an error: the new progress measures neglected to make provision for those pupils that sat key stage 2 tests but failed to gain enough marks to reach the minimum scaled score threshold of 80. They were, of course, counted as having not reached the expected standard but without a score to work with, these pupils could not be included in progress measures. They were therefore treated differently to those pupils that didn’t sit the tests, who were instead assigned a ‘nominal score’ according to their pre-key stage level.
The DfE had stated in the original technical guidance that ‘it is important that schools are held to account and given recognition for the progress made by all pupils’; and the system of nominal scores was conceived to ensure as many pupils as possible were included in the headline progress measures, even if they were working below the standard of the tests. But they didn’t consider the possibility that pupils could sit tests but not gain enough marks to register on the scale; and rather than apply a nominal score, they omitted them from measures. We, therefore, had a bizarre situation where schools would benefit if pupils just sat there and did nothing. Get 3 marks or fewer and you’re not included in progress measures; get 4 marks or more…uh oh! I branded it ‘The Progress Loophole of Despair‘, and perhaps my blog on the subject was partially responsible for the closure of said loophole the following year (sorry!). In 2017, any pupil that scored below the minimum mark threshold to achieve a scaled score was assigned a nominal score of 79 – putting them in line with the highest of the pre-key stage assessments – and could therefore be included in progress measures. Joy!
That was one loophole; there are others but they are diminishing. Every year I get emails from senior leaders seeking clarification on which pupils are and are not included in measures and what – hypothetically speaking – are the most beneficial arrangements in terms of future headline figures. When the stakes are so high, it is understandable – perhaps inevitable – that these questions will be asked. Unfortunately, there is no simple answer. In short, it’s a minefield.
Who should participate in tests?
The Assessment and Reporting Arrangements (ARA) states that pupils should not take tests if any of the following apply:
- they have not completed the relevant KS2 programme of study
- they are working at the pre-key stage 2 standards or being assessed using the engagement model
- they are working at the standard of the KS2 tests but are unable to participate, even when using suitable access arrangements
The last two points are fairly self-explanatory – if a pupil is working below the standards of the key stage or is unable to access the test they should not be entered for them – but the first point is ambiguous and causes a lot of confusion. ‘Not completed the relevant KS2 programme of study’ is often taken to mean ‘has not completed all key stage 2 curriculum content’. This could result in schools thinking that a year 6 pupil who is, say, working within the year 4 curriculum should not sit the tests, but this is not the case. The ARA explains that ‘the tests are designed for pupils who have completed the KS2 programmes of study and are working at the overall standard of the tests. This means that if pupils are working above the pre-key stage standards, they should be entered for the tests.’ Again, the first sentence of that statement could easily be misinterpreted to mean that only those pupils that have completed all curriculum content should be entered for tests. The second sentence, however, makes it clear any pupil that is working above pre-key stage standards should participate and this includes our year 6 pupil working within the year 4 curriculum.
So what does that first bullet point mean? Pupils that ‘have not completed the relevant KS2 programme of study’ actually refers to those that have been held back a year, or perhaps started a year later than normal due to a late summer birthday. These are rare cases and pupils to whom this applies will be assigned an F code to indicate that they will take tests the following year when they reach the end of KS2.
Ultimately, ‘headteachers make the final decision about whether it is appropriate for a pupil to take the tests‘ but clearly the vast majority of pupils should participate, even if they are likely to achieve a few marks.
Who’s included in attainment measures?
There are two rules of attainment measures:
- Pretty much every pupil is included in attainment measures
- It’s almost impossible to get pupils removed from attainment measures
Initially, all pupils are likely to be included in a school’s percentages achieving expected and higher standards but the autumn checking exercise provides an opportunity to apply for removal from measures in certain cases. There are four categories:
- Admitted from abroad with English not their first language
- Pupil not at the end of Key stage 2 in all subjects
- Left school roll before the tests
- Other
The first case is by far the most common but criteria are strict and apply only if:
- they were admitted to an English school for the first time in the last 2 years; and
- they arrived from overseas prior to their admission; and
- English is not an official language of the country they came from; and
- English is not their first language
It is up to the school whether they want to exclude such children from their measures and, obviously, if they do well on tests and meet expected standards, it is unlikely they would seek to remove them.
The next point – pupils not at the end of KS2 in all subjects – is rather obscure but guidance makes it clear that ‘SEN pupils working below the standard of the tests or unable to access the tests cannot be discounted from the cohort list‘. Such pupils will be included as having not achieved expected standards. This scenario, therefore, most likely applies to those that have been held back a year or have been disapplied from aspects of the national curriculum. As mentioned above, The former will receive an F code to indicate that they will take tests in future. The latter case would, historically, have received D code to show that they have been disapplied from the national curriculum but this code – which has caused so much confusion in previous years – has been removed for 2021/22.
The next case – left the school roll before the tests – is not as straightforward as it seems. In most cases, if a pupil has left then the school would want them removed from the list. However, if a pupil changed school on or after the first day of the summer term, then the previous school can request that that pupil’s results are included in their measures rather than the new school’s. This scenario would likely arise if the pupil is a high achieving child. Otherwise, the new school will be accountable for the results and would need to apply to have them excluded via the checking exercise.
This leaves the ‘other’ option. This might include the recent arrival (i.e. On or after the first day of the summer term) scenario mentioned above, or some other seemingly valid case, but applications made under the ‘other’ category are rarely successful if we consider that pupils with SEND and those that are unable to access the tests for whatever reason cannot be discounted.
Who’s included in progress measures?
Far more pupils are omitted from progress measures than from attainment measures for the simple reason that valid results are needed at both ends. In other words, we need both a baseline (which, currently, is KS1) and KS2 results in reading, writing, and maths.
Progress is measured from a combined start point at KS1 (which defines the pupil’s prior attainment group) to the result in a single subject at KS2. Progress is not measured from reading at KS1 to reading at KS2; it is measured from reading, writing, and maths at KS1 to reading and KS2 (note that maths at KS1 is double-weighted to give it equal weighting with English). Pupils with no KS1 results or with invalid codes (e.g. A (absent), D (disapplied)) in all subjects have no usable start point and therefore cannot be included; but pupils with partial results at KS1 (e.g. valid assessments for reading and writing and an A code for maths) can be counted. Obviously, pupils that have arrived from overseas since KS1 will not have a baseline and will be omitted from progress measures. The same applies to pupils who, at the age of 7 (KS1 age), were:
- Living in Scotland, Wales or Northern Ireland
- Home schooled
- Attending an independent school
If they have no valid KS1 results they will not be included in progress measures. No assumption is made about their attainment and no nominal start point is assigned.
The rules are the same at the KS2 end: a valid result is required. This means a pupil needs a scaled score in reading and maths, and a teacher assessment in writing to be included in the progress measures. Pupils that are working below the standard of the tests and assessed as pre-key stage at KS2 will be assigned a nominal scaled score of 79 or below (depending on their PKS band) and this allows them to be included in the measures. In the case of writing, of which there is no test, all teacher assessments – not just pre-key stage but also WTS, EXS or GDS – are converted to nominal scores, a process that is not without its issues.
Note: nominal scores are for the sole purpose of calculating progress. They are not used in the calculation of a school’s average scaled score, which is based on test scores alone. This means that if a pupil does not sit the test and is instead assigned a nominal score of 79 then it will likely have negative impact on progress measures, but will not affect average scaled scores. If they sit the test and score 80, it will still have a negative impact on progress scores but will also adversely affect the average scaled scores measure.
Pupils with KS2 codes other than scaled scores from tests or nominal scores relating to pre-key stage bands and teacher assessments of writing will not be included in progress measures. These codes include:
- A – absent
- D – disapplied*
- F – pupil will take the test in the future
- J – just arrived (abilities not yet determined)
- L – left
- M – missing
- P – pupil has taken the test in the past
- Q – disregard (used in cases of maladministration)
- U – unable to access test
- X – lost Z – ineligible (pupil not at the end of KS2)
*D (disapplied) has been removed from reporting arrangements for 2021/22. It should only have been used in rare cases of pupils that were officially disapplied from the national curriculum. It does not refer to pupils ‘disapplied from tests’, who should have been marked as ‘B’ (below) on the test register, assigned an appropriate pre-key stage band, and then included in progress measures.
Whilst, in most cases, pupils with these codes will be counted as having not met thresholds and therefore included in attainment measures (unless an application to have them discounted is successful), they will result in the pupil’s omission from progress measures. To investigate this yourself, experiment with various combinations of KS1 and KS2 outcomes using the DfE’s pupil progress ready reckoner. It’s out of date now, but it will give you a good idea of how it works and you will instantly see that selecting any code other than a valid KS2 result generates a red ‘Pupil Excluded’ flag.
The big issue with this arcane system is the potential for manipulation, either through the deliberate depression of results at KS1 or through the misuse of certain codes (it happens!). An accountability system that relies on biased and possibly distorted teacher assessment is never going to be reliable – the high stakes tend to distort the very data it requires to function. Depression of KS1 results is well known – accepted perhaps – but the misuse of codes listed above is less well documented or discussed. I have encountered incidences of D codes used in error (an honest mistake that nonetheless did the school a favour), and of F codes misapplied in order to omit pupils from both attainment and progress measures.
As long as the system remains as it is, these problems will persist.
Change is needed.