The following notes relate to some work I’m doing to support a junior school with a bit of a baseline issue. A case of significantly high prior attainment, good KS2 results, and abysmal progress. So, I’ve been trying to build a case for the defense against the duality of the RAISE report – a sort of lush green lawn of attainment next to a deep blue pond of progress. Obviously, the head has given their permission for me to blog this. Hopefully it will be useful or interesting or both.
This is not intended as an infant school bashing piece by the way. Some junior schools have a baseline issue, some don’t.
Junior School A is characterised by intakes that are consistently significantly above national average on entry. The 2014 year 6 cohort comprised 57.4% high attainers – more than double the national proportion of 24.7% – constituting the largest proportion of high attainers the school has ever had. Percentages of low attainers are correspondingly low, ranging from 3.5 to 9.3%. Attainment is generally above or significantly above national average but progress, in terms of VA is significantly below.
VA scores for 2014 cohort are as follows:
All subjects: 97.8 (sig- and a significant decline)
Maths: 96.9 (sig- and significant decline)
Reading: 98.7 (sig-)
Writing: 98.7 (sig-)
VA is the comparison between a pupil’s actual attainment and a calculated estimate for the end of KS2, the estimate being based on national average attainment for pupils with the same prior attainment. The validity of the estimate is entirely dependent on the accuracy of the baseline used to generate the estimates, i.e. the KS1 results.
Our analyses question the validity of the baseline used to calculate VA. We have compared VA estimates based on the cohorts’ key stage 1 results against VA estimates based on the school’s own baseline taken in year 3, and against the predictions generated by CATS tests taken in year 5. The differences are considerable and need to be taken into account when assessing progress made by this cohort.
Key Stage 1 vs School’s own Year 3 baseline
The school carries out a baseline assessment early in year 3 and this conflicts substantially with the key stage 1 data. The following table shows percentages of the 2014 year 6 cohort in each attainment band based on KS1 data and the school’s Y3 baseline:
|
|
Based on KS1
|
Based on Y3 baseline
|
|
Low (APS <12)
|
5.6%
|
14.8%
|
|
Middle (APS 12-18)
|
37.0%
|
79.6%
|
|
High (APS >=18)
|
57.4%
|
5.6%
|
The percentage of high attainers is considerably lower when using the year 3 baseline data in place of KS1 results.
Value added
We then generated VA estimates for the 2014 year 6 cohort by entering their year 3 baselines into a VA calculator tool (using RAISE 2014 methodology) and compared these estimates to those produced from the KS1 results. The following table shows the attainment, in terms of APS, that the 2014 year 6 cohort were required to achieve on average in order to obtain a VA score of 100 (i.e. make average progress). In each case, the estimated (expected) attainment based on KS1 results is far higher than that based on the year 3 baseline.
|
|
Reading
|
Writing
|
Maths
|
Overall
|
|
Based on Y3 baseline
|
28.3
|
27.5
|
27.5
|
27.7
|
|
Based on KS1 results
|
31.0
|
30.6
|
32.3
|
31.5
|
|
Difference
|
2.7
|
3.1
|
4.8
|
3.8
|
The APS difference indicates the additional progress the cohort is required to make in order to ‘break even’. Using the VA tool we then calculated revised VA scores where each pupil’s actual attainment is compared against the lower, more realistic estimate generated from the year 3 baseline rather than the KS1 result. This shifts the VA from significantly below to significantly above average. The table below compares the revised VA scores, based on the Y3 baseline, against VA presented in RAISE. The values in the bottom row show the differences between the two VA scores and can be construed as the progress that is lost when key stage 1 results are used as the baseline.
|
|
Reading
|
Writing
|
Maths
|
Overall
|
|
VA score based on Y3 baseline
|
100.8
|
101.1
|
100.8
|
100.9
|
|
VA score based on KS1 results
|
98.7
|
98.7
|
96.9
|
97.8
|
|
Difference
|
2.1
|
2.4
|
3.9
|
3.1
|
Average differences between raw VA scores (i.e. unadjusted data before shrinkage factor is applied) are even greater. This is perhaps the best indication of potential lost progress in the school’s VA data.
|
|
Reading
|
Writing
|
Maths
|
Overall
|
|
APS difference
|
2.3
|
2.6
|
4.3
|
3.4
|
Key Stage 1 data vs CATS tests
In order to validate the estimates generated from the year 3 baseline, we compared them against the predictions derived from the CATS tests taken by the 2014 year 6 cohort in October of year 5. We converted the sublevel predictions from the CATS test to APS and compared them with VA estimates generated from the key stage 1 results. The estimates and differences are presented in the following table:
|
|
English
|
Maths
|
Overall
|
|
CATS predictions (APS)
|
27.4
|
27.6
|
27.5
|
|
VA estimates (from KS1 results)
|
30.8
|
32.3
|
31.5
|
|
Difference
|
3.4
|
4.7
|
4.0
|
NB: 2014 RAISE does not provide estimates for English. For purposes of comparison, the average APS of reading and writing was used.
The CATS predictions closely match the estimates derived from the Y3 baselines shown above and therefore help to substantiate the school’s on entry assessment. If the school’s Y3 baseline is accurate, as indicated by the CATS data, then there is compelling evidence to suggest that this cohort made far more progress than official RAISE data implies (between 2 and 4 points based upon the VA differences presented above). The school’s Y3 baseline should therefore be used in place of key stage 1 data for an reliable assessment of pupil progress.
Level 5 attainment
Attainment at both Level 4 and Level 5 is above national average for all groups and all subjects with the exception of disadvantaged pupils and high prior attainers at level 5. According to RAISE there were 31 high prior attainers in the 2014 Year 6 cohort, and the level 5 attainment of this group was below average overall and significantly below average in writing. However, based on the school’s Y3 baseline data only 3 pupils are included in this group and all of them achieved level 5 in each subject. The blue (sig-) box on p21 of the RAISE report, relating to a high prior attainment group comprising 31 pupils, should therefore be discounted.
Using the Y3 baseline, the size of the middle prior attainment group increases from 20 pupils to 43. The level 5 outcomes for this group are presented in the table below:
|
|
L5+ MRW
|
Maths
|
Reading
|
Writing
|
|
RAISE MAP (20 pupils)
|
5%
|
15%
|
45%
|
30%
|
|
School MAP (43 pupils)
|
35%
|
44%
|
65%
|
54%
|
|
National average
|
13%
|
36%
|
46%
|
24%
|
These figures for the revised middle prior attainment group are considerably higher than level 5 data in RAISE and are above national average in all subjects.
Disadvantaged pupils
A higher than average proportion of disadvantaged pupils achieved level 4 in all subjects, and above average proportions reached level 5 in reading and writing but only 1 did so in maths; and no disadvantaged pupils attained level 5 in all subjects combined. Further analysis of this group using the Y3 baseline data reveals that none of this group were high attainers at key stage 1, six were middle attainers, and five were categorised in the low prior attainment group. This contrasts with official data where numbers in the high, middle and low prior attainment groups were 3, 6 and 2 respectively. Moreover, the Y3 baseline data indicates that no pupils had a KS1 APS above 16 and so it is perhaps less likely that such a group would attain level 5. The fact that some did achieve level 5 is therefore notable when their on entry assessments are taken into account.
The following table summarises percentages of disadvantaged pupils making expected and better than expected progress; and compares against national benchmarks for other pupils. Figures increase when progress is measured from the year 3 baseline and are closer to national comparators. The percentage of disadvantaged pupils making 3LP is particularly high in reading.
|
|
2LP
|
3LP
|
||||
|
|
RAISE
|
Y3 baseline
|
National non-FSM
|
RAISE
|
Y3 baseline
|
National non-FSM
|
|
Reading
|
73%
|
82%
|
92%
|
36%
|
64%
|
34%
|
|
Writing
|
82%
|
82%
|
94%
|
27%
|
36%
|
34%
|
|
Maths
|
73%
|
82%
|
91%
|
9%
|
27%
|
38%
|
Levels of progress for 2014 Year 6 cohort
The following table shows revised percentages making expected progress and better than expected progress using the school’s Y3 baseline. 2LP figures are above 90% and in line with national figures, whilst %3LP figures are above 40% and higher than national averages.
|
|
2LP
|
3LP
|
||||
|
|
RAISE
|
Y3 baseline
|
National
|
RAISE
|
Y3 baseline
|
National
|
|
Reading
|
76%
|
96%
|
91%
|
24%
|
46%
|
35%
|
|
Writing
|
89%
|
91%
|
93%
|
33%
|
46%
|
33%
|
|
Maths
|
85%
|
94%
|
89%
|
7%
|
41%
|
35%
|