A primary school I work with has been on the same journey through assessment land as many other schools up and down the country. Around two years ago they began to have doubts about the tracking system they were using – it was complex and inflexible, and the data it generated had little or no impact on learning. After much deliberation, they ditched it and bought in a more simple, customisable tool that could be set up and adapted to suit their needs. A year later and they have an effective system that teachers value, that provides all staff with useful information, and is set up to reflect their curriculum. A step forward.
Then they joined a MAT.
The organisation they are now part of is leaning on them heavily to scrap what they are doing and adopt a new system that will put them back at square one. It’s one of those best-fit systems in which all pupils are ’emerging’ (or ‘beginning’) in autumn, mastery is a thing that magically happens after Easter, and everyone is ‘expected’ to make one point per term. In other words, it’s going back to levels with all their inherent flaws, risks and illusions. The school tries to resist the change in a bid to keep their system but the MAT sends data requests in their desired format, and it is only a matter of time before the school gives in.
It is, of course, important to point out that not all MATs are taking such a remote, top down, accountability driven approach, but some are still stuck in a world of (pseudo-) levels and are labouring under the illusion that you can use teacher assessment to monitor standards and compare schools, which is why I recently tweeted the following:
This resulted in a lengthy discussion about the reliability of various tests, and the intentions driving data collection in MATs. Many stated that assessment should only be used to identify areas of need in schools, in order to direct support to the pupils that need it; data should not be used to rank and punish. Of course I completely agree, and this should be a strength of the MAT system – they can share and target resources. But whatever the reasons for collecting data – and lets hope that its done for positive rather than punitive reasons – let’s face it: MATs are going to monitor and the compare schools and usually this involves data. This brings me back to the tweet: if you want to compare schools, don’t use teacher assessment, use standardised tests. Yes, there may be concerns about the validity of some tests on the market – and it is vital that schools thoroughly investigate the various products on offer and choose the one that is most robust, best aligned with their curriculum, and will provide them with the most useful information – but surely a standardised test will afford greater comparability than teacher assessment.
I am not saying that teacher assessment is always unreliable; I am saying that teacher assessment can be seriously distorted when it is used for multiple purposes (as stated in the final report of the Commission on Assessment without Levels). We need only look at the issues with writing at key stage 2, and the use of key stage 1 assessments in the baseline for progress measures to understand how warped things can get. And the distortion effect of high stakes accountability on teacher assessment is not restricted to statutory assessment; it is clearly an issue in schools’ tracking systems when that data is not only used for formative purposes, but also to report to governors, LAs, Ofsted, RSCs, and senior managers in MATs. Teacher assessment is even used to set and monitor teachers’ performance management targets, which is not only worrying but utterly bizarre.
Essentially, using teacher assessment to monitor standards is counter productive. It is likely to result in unreliable data, which then hides the very things that these procedures were put in place to reveal. And even if no one is deliberately massaging the numbers, there is still this issue of subjectivity: one teacher’s ‘secure’ is another teacher’s ‘greater depth’. We could have two schools with very different in-year data: school A has 53% of pupils working ‘at expected’ whereas school B has 73%. Is this because school B has higher attaining pupils than school A? Or is it because school A has a far more rigorous definition of ‘expected’?
MATs – and other organisations – have a choice: either use standardised assessment to compare schools or don’t compare schools. In short, if you really want to compare things, make sure the things you’re comparing are comparable.