Thursday, 20 July 2017

Making the cut

Assessment. It should be simple really: just checking what pupils do and don't know. But assessment appears to have turned into some kind of war, with the legions of accountability amassed on one side, and the special forces of teaching and learning besieged upon a slippery slope with nowhere to go. We become so focussed on outcomes - on floor standards, and coasting thresholds, and league tables - that we risk losing sight of what's important: the here and now. But it is focussing on the here and now that makes all the difference. The irony is that concentrating on accountability - on those distant and unpredictable performance measures - can jeopardise the very results you are striving for. In short, focus on teaching and learning, and the results take care of themselves.

This is, of course, easier said than done but something has to change. In too many schools assessment has become a burden: a top down directive disconnected from learning; an interminable, box-ticking, data-collecting, drain on teachers' time. The risks are clear: morale nose dives and pupils' learning is put at risk. We therefore need to ditch some of our assessment baggage - aim to do more with less - and this requires some serious rationalisation of our processes. It all comes down to one simple question:

Does this have a positive impact on learning?

We need to go through everything we do in the name of assessment and school improvement and ask that question, and we need to be ruthless and honest. What is the benefit and what is the cost? How long does this take? Does it tell us anything we don't already know? Is it having a negative impact? Is it taking teachers' time away from teaching? Ultimately, the only way to improve a school is to teach children well, and anything that distracts from that purpose is a risk.

So let's deconstruct our entire approach to assessment and lay it all out on the hall floor: the various tests you use, your marking policy, target setting (both for pupils and performance management), those lists of learning objectives stapled into pupils' books, and the component parts of your tracking system (yes! every single measure, category, grid, table, graph, chart and report). We now separate these into two piles: those that have a demonstrable, positive impact on teaching and learning, and those that are purely done for the purposes of accountability.

We keep the first pile and ditch the rest.

We now have a stripped down system that is fit for purpose, that is focussed on the right things. From now on, the information we provide to governors and external agencies is a byproduct of our assessment system, which exists to serve teaching and learning alone. If it works, it's right, no matter what others may say. Many will try to convince you that you're mad but deep down they probably just wish they could do the same. If you think this is all too radical, it's really not. There are many schools with extremely minimalist approaches to assessment that have had very successful inspections. Just as long as your approach is informative and has impact, then it's fine. If anything, the simpler the better. And Ofsted are not asking you to generate data purely for their benefit anyway. The Handbook states:

Ofsted does not expect performance- and pupil-tracking information to be presented in a particular format. Such information should be provided to inspectors in the format that the school would ordinarily use to track and monitor the progress of pupils in that school

And the workload review group report on data management had this to say:

Be ruthless: only collect what is needed to support outcomes for children. The amount of data collected should be proportionate to its usefulness. Always ask why the data is needed.

In alpine climbing there are two popular adages: 'if in doubt, leave it out', and 'if you haven't got it, you can't use it'. The first one is obvious, and it's what I'm trying to get schools to think about when they go about rationalising what they do. The second one links to it and recognises that if we do decide to carry something we'll most likely try to use it, in which case it becomes a potential distraction that can slow us down. It is common to hear headteachers and senior leaders say "we don't use all those bits of our system, we just use this grid". But the problem is that whilst all those other bits exist there is a temptation to use them, to waste your evenings and weekends wading through various reports and charts, and for governors to ask for them. Even worse, there is the potential for a 'visitor' to say "Oh, you use that system! Can you run this report for me please?"

Ditch it. If you haven't got it, you can't use it, and so it ceases to be an issue.

And when inevitably you do come up against someone asking for something they shouldn't be asking for, this should be your response:

"We don't do that in this school. It has no impact on learning"

Have a great summer.


Thursday, 6 July 2017

Predicting progress using the VA calculator: some things to bear in mind

It was great to read in Ofsted's March update that "Ofsted does not expect any prediction by schools of a progress score, as they are aware that this information will not be possible to produce due to the way progress measures at both KS2 and KS4 are calculated." Sean Harford went even further in his blog, describing the process of predicting progress as a 'mug's game'. This is welcome guidance from Ofsted and shows that they understand the complexities of value added measures in comparison to the old levels of progress measure.

However, whilst Ofsted won't be asking for such data, I recognise that schools still like to have an idea of progress scores before they pack up for the summer, especially with floor standards link to these measures, and that's why I produce the VA calculator. It's a free tool and I'm happy for all primary schools to use and share it. It's available in two formats: old school excel, and new-fangled web tool. Feel free to have a play around with both.
.
But, if you do use it, it's important that you understand its limitations, which I've outlined below in order of impact and likelihood of change.

1) Estimates
Value Added measures involve comparing each pupil's actual result against an estimated result. The whole school progress score is the average of the differences between the actual results and estimates. An estimate is the national average score for pupils with similar prior attainment in that particular year. For example, let's take a year 6 pupil that was 2c in reading, writing and maths at KS1. They have a KS1 APS of 13 and are in prior attainment group 9. The DfE have to identify all pupils nationally with the same KS1 APS score, and they calculate the average scaled score for this prior attainment group in order to generate the estimates . In 2016, pupils in this prior attainment group son average scored 97.26, 96.69 and 98.33 in reading, writing and maths respectively (if you want to know more about the crazy world of writing progress, read this). Our 2c pupil's KS2 scores are therefore compared against these benchmarks - if they score higher then they get a positive score; if they score lower they get a negative score. The issue with trying to work out progress for the current year 6 cohort is that we are comparing them against last year's estimates, whereas we should be comparing them against the national average score for pupils with similar prior attainment in the same cohort. We don't have this data yet and won't have it until September. Judging by the overall improvement in results nationally, we can safely assume that the estimates for each prior attainment group will change and will no doubt rise in most if not all cases (they may drop for lowest PA groups because DfE intend to introduce data from special schools to mitigate this problem).
Verdict: definite change

2) Standard deviations
Standard deviations change every year, and these form part of the calculation of confidence intervals on which statistical significance depends. This means your data might not be significantly above or below average on the VA calculator but might be when we get the proper data in September. Or vice versa.
Verdict: definite change

3) Pupils that are entered for tests but do not score will be assigned a nominal score
This was covered in the progress loophole of despair post here: pupils assessed as HNM or EXS in reading and maths that sat the test but did not score enough marks to achieve a scaled score were excluded from progress measures last year. It looks like this particular loophole will close this year, which is a good thing, but we have no idea what nominal score these pupils will be assigned. I'm voting for 79 (1 point below the lowest scaled score of 80). The loophole was a particular issue in reading last year, when over 3000 pupils were entered for tests but did not achieve enough marks to gain a scaled score (compared with around 300 in maths). It is likely to be a much smaller number this year, but it will still affect quite a few schools.

Please note: this specifically relates to pupils assessed as HNM or EXS that did not manage to achieve a scaled score. Pre-key stage pupils that were entered for tests (for whatever reason) and did not manage to achieve a scaled score had a fall back, nominal score and therefore were included in progress measures (see No. 4 below).
Verdict: definite change

4) Floor standards & Coasting
Not really related to the VA calculator but something schools will have their eye on. Last year the progress floor thresholds were set at -5, -7 and -5 for reading, writing and maths respectively. These were incredibly low and reflected the fact that attainment was low (they just want the right amount of schools below floor after all). These threshold will go up this year, but by how much is anyone's guess. Same applies to coasting - they just halved the floor standards, remember? - unless they scrap the coasting measure. Please let them scrap the coasting measure. I assume we'll get the new floor and coasting thresholds at some point in the autum term but for some reason they don't apply them in the Ofsted dashboard until validated data is released.
Verdict: almost certain change

5) Nominal scores for writing
Currently, pupils are assigned the following nominal scores according to their writing teacher assessment:

WTS: 91
EXS: 103
GDS: 113

It is likely that these will change, which will have an impact on progress scores. If they go up, then that will mitigate the inevitable increase in the estimates; if they stay the same then there will be a negative impact on overall progress scores. For what it's worth, I'd like to see the writing progress measure scrapped because it's frankly ridiculous.
Verdict: change likely

6) Nominal scores for pre-key stage pupils
Last year, pre-key stage pupils were assigned nominal scores according to their specific teacher assessment, which caused a huge amount of damage to the progress scores of those schools that had pre-key stage pupils. These scores, used for progress measures only, are as follows:

BLW: 70
PKF: 73
PKE: 76
PKG: 79

We have no idea if these nominal scores will change but I suspect they won't because increasing them will encroach upon the actual scale score range, which doesn't make sense, and decreasing them will further penalise schools with low attaining, SEN pupils. Obviously, any change to these scores will have an impact on progress calculations.
Verdict: change unlikely

NB: If you want to recalculate progress with pre-key stage pupils removed, this is something the VA calculator can help with.

7) Prior Attainment Groups
Currently there are 21 prior attainment groups, as detailed on p17-18 of the Primary Accountability Guidance. These primary attainment groups (PAGs) are integral to the progress measures and are, like the other factors detailed above, a vital part of progress calculations and the VA calculator. They are the bolts onto which the estimate nuts fix. If these change then it will fundamentally alter structure of the progress measures, and make latest data incomparable with last year. I suspect they will remain the same. 

Verdict: change unlikely

We will have to wait for the release of progress data in September to get answers to these questions. In the interim, the VA calculator can be used as a guide to indicate whether progress is well above or below average, or broadly in line. We will of course update the VA calculator with 2017 estimates in September so you can use it to accurately recalculate progress with specific pupils removed. I will have more confidence in the forecasts next year, when the estimates are based on this year's more reliable results, and many of the issues listed above have been resolved.

So use it and share it; just be aware that data is likely to change.