Saturday, 15 October 2016

The Zero Game

Until this year VA has played second fiddle to the levels of progress measure. This was mainly because there were no floor standards linked to VA. But it was also because - let's be honest here - not many people really understood it. Everyone understood pupils needing to make two levels of progress (and hopefully three levels of progress) but we struggled with the world of prior attainment groups, estimates, and confidence intervals. But now that levels are gone and all we're left is a value added progress measure, we have no choice but to get our heads round it. So, we read the primary accountability document and have seen the lookup table on p16-17; we understand there are 21 prior attainment groups (a reduction in start points in previous years due to change in methodology); that each of these prior attainment groups has an estimate in reading, writing and maths, which represents the national average score for pupils in that group; that these estimates form the benchmark for each pupil; and that exceeding these scores ensures a positive progress score for each child, which will aggregate to a positive progress score overall. We get this now.

And that's where the trouble started. 

Up until recently, schools were flying blind. With a new curriculum and new tests, unsure of what constituted expected standards, and no idea of 'expectations' of progress, schools just concentrated on teaching and tracking the gaps in pupil's learning. We even started to question the methods of our tracking systems, with their pseudo-levels and points-based progress measures. Things were looking positive. The future was bright.
 
But then we saw the checking data, and that lookup table appeared, and I produced my VA calculator, and FFT published their 2016 benchmarked estimates.  Now it seems that many schools are playing a VA game, working out where each pupil needs to get to in order to ensure a positive progress score; comparing benchmarked estimates (that are no doubt too low for next year) against predicted results to model VA in advance, to provide figures to settle nerves and satisfy those scrutinising schools' performance.

I understand that schools want a positive VA score when the stakes are so high but we have to consider the potential risks to pupils' learning by focussing on minimum 'expected' outcomes. I am particularly concerned to hear that schools are building systems that track towards these estimated outcomes, using teacher assessment or optional tests as a proxy for expected standards, as a predictor of outcome that can then be compared against the end of key stage progress estimate. I think of the ideals of 'learning without limits' and the sound principles for the removal of levels, and wonder if anything has really changed. I also wonder if it was wise to publish my VA calculator. All those schools inevitably using it to generate estimates for current cohorts; estimates that are being entered into systems and somehow tracked towards. Am I part of the problem? 

Has a knowledge of progress measures become a risk to children's learning? 

How about we just put the blinkers on and concentrate on teaching? Look after the pennies and the pounds will take care of themselves. 

Just a thought. 

Tuesday, 11 October 2016

A clear and present danger: the problem with writing VA

Back in February I wrote this post, speculating on the potential issues of the proposed writing progress measure. That measure is now a reality and the issues predicted in the blog post have taken root. I don't have a problem with VA measures per se - although I would prefer it to be contextualised - but I do have an issue with VA when it's based on teacher assessment. The problem is that teacher assessments are too broad, too inconsistent, and too prone to bias when high stakes are involved. I certainly echo the proposals of the Headteachers' roundtable alternative green paper, which calls for the removal of teacher assessment from high stakes accountability measures. In my blog I discussed the problems of the writing VA measure under levels and suggested that it would only get worse in the post-levels world. To understand this we need to look at how VA works and why writing differs from reading and maths.

VA involves comparing a pupil's attainment score at KS2 against the national average score for pupils with the same KS1 prior attainment. Many of us will now be familiar with the lookup table on p16-17 of primary accountability document, which shows the average scores in reading, writing and maths for each of the 21 prior attainment groups. These are the benchmarks against which each pupil's actual KS2 scores in the three subjects are compared this year (please note: they'll almost certainly increase next year). I don't have a big problem with the reading and maths benchmarks - they are fine graded, and pupils, having sat tests in those subjects, will have a fine grade to compare it against. In short, the reading and maths benchmarks are achievable.

That is not the case for writing. Again the benchmarks are fine graded but unlike reading and maths, in writing they are unachievable. Because there is no test score in writing the decision was taken to apply a nominal score to the teacher assessment. In my blog, I speculated that 90, 100, and 110 would be possible nominal scores assigned to the WTS, EXS and GDS assessments. We now know that these scores are 91, 103 and 113. This doesn't really change the problem; if anything it makes it worse. To try to understand this a bit better, I put together the following table, which shows every possible progress score in writing this year for every possible TA outcome from every start point:


The first thing to note - and this is something that confuses quite a few people - is that writing benchmarks in column 3 are fine graded. How can this be when they haven't sat a test? Well, consider all those scores of 91, 103 and 113 (and the nominal scores attached to pre-key stage judgements - another issue). If we add up all the various nominal scores for all pupils in a particular prior attainment group (thousands of pupils) and divide by the number of pupils in that group, we will of course end up with a fine grade. But as mentioned above, it's a fine grade that in many cases, due to the scoring system, pupils can't get close to.

Now have a look at the next few columns. Negative numbers indicate pupils making supposedly less than average progress. It is therefore not possible for a pupil assessed as BLW, PKF or PKE to make more than average progress, and the only PKG pupils that make more than average progress are those that are in prior attainment group 1, i.e. pupils that were below P7 average at KS1. Now take a look at pupils with a KS1 average point score of 10 or more - they need to be at least EXS to make more than average progress. In reading and maths, they just need a few more marks on the test to shift from negative to positive but in writing they need to make the leap from WTS to EXS. For prior attainment group 7 (KS1 APS 10 to <12) this leap changes their progress score from -1.6 to +10.4; and for prior attainment group 9 pupils (2C average at KS1) the progress swings from -5.69 to +6.31 if they are awarded EXS instead of WTS. Meanwhile pupils in prior attainment group 12 (2B average at KS1) assessed as WTS at KS2 end up with a deficit of -9.75. And EXS isn't good enough for prior attainment group 16 (2A average at KS1). Pupils in this group need to achieve GDS in order to make more than average progress. 

This is a measure based on an illogical scoring system applied to unreliable data. The methodology results in wild wings into positive or negative territory, which have a profound impact on overall progress scores, especially in small schools. Consequently, many schools are likely to have one eye on the progress lookup table when making their writing assessments next year. High stakes influencing outcome. Yes, there have been reassuring statements from Ofsted and the DfE regarding this year's writing teacher assessments, but I can't see how the writing progress measure is going to improve much next year. This measure is so flawed that one has ask 'why bother?' Is it really worth it? Is a writing progress measure really so important that we are willing to sacrifice accuracy and fairness? Not really - it makes no sense - so let's ditch it.

It's time to admit that we can't measure progress in this way.