Sunday, 21 May 2017

This is a low

I often use a race analogy to explain value added, to help people understand how we can measure progress without levels and why we don't need to have data in the same format at either end. Here's a simple example:

Imagine you enter a 10k race, which is part of a series of 10k races being held across the country on the same day. When you register you are asked what pace group you’d like to run in: slow, medium, fast. A keen runner, you choose to go in the fast group and you're handed a green vest to wear. Obviously, the medium pace runners get orange vests, and the slower group wear red (everyone loves a RAG rating system). You feel good that day, having trained hard, and run your race in 41 minutes. You’re thrilled because you’ve run a PB and you’re 10 minutes faster than the average time for your race that day. Even better, you find out you are 12 minutes faster than the national average time for the whole series. Unfortunately that’s not what the race organisers are interested in; they’re interested in how your time compares against the national average time for the green vest group, which happens to be 37 minutes. Despite being way faster than the overall average time, you are 4 minutes down on the average time for your group. Your value added score is therefore -4.

This is how VA works: it involves comparing one result against the average result of those in the same start group nationally. Here we have a start defined by a colour and a result in a time format; for KS1-2 measures we currently have a start defined by a sub level and a result in scaled score format. Same thing. 

Whilst I much prefer value added to the old levels of progress measure - it's rooted in some form of reality after all - it does have one serious flaw: SEN and EAL pupils are often expected to cross the same line in the same time. 

The issue is that many EAL and SEN pupils have comparably low start points, and are therefore placed into the same prior attainment groups, effectively treating them as similar pupils. And this means they will be compared against the same benchmarks at KS2. As we know, many EAL pupils make rapid progress and score well in their KS2 tests, whereas SEN pupils do less well. The end of KS2 estimates against which each pupil in the prior attainment group is compared, being an average of the performance of SEN and EAL pupils, tend to be too high for the former and easily attainable for the latter. The issue is exacerbated by the current system of low nominal scores assigned to the pre-key stage assessments, which almost guarantees that SEN pupils can only obtain negative progress scores whilst EAL pupils excel against their benchmarks.

We can return to our race analogy to illustrate this issue further. Imagine our pace groups are defined by how many steps runners could take when they were 18 months old. The runners wearing red vests were those that couldn't walk at that age. But perhaps some of those have gone on to be fast runners whilst others have continued to have difficulty walking. On race day they are in the same group, in the same vest, and each of their times will be compared against the overall average time for group. Hardly fair.

This issue needs resolving somehow. Introducing some form of CVA is an obvious answer - a measure that recognises the difference between SEN and EAL pupils - but is likely to lead to a proliferation of SEN pupils and a corresponding decline in those registered as EAL. Removal of pre-key stage pupils from progress measures is also a possibility but that may result in a big increase in pre-key stage pupils as schools seek to get certain pupils discounted. 

I'm not sure what the answer is but it needs serious thought because as it stands, schools are hammered if they have SEN pupils, especially if they are pre-key stage. 

The DfE stated that they wanted measures to reflect the progress made by all pupils.

Time to make good on that. 

Friday, 12 May 2017

Pupils included and not included in KS2 measures

The issue of who is and who isn't included in KS2 measures is still causing major headaches for many, which is understandable because it's a bloody minefield. With SATS looming, and no doubt many senior leaders already thinking about those not so distant headline measures, I thought I'd attempt to provide some clarity. There is, of course, a chance I've got some of this wrong the DfE have a habit of changing things - but it's worth a try.


Attainment can be broken down into two main measures: 1) threshold measures (% attaining expected and high standards) and 2) average scaled scores.

1) Pupils included in and excluded from threshold measures

All pupils are included in this measure initially. Pupils can be discounted if they are recent arrivals from overseas, are EAL and from a non-English speaking country. Such pupils are identified during the checking exercise in September, using the results list sent via tables checking website. Consequently, due to timing, discounted pupils are included in unvalidated data but are removed from later, validated data releases including the performance tables. All other pupils are included in the measure including pupils that were absent, below standard of tests, or disapplied. Pupils that achieve 100+ (or who have a TA of EXS in writing) are deemed to have met the expected standard; those that achieve a score of 110+ (or have a TA of GDS in writing) are deemed to have met the high standard. They are the only pupils in the numerator. All other pupils are in the denominator with exception of any discounted pupils. Again, discounting does not take effect until validated data release. 

2) average scaled scores

Only pupils with a scaled score of 80+ (ie with a test score) are included in this measure. Nominal scores assigned to pre-key stage assessments (59-79) are only used in the progress measure. Nominal scores are not used in average scaled score calculation. 


This is the real minefield. Essentially, to be included in the progress measure a pupil needs a start point (KS1 result) and an end point (KS2 score). The score can be a scaled score from a test or a nominal score assigned to a below/pre-key stage assessment; or any teacher assessment in the case of writing (because there are no test scores in writing). 

Scaled scores range from 80 to 120. 

The scores assigned to the main writing teacher assessments are as follows:
WTS = 90
EXS = 103
GDS = 113

Note: these may change, but seems unlikely as they were kept the same for 2017. 

Nominal scores - used for pupils below the standard of curriculum tests (i.e. pupils that have a B code), or working below the standard of the writing frameworks - are as follows:

P1-3 = 59
P4 = 61
P5 = 63
P6 = 65
P7 = 67
P8 = 69
BLW (without p-scale ie NOTSEN) = 79
PKF = 73
PKE = 76
PKG = 79
N (HNM/EXS TA and took the test but did not achieve enough marks for scaled score) = 79

Note: these are somewhat different to the more simplified nominal scores used in 2016. They may change again but it seems unlikely.

So, if a pupil has a) a KS1 start point, and b) and KS2 score - either scaled or nominal score as detailed above - they will be included in progress measures.

Those that are excluded from progress measures are as follows:

No KS1 result
pupils without a start point are not included. They are not assigned a nominal baseline. 

Absent (A code)
Pupils that have an A code - are absent from tests or have long periods of absence - are excluded from progress measures even if they also have a pre-key stage assessment. Please refer to ARA for more detail on this code - link below. 

Disapplied (D code)
Pupils that are disapplied are also excluded from progress measures. This is a commonly misunderstood term, confused with 'below standard of test' (often referred to as ‘disapplied from tests’). This code should only be used in cases where a pupil has been disapplied from the national curriculum and it is therefore not possible to make a teacher assessment. A disapplied pupil cannot therefore have a pre-key stage assessment. I have seen numerous examples of pupils coded as D, that should have had a B code and accompanying PKS assessment. Disapplied should be a rare occurrence in mainstream settings. Please refer to ARA  for more detail on this code - link below. 

Missing result (M code) 
If the result is missing, the pupil is excluded from progress measures. 

Unable to access test (U code)
These are pupils that are working at the standard of the test but are unable to access them due to a disability or severe emotional trauma. Pupils with a U code are excluded from progress measures. Please refer to ARA  for more detail on this code - link below. 

Disregard (Q code)
This code is used in cases of maladministration. Again, a pupil with this code will be excluded from progress measures. 

If you want to model examples, use the DfE pupil ready reckoner tool, which can be downloaded here.

Note about the 'Progress Loophole of Despair'

In 2016, pupils that had a teacher assessment of HNM or EXS, that sat the test but failed to achieve enough marks to get the lowest scale score of 80 were excluded from measures. This meant that many schools benefitted from pupils scoring below the minimum mark threshold because such pupils were omitted from overall progress scores rather than be included with a nominal score. In 2017, this loophole was closed by assigning a nominal score of 79 in such cases (see above). There were cases in 2017 of pupils with a teacher assessment of HNM not sitting the test, who did not achieve a test score but were not assigned a nominal score either, and were therefore excluded from progress measures. This is because HNM pupils were only assigned a nominal score (79) if they sat the test and failed to score, not if they didn't sit the test, which is clearly another loophole. Please note that pupils with an HNM assessment should sit the tests unless there is good reason for them not to do so (refer to section 5.2, p18-19 of Assessment and Reporting Arrangements (pupils unable (U) to access tests).

Think that's pretty much everything.

Hope it's useful.