Sunday, 21 May 2017

This is a low

I often use a race analogy to explain value added, to help people understand how we can measure progress without levels and why we don't need to have data in the same format at either end. Here's a simple example:

Imagine you enter a 10k race, which is part of a series of 10k races being held across the country on the same day. When you register you are asked what pace group you’d like to run in: slow, medium, fast. A keen runner, you choose to go in the fast group and you're handed a green vest to wear. Obviously, the medium pace runners get orange vests, and the slower group wear red (everyone loves a RAG rating system). You feel good that day, having trained hard, and run your race in 41 minutes. You’re thrilled because you’ve run a PB and you’re 10 minutes faster than the average time for your race that day. Even better, you find out you are 12 minutes faster than the national average time for the whole series. Unfortunately that’s not what the race organisers are interested in; they’re interested in how your time compares against the national average time for the green vest group, which happens to be 37 minutes. Despite being way faster than the overall average time, you are 4 minutes down on the average time for your group. Your value added score is therefore -4.

This is how VA works: it involves comparing one result against the average result of those in the same start group nationally. Here we have a start defined by a colour and a result in a time format; for KS1-2 measures we currently have a start defined by a sub level and a result in scaled score format. Same thing. 

Whilst I much prefer value added to the old levels of progress measure - it's rooted in some form of reality after all - it does have one serious flaw: SEN and EAL pupils are often expected to cross the same line in the same time. 

The issue is that many EAL and SEN pupils have comparably low start points, and are therefore placed into the same prior attainment groups, effectively treating them as similar pupils. And this means they will be compared against the same benchmarks at KS2. As we know, many EAL pupils make rapid progress and score well in their KS2 tests, whereas SEN pupils do less well. The end of KS2 estimates against which each pupil in the prior attainment group is compared, being an average of the performance of SEN and EAL pupils, tend to be too high for the former and easily attainable for the latter. The issue is exacerbated by the current system of low nominal scores assigned to the pre-key stage assessments, which almost guarantees that SEN pupils can only obtain negative progress scores whilst EAL pupils excel against their benchmarks.

We can return to our race analogy to illustrate this issue further. Imagine our pace groups are defined by how many steps runners could take when they were 18 months old. The runners wearing red vests were those that couldn't walk at that age. But perhaps some of those have gone on to be fast runners whilst others have continued to have difficulty walking. On race day they are in the same group, in the same vest, and each of their times will be compared against the overall average time for group. Hardly fair.

This issue needs resolving somehow. Introducing some form of CVA is an obvious answer - a measure that recognises the difference between SEN and EAL pupils - but is likely to lead to a proliferation of SEN pupils and a corresponding decline in those registered as EAL. Removal of pre-key stage pupils from progress measures is also a possibility but that may result in a big increase in pre-key stage pupils as schools seek to get certain pupils discounted. 

I'm not sure what the answer is but it needs serious thought because as it stands, schools are hammered if they have SEN pupils, especially if they are pre-key stage. 

The DfE stated that they wanted measures to reflect the progress made by all pupils.

Time to make good on that. 


Friday, 12 May 2017

Pupils included and not included in KS2 measures

The issue of who is and who isn't included in KS2 measures is still causing major headaches for many, which is understandable because it's a bloody minefield. As we have just got through SATS week, and no doubt many senior leaders are now turning their attention to those not so distant reports, I thought I'd attempt to provide some clarity. There is, of course, a chance I've got some of this wrong, but it's worth a try.

Attainment 

Attainment can be broken down into two main measures: 1) threshold measures (% attaining expected and high standards) and 2) average scaled scores.

1) Pupils included in and excluded from threshold measures

All pupils are included in this measure initially. Pupils can be discounted if they are recent arrivals from overseas, are EAL and from a non-English speaking country. Such pupils are identified during the checking exercise in September, using the results list sent via NCA Tools. Consequently, due to timing, discounted pupils are included in unvalidated data but are removed from later, validated data releases including the performance tables. All other pupils are included in the measure including pupils that were absent, below standard of tests, or disapplied. Pupils that achieve 100+ (or who have a TA of EXS in writing) are deemed to have met the expected standard; those that achieve a score of 110+ (or have a TA of GDS in writing) are deemed to have met the high standard. They are the only pupils in the numerator. All other pupils are in the denominator with exception of any discounted pupils. Again, discounting does not take effect until validated data release. 

2) average scaled scores

Only pupils with a scaled score of 80+ are included in this measure. Nominal scores assigned to pre-key stage assessments (70-79) are only used in the progress measure. Nominal scores are not used in average scaled score calculation. 

Progress

This is the real minefield and I created the following diagram to help navigate it:

 
Essentially to be included in the progress measure a pupil needs a start point (KS1 result) and an end point (KS2 score). The score can be a scaled score from a test or a nominal score assigned to a pre-key stage assessment or teacher assessment in the case of writing.

Scaled scores range from 80 to 120. 

Writing scores as follows:
WTS = 90
EXS = 103
GDS = 113
(Note: these may change this year)

Nominal scores are as follows:
BLW = 70
PKF = 73
PKE = 76
PKG = 79
(Note: these may also change this year)

So, if a pupil has a) a KS1 start point, and b) and KS2 score as detailed above, they will be included in progress measures.

Those that are excluded from progress measures (and this is where it gets complicated and may well change this year) are as follows:

No KS1 result
pupils without a start point are not included. They are not assigned a nominal baseline. 

Absent (A code)
Pupils that are absent from tests are excluded from progress measures even if they also have a pre-key stage assessment

Disapplied (D code)
Pupils that are disapplied are also excluded from progress measures. This is a commonly misunderstood term, confused with 'below standard of test'. It actually should only be used in cases where a pupil has been disapplied from the national curriculum and it is therefore not possible to make a teacher assessment. A disapplied pupils cannot therefore have a pre-key stage assessment. I have seen numerous examples of pupils coded as D, that actually should have had a B code and accompanying PKS assessment. Disapplied should be a rare occurrence in mainstream settings. 

No scale score awarded and has HNM or EXS teacher assessment
These are the progress loophole pupils. They sat the test but failed to achieve enough marks to get the lowest scale score of 80. There were around 350 in maths last year but around 3500 in reading. Because HNM and EXS do not have an associated nominal score, if a pupil with one of those teacher assessments fails to achieve a scale score, then they end up with no score at all, and no score means they cannot be included in progress measures. The STA have stated that they intend to close the loophole this year, which probably means assigning a nominal score to HNM and EXS for such instances. We don't know what the nominal score would be but it seems logical to assume it would be capped at 79. 

Missing result (M code) 
If the result is missing, the pupil is excluded from progress measures.

Unable to access test (T or U code)
Again, pupils with these codes are excluded from progress measures.

Think that's pretty much evevrything.

Hope it's useful.