Thursday, 22 January 2015

The Data Trap

The most vulnerable schools - those in RI or special measures, or those identified as weak by an LA or sponsor - have the greatest data burden placed upon them. That's a given. Despite the drive for schools to develop new ways of assessing and tracking pupil progress, these 'schools causing concern' have less room to innovate, and are probably less willing to do so. Better not rock the boat, just give them what they want. Even if what's being asked for is a) going to take a huge amount of time and resources, and b) complete, irrelevant nonsense.

Meanwhile, schools in a more secure position - those that have recently been inspected and had a good outcome - are left alone to their own devices. They can discuss, research, develop and test new methods of assessments - find a system that works for them. They are free to innovate and take the necessary time to get it their approach to assessment without levels right, thus ensuring it is tailored to meet the needs of their curriculum. The way it should be and the whole point of removing levels in the first place. 

The vulnerable schools on the other hand are often finding that they are straightjacketed by data requirements placed upon them, and that these begin to dictate how they operate. Often, various parties involved - LA advisors, consultants, HMIs - are requesting different data sets, ramping up the workload even further. This results in a increasingly unnecessary data burden requiring greater system complexity, which is completely at odds with a general drive for simplification. Data risks becoming disconnected from its primary function - to inform teaching and learning - and, at worst, ends up as an exercise itself.

Rules for one

I recently visited a fantastic primary school in Gloucester to catch up on developments of their new assessment and tracking system. They are designing a hybrid system, taking elements and ideas from two assessment models, and working closely with a tracking system company to ensure that the data can be easily captured and analysed. This is as it should be: design the assessment system that suits your curriculum and then tailor the tracking to fit; and they have the time, resources and confidence to do this. Exciting stuff! 

Then I spoke to a headteacher in another county who is in a very different situation. Her school has been identified by the LA as one causing concern. Some of their results were lower than expected and she is now under pressure to prove that pupils are making better than expected progress, and that results will improve this year. Worryingly she told me that the advisor that the LA has assigned to her school made her feel deeply inadequate when it came to her understanding of data. However, in the ensuing conversation she explained her ideas for assessment and tracking in the new curriculum, and it almost exactly matched the plans of the school I mentioned above. Unfortunately this headteacher didn't really feel confident enough to implement radical changes right now, opting instead to tow the party line. This headteacher had the right ideas but was being prevented from developing them because of the data demands placed on her school. An example of producing data purely for the sake of accountability rather than to meet the needs of the new curriculum and the pupils in the school. Bureaucracy stifles innovation.

One school I've been supporting has been told by their HMI that all pupils need to make at least 2 points of progress this term. Seriously, what is 2 points of progress these days? How does a pupil make a sublevel of progress in a curriculum without levels? The school lets out a collective sigh, shrugs its shoulders and mutters something along the lines of "well, I suppose we'd better give them what they want". Consequently the school has decided to delay the transition to assessment without levels until they are out of the situation they're in. The data has therefore become disconnected from its primary function of informing teaching and learning, and instead exists predominantly for the purposes of accountability. I struggle to see how this helps a school improve. 

In another school, a consultant, working on behalf of a sponsor, issued the Head with a data pack to complete. A behemoth requiring every conceivable shred of data broken down into every subject, cohort and group. It would take ages to complete; days at least, probably weeks. Meanwhile they have a school to run. 

And the madness is that the data pack was almost entirely concerned with percentages making expected and better than expected progress in terms of APS. None of this fitted with the tracking system the school has in place so the options were to a) adapt the data pack to fit the data (sensible), b) make some data up (not advisable but essentially what you are doing by producing progress data in APS), or c) rip it up and instead present some meaningful data that actually relates to what is happening in the school (my preferred option). 

These situations arise because someone in a position of influence and power is poorly informed about assessment and tracking; about what a school can and should provide. And these issues are disproportionately affecting vulnerable schools that really should be concentrating on other things. The big risk is that these schools either delay implementing a new assessment system in order to continue to satisfy these inappropriate and unnecessary data demands; or they end up running two systems in parallelI: one for teaching and learning, and one for bureaucracy. Neither are acceptable.

What's reasonable, useful and relevant can now be boiled down to 4, maybe 5 indicators: % at/above age-related expectations, % making expected/better than expected progress, % on track to meet targets (for smaller cohorts, numbers may be more appropriate). That's pretty much all you need and a year group can be presented on a single sheet, with rows for different groups. Such generic data is, and always will be, readily available from tracking systems, and should be enough to satisfy any LA advisor, SIP, Ofsted inspector or HMI. 

In case anyone needs convincing further, refer to paragraph 191 of the Ofsted handbook and commit it to memory. It could help keep control of your data:

'Inspectors will not expect to see a particular assessment system in place and will recognise that schools are still working towards full implementation of their preferred approach. However they will evaluate how well pupils are doing against relevant age-related expectations as set out by the school and the national curriculum (where this applies).

And if you are ever asked to produce meaningless, obsolete data by someone who really should know better, then slap down a ten pound note on the desk in front of them and ask them to convert it into shillings. It's pretty much the same thing.

I'll wrap this blog up with the following thought:

In order to get assessment and tracking right we need to ignore the pressures of accountability and performance management, and pretend Ofsted doesn't exist. Assessment is for learning; the rest is noise.





Friday, 9 January 2015

Junior school VA: It's all about that base(line)

The following notes relate to some work I'm doing to support a junior school with a bit of a baseline issue. A case of significantly high prior attainment, good KS2 results, and abysmal progress. So, I've been trying to build a case for the defense against the duality of the RAISE report - a sort of lush green lawn of attainment next to a deep blue pond of progress. Obviously, the head has given their permission for me to blog this. Hopefully it will be useful or interesting or both.


This is not intended as an infant school bashing piece by the way. Some junior schools have a baseline issue, some don't.

Junior School A is characterised by intakes that are consistently significantly above national average on entry. The 2014 year 6 cohort comprised 57.4% high attainers - more than double the national proportion of 24.7% - constituting the largest proportion of high attainers the school has ever had. Percentages of low attainers are correspondingly low, ranging from 3.5 to 9.3%. Attainment is generally above or significantly above national average but progress, in terms of VA is significantly below.

VA scores for 2014 cohort are as follows:

All subjects: 97.8 (sig- and a significant decline)
Maths: 96.9 (sig- and significant decline)
Reading: 98.7 (sig-)
Writing: 98.7 (sig-)

VA is the comparison between a pupil’s actual attainment and a calculated estimate for the end of KS2, the estimate being based on national average attainment for pupils with the same prior attainment. The validity of the estimate is entirely dependent on the accuracy of the baseline used to generate the estimates, i.e. the KS1 results.

Our analyses question the validity of the baseline used to calculate VA. We have compared VA estimates based on the cohorts’ key stage 1 results against VA estimates based on the school’s own baseline taken in year 3, and against the predictions generated by CATS tests taken in year 5. The differences are considerable and need to be taken into account when assessing progress made by this cohort.

Key Stage 1 vs School’s own Year 3 baseline
The school carries out a baseline assessment early in year 3 and this conflicts substantially with the key stage 1 data. The following table shows percentages of the 2014 year 6 cohort in each attainment band based on KS1 data and the school’s Y3 baseline:


Based on KS1
Based on Y3 baseline
Low (APS <12)
5.6%
14.8%
Middle (APS 12-18)
37.0%
79.6%
High (APS >=18)
57.4%
5.6%

The percentage of high attainers is considerably lower when using the year 3 baseline data in place of KS1 results.

Value added
We then generated VA estimates for the 2014 year 6 cohort by entering their year 3 baselines into a VA calculator tool (using RAISE 2014 methodology) and compared these estimates to those produced from the KS1 results. The following table shows the attainment, in terms of APS, that the 2014 year 6 cohort were required to achieve on average in order to obtain a VA score of 100 (i.e. make average progress). In each case, the estimated (expected) attainment based on KS1 results is far higher than that based on the year 3 baseline.


Reading
Writing
Maths
Overall
Based on Y3 baseline
28.3
27.5
27.5
27.7
Based on KS1 results
31.0
30.6
32.3
31.5
Difference
2.7
3.1
4.8
3.8

The APS difference indicates the additional progress the cohort is required to make in order to ‘break even’. Using the VA tool we then calculated revised VA scores where each pupil’s actual attainment is compared against the lower, more realistic estimate generated from the year 3 baseline rather than the KS1 result. This shifts the VA from significantly below to significantly above average. The table below compares the revised VA scores, based on the Y3 baseline, against VA presented in RAISE. The values in the bottom row show the differences between the two VA scores and can be construed as the progress that is lost when key stage 1 results are used as the baseline.


Reading
Writing
Maths
Overall
VA score based on Y3 baseline
100.8
101.1
100.8
100.9
VA score based on KS1 results
98.7
98.7
96.9
97.8
Difference
2.1
2.4
3.9
3.1

Average differences between raw VA scores (i.e. unadjusted data before shrinkage factor is applied) are even greater. This is perhaps the best indication of potential lost progress in the school's VA data.


Reading
Writing
Maths
Overall
APS difference
2.3
2.6
4.3
3.4

Key Stage 1 data vs CATS tests
In order to validate the estimates generated from the year 3 baseline, we compared them against the predictions derived from the CATS tests taken by the 2014 year 6 cohort in October of year 5. We converted the sublevel predictions from the CATS test to APS and compared them with VA estimates generated from the key stage 1 results. The estimates and differences are presented in the following table:


English
Maths
Overall
CATS predictions (APS)
27.4
27.6
27.5
VA estimates (from KS1 results)
30.8
32.3
31.5
Difference
3.4
4.7
4.0
NB: 2014 RAISE does not provide estimates for English. For purposes of comparison, the average APS of reading and writing was used.

The CATS predictions closely match the estimates derived from the Y3 baselines shown above and therefore help to substantiate the school's on entry assessment. If the school's Y3 baseline is accurate, as indicated by the CATS data, then there is compelling evidence to suggest that this cohort made far more progress than official RAISE data implies (between 2 and 4 points based upon the VA differences presented above). The school’s Y3 baseline should therefore be used in place of key stage 1 data for an reliable assessment of pupil progress.

Level 5 attainment
Attainment at both Level 4 and Level 5 is above national average for all groups and all subjects with the exception of disadvantaged pupils and high prior attainers at level 5. According to RAISE there were 31 high prior attainers in the 2014 Year 6 cohort, and the level 5 attainment of this group was below average overall and significantly below average in writing. However, based on the school’s Y3 baseline data only 3 pupils are included in this group and all of them achieved level 5 in each subject. The blue (sig-) box on p21 of the RAISE report, relating to a high prior attainment group comprising 31 pupils, should therefore be discounted.

Using the Y3 baseline, the size of the middle prior attainment group increases from 20 pupils to 43. The level 5 outcomes for this group are presented in the table below:


L5+ MRW
Maths
Reading
Writing
RAISE MAP (20 pupils)
5%
15%
45%
30%
School MAP (43 pupils)
35%
44%
65%
54%
National average
13%
36%
46%
24%

These figures for the revised middle prior attainment group are considerably higher than level 5 data in RAISE and are above national average in all subjects.

Disadvantaged pupils
A higher than average proportion of disadvantaged pupils achieved level 4 in all subjects, and above average proportions reached level 5 in reading and writing but only 1 did so in maths; and no disadvantaged pupils attained level 5 in all subjects combined. Further analysis of this group using the Y3 baseline data reveals that none of this group were high attainers at key stage 1, six were middle attainers, and five were categorised in the low prior attainment group. This contrasts with official data where numbers in the high, middle and low prior attainment groups were 3, 6 and 2 respectively. Moreover, the Y3 baseline data indicates that no pupils had a KS1 APS above 16 and so it is perhaps less likely that such a group would attain level 5. The fact that some did achieve level 5 is therefore notable when their on entry assessments are taken into account.

The following table summarises percentages of disadvantaged pupils making expected and better than expected progress; and compares against national benchmarks for other pupils. Figures increase when progress is measured from the year 3 baseline and are closer to national comparators. The percentage of disadvantaged pupils making 3LP is particularly high in reading.


2LP
3LP

RAISE
Y3 baseline
National non-FSM
RAISE
Y3 baseline
National non-FSM
Reading
73%
82%
92%
36%
64%
34%
Writing
82%
82%
94%
27%
36%
34%
Maths
73%
82%
91%
9%
27%
38%

Levels of progress for 2014 Year 6 cohort
The following table shows revised percentages making expected progress and better than expected progress using the school’s Y3 baseline. 2LP figures are above 90% and in line with national figures, whilst %3LP figures are above 40% and higher than national averages.


2LP
3LP

RAISE
Y3 baseline
National
RAISE
Y3 baseline
National
Reading
76%
96%
91%
24%
46%
35%
Writing
89%
91%
93%
33%
46%
33%
Maths
85%
94%
89%
7%
41%
35%