Friday, 18 August 2017

The Progress Bank

For primary schools, there are two main issues with the current system of measuring progress: 1) it is high stakes, 2) it involves teacher assessment. Whether we are talking about our own internal tracking systems, or those official end of key stage DfE measures, the former clearly influences the latter. Imagine you are told that you have to run a marathon in under 3 hours (yes, I do like a running analogy) and that if you fail to do so, the consequences will be dire. Of course, there are some good marathon runners out there for whom this is feasible, and a handful of really good ones for whom this is no problem at all, but for the majority of us this is near impossible. Then we are told that no one will be monitoring your efforts; they will just be basing their judgement on your time, which you alone are responsible for keeping. Now how many sub-3 hours marathon times will we see?

This is the weird situation we find ourselves in: an incompatible mix of high stakes and self-evaluation. Like fire and duvets, they make for strange and dangerous bedfellows, and clearly it's not working. We all know that really.

Consider that weird, near vertical cliff that occurs at 32 marks every year in the national phonics screening check results; or the impact that current KS1 measures will have on the EYFSP now that the profile is used to establish prior attainment groups for those pseudo-progress measures in the dashboard. Then there are those well documented problems with using KS1 results as the baseline for progress measures. The DfE are attempting to solve this by implementing a reception baseline, but this is highly contentious and unpopular; and even if it's implemented in 2019 as planned, we won't see the results until 2026. Then there is the thorny Infant/junior/middle school problem that the reception baseline won't adequately solve; it will just blur the issue so no one really knows where the problems lie, or if there are any problems anyway.

And then, of course, there are those highly variable and suspiciously high KS2 writing results, and associated issues around moderation. Why do you think that KS2 writing was ditched from the baseline for the Progress 8 measure the minute we ran out of secondary pupils with KS2 writing test results? (That was last year by the way; the 2017 GCSE cohort were the first to have writing TA at KS2 and that won't be used in their progress 8 measures).

These are some of the well known issues relating to statutory teacher assessment, assessments that are done almost entirely for the purposes of accountability. It is little wonder that no one seems to have much faith in progress measures that rely on such data. But these issues are not restricted to statutory assessment; they also exist in the various tracking systems schools use, tracking systems which still rely on teacher assessment for measuring progress.

The final report ofthe Commission on Assessment without Levels warns about the risks of using teacher assessment for multiple purposes, yet this is still the norm in many schools. Teacher assessment is commonly used not only for formative purposes, but also for measuring pupil progress, monitoring standards, reporting to Governors, evaluating teacher performance, and even comparing schools. These multiple purposes exert conflicting pressures on the data that can and will lead to its distortion; and let's face it, teacher assessment is too subjective anyway. One teacher's 'secure' is another teacher's 'greater depth', so even if there were no high stakes attached, we still wouldn't have an accurate picture. This is made even more complicated by the various methods of the tracking systems themselves: different steps, bands and point scores; varying lists of key objectives; and contrasting definitions of 'age related expectations' based on spurious algorithms and arbitrary thresholds. And let's be honest, progress measures based on teacher assessment pretty much always involve reinventing levels anyway. No one is talking the same language; no one knows what's going on. No one can measure and compare progress.

That's the point of Progress Bank.

This is something I've been thinking about for a long time: a system that wouldn't rely on teacher assessment to measure progress and wouldn't rely on end of key stage results either. A system that is powered by whatever standardised tests the school chooses to use, and can measure progress from any point to any point, benchmarked against other pupils with similar start points.

This is obviously a rather ambitious project and not something I'm capable of building, which is why I approached the people at Insight Tracking. I like their current system - it's intuitive and highly customisable  - and they tend to build things quick. I needed their expertise and thankfully they agreed.

The system will work like this: schools upload standardised scores from the various tests they use, pick a start point (i.e. previously uploaded data, say at the start of Y1 or Y3) and an end point (probably the most recent upload), and they receive zero-centred VA scores for cohorts, key groups and individual pupils in whatever subjects are tested. The methodology is essentially the same as that used by the DfE to measure progress, so the data is in a common format, but the system is far more flexible in terms of start and end points, and is based on more regular, lower stakes testing. Schools will be able to interrogate their data using simple, interactive reports, which will focus not only on progress, but on attainment gaps, too.

The neat thing is it doesn't matter what tests you use, as long as they're standardised; and if results aren't standardised, the Progress Bank can standardise them if enough data is uploaded by enough schools to provide a suitable sample. If you use tests from multiple providers, that's fine; and if you change test provider, your old data will be stored in Progress Bank, so you won't lose it and can continue to use it. We can, if permission is given, even transfer test results when a pupil changes school. And if you decide to leave, the data is deleted. You own it.

The Progress Bank will be especially useful to junior and middle schools, which have a particular issue when it comes to progress measures, and this project has been expedited by the JUSCO Conference and conversations with members including Chris McDonald, the chair of the group. Obviously, the ideal solution for junior schools is to enable them to measure progress from a Year 3 on-entry baseline, not from KS1 results or from a reception baseline as proposed. Progress Bank will allow them to do that.

But it's not just for junior schools; it is aimed at any school that is interested in alternative, benchmarked progress measures. The system can even measure progress from KS1 scaled scores instead of KS1 teacher assessment used in official measures, or from any current standardised reception baseline assessment. We hope that the data will help schools challenge the flawed measures that is currently used to hold them to account. And by using standardised scores to measure progress, hopefully we can protect the integrity of teacher assessment by ensuring it is used solely for formative purposes, and perhaps reduce workload in terms of tracking and analysis too.

Now we just need lots of schools to get on board.

Find out more about the project and register your interest here:



And follow us on twitter @theprogressbank

Thursday, 20 July 2017

Making the cut

Assessment. It should be simple really: just checking what pupils do and don't know. But assessment appears to have turned into some kind of war, with the legions of accountability amassed on one side, and the special forces of teaching and learning besieged upon a slippery slope with nowhere to go. We become so focussed on outcomes - on floor standards, and coasting thresholds, and league tables - that we risk losing sight of what's important: the here and now. But it is focussing on the here and now that makes all the difference. The irony is that concentrating on accountability - on those distant and unpredictable performance measures - can jeopardise the very results you are striving for. In short, focus on teaching and learning, and the results take care of themselves.

This is, of course, easier said than done but something has to change. In too many schools assessment has become a burden: a top down directive disconnected from learning; an interminable, box-ticking, data-collecting, drain on teachers' time. The risks are clear: morale nose dives and pupils' learning is put at risk. We therefore need to ditch some of our assessment baggage - aim to do more with less - and this requires some serious rationalisation of our processes. It all comes down to one simple question:

Does this have a positive impact on learning?

We need to go through everything we do in the name of assessment and school improvement and ask that question, and we need to be ruthless and honest. What is the benefit and what is the cost? How long does this take? Does it tell us anything we don't already know? Is it having a negative impact? Is it taking teachers' time away from teaching? Ultimately, the only way to improve a school is to teach children well, and anything that distracts from that purpose is a risk.

So let's deconstruct our entire approach to assessment and lay it all out on the hall floor: the various tests you use, your marking policy, target setting (both for pupils and performance management), those lists of learning objectives stapled into pupils' books, and the component parts of your tracking system (yes! every single measure, category, grid, table, graph, chart and report). We now separate these into two piles: those that have a demonstrable, positive impact on teaching and learning, and those that are purely done for the purposes of accountability.

We keep the first pile and ditch the rest.

We now have a stripped down system that is fit for purpose, that is focussed on the right things. From now on, the information we provide to governors and external agencies is a byproduct of our assessment system, which exists to serve teaching and learning alone. If it works, it's right, no matter what others may say. Many will try to convince you that you're mad but deep down they probably just wish they could do the same. If you think this is all too radical, it's really not. There are many schools with extremely minimalist approaches to assessment that have had very successful inspections. Just as long as your approach is informative and has impact, then it's fine. If anything, the simpler the better. And Ofsted are not asking you to generate data purely for their benefit anyway. The Handbook states:

Ofsted does not expect performance- and pupil-tracking information to be presented in a particular format. Such information should be provided to inspectors in the format that the school would ordinarily use to track and monitor the progress of pupils in that school

And the workload review group report on data management had this to say:

Be ruthless: only collect what is needed to support outcomes for children. The amount of data collected should be proportionate to its usefulness. Always ask why the data is needed.

In alpine climbing there are two popular adages: 'if in doubt, leave it out', and 'if you haven't got it, you can't use it'. The first one is obvious, and it's what I'm trying to get schools to think about when they go about rationalising what they do. The second one links to it and recognises that if we do decide to carry something we'll most likely try to use it, in which case it becomes a potential distraction that can slow us down. It is common to hear headteachers and senior leaders say "we don't use all those bits of our system, we just use this grid". But the problem is that whilst all those other bits exist there is a temptation to use them, to waste your evenings and weekends wading through various reports and charts, and for governors to ask for them. Even worse, there is the potential for a 'visitor' to say "Oh, you use that system! Can you run this report for me please?"

Ditch it. If you haven't got it, you can't use it, and so it ceases to be an issue.

And when inevitably you do come up against someone asking for something they shouldn't be asking for, this should be your response:

"We don't do that in this school. It has no impact on learning"

Have a great summer.


Thursday, 6 July 2017

Predicting progress using the VA calculator: some things to bear in mind

It was great to read in Ofsted's March update that "Ofsted does not expect any prediction by schools of a progress score, as they are aware that this information will not be possible to produce due to the way progress measures at both KS2 and KS4 are calculated." Sean Harford went even further in his blog, describing the process of predicting progress as a 'mug's game'. This is welcome guidance from Ofsted and shows that they understand the complexities of value added measures in comparison to the old levels of progress measure.

However, whilst Ofsted won't be asking for such data, I recognise that schools still like to have an idea of progress scores before they pack up for the summer, especially with floor standards link to these measures, and that's why I produce the VA calculator. It's a free tool and I'm happy for all primary schools to use and share it. It's available in two formats: old school excel, and new-fangled web tool. Feel free to have a play around with both.
.
But, if you do use it, it's important that you understand its limitations, which I've outlined below in order of impact and likelihood of change.

1) Estimates
Value Added measures involve comparing each pupil's actual result against an estimated result. The whole school progress score is the average of the differences between the actual results and estimates. An estimate is the national average score for pupils with similar prior attainment in that particular year. For example, let's take a year 6 pupil that was 2c in reading, writing and maths at KS1. They have a KS1 APS of 13 and are in prior attainment group 9. The DfE have to identify all pupils nationally with the same KS1 APS score, and they calculate the average scaled score for this prior attainment group in order to generate the estimates . In 2016, pupils in this prior attainment group son average scored 97.26, 96.69 and 98.33 in reading, writing and maths respectively (if you want to know more about the crazy world of writing progress, read this). Our 2c pupil's KS2 scores are therefore compared against these benchmarks - if they score higher then they get a positive score; if they score lower they get a negative score. The issue with trying to work out progress for the current year 6 cohort is that we are comparing them against last year's estimates, whereas we should be comparing them against the national average score for pupils with similar prior attainment in the same cohort. We don't have this data yet and won't have it until September. Judging by the overall improvement in results nationally, we can safely assume that the estimates for each prior attainment group will change and will no doubt rise in most if not all cases (they may drop for lowest PA groups because DfE intend to introduce data from special schools to mitigate this problem).
Verdict: definite change

2) Standard deviations
Standard deviations change every year, and these form part of the calculation of confidence intervals on which statistical significance depends. This means your data might not be significantly above or below average on the VA calculator but might be when we get the proper data in September. Or vice versa.
Verdict: definite change

3) Pupils that are entered for tests but do not score will be assigned a nominal score
This was covered in the progress loophole of despair post here: pupils assessed as HNM or EXS in reading and maths that sat the test but did not score enough marks to achieve a scaled score were excluded from progress measures last year. It looks like this particular loophole will close this year, which is a good thing, but we have no idea what nominal score these pupils will be assigned. I'm voting for 79 (1 point below the lowest scaled score of 80). The loophole was a particular issue in reading last year, when over 3000 pupils were entered for tests but did not achieve enough marks to gain a scaled score (compared with around 300 in maths). It is likely to be a much smaller number this year, but it will still affect quite a few schools.

Please note: this specifically relates to pupils assessed as HNM or EXS that did not manage to achieve a scaled score. Pre-key stage pupils that were entered for tests (for whatever reason) and did not manage to achieve a scaled score had a fall back, nominal score and therefore were included in progress measures (see No. 4 below).
Verdict: definite change

4) Floor standards & Coasting
Not really related to the VA calculator but something schools will have their eye on. Last year the progress floor thresholds were set at -5, -7 and -5 for reading, writing and maths respectively. These were incredibly low and reflected the fact that attainment was low (they just want the right amount of schools below floor after all). These threshold will go up this year, but by how much is anyone's guess. Same applies to coasting - they just halved the floor standards, remember? - unless they scrap the coasting measure. Please let them scrap the coasting measure. I assume we'll get the new floor and coasting thresholds at some point in the autum term but for some reason they don't apply them in the Ofsted dashboard until validated data is released.
Verdict: almost certain change

5) Nominal scores for writing
Currently, pupils are assigned the following nominal scores according to their writing teacher assessment:

WTS: 91
EXS: 103
GDS: 113

It is likely that these will change, which will have an impact on progress scores. If they go up, then that will mitigate the inevitable increase in the estimates; if they stay the same then there will be a negative impact on overall progress scores. For what it's worth, I'd like to see the writing progress measure scrapped because it's frankly ridiculous.
Verdict: change likely

6) Nominal scores for pre-key stage pupils
Last year, pre-key stage pupils were assigned nominal scores according to their specific teacher assessment, which caused a huge amount of damage to the progress scores of those schools that had pre-key stage pupils. These scores, used for progress measures only, are as follows:

BLW: 70
PKF: 73
PKE: 76
PKG: 79

We have no idea if these nominal scores will change but I suspect they won't because increasing them will encroach upon the actual scale score range, which doesn't make sense, and decreasing them will further penalise schools with low attaining, SEN pupils. Obviously, any change to these scores will have an impact on progress calculations.
Verdict: change unlikely

NB: If you want to recalculate progress with pre-key stage pupils removed, this is something the VA calculator can help with.

7) Prior Attainment Groups
Currently there are 21 prior attainment groups, as detailed on p17-18 of the Primary Accountability Guidance. These primary attainment groups (PAGs) are integral to the progress measures and are, like the other factors detailed above, a vital part of progress calculations and the VA calculator. They are the bolts onto which the estimate nuts fix. If these change then it will fundamentally alter structure of the progress measures, and make latest data incomparable with last year. I suspect they will remain the same. 

Verdict: change unlikely

We will have to wait for the release of progress data in September to get answers to these questions. In the interim, the VA calculator can be used as a guide to indicate whether progress is well above or below average, or broadly in line. We will of course update the VA calculator with 2017 estimates in September so you can use it to accurately recalculate progress with specific pupils removed. I will have more confidence in the forecasts next year, when the estimates are based on this year's more reliable results, and many of the issues listed above have been resolved.

So use it and share it; just be aware that data is likely to change.







Wednesday, 28 June 2017

Primary assessment for non-primary people: a very quick guide

The DfE collects a bewildering array of data from primary schools. This quick guide to statutory assessment is here to help.

Early Years Foundation Stage Profile (EYFSP)
This assessment takes place at the end of the reception year when pupils are 5 years old. It 
comprises 17 early learning goals (ELGs) against which pupils are assessed as emerging, expected, or exceeding. If pupils meet the expected level of development in the 12 prime and specific ELGs then they are deemed to have reached a ‘good level of development’ (GLD). The percentage reaching GLD is the key performance measure, which is shown in the Ofsted Inspection dashboard but is not in the publicly available performance tables.

Phonics Screening Check (PSC)
Carried out at the end of Year 1, pupils attempt to decode 40 words, half of which are real and the other half made-up. Pupils managing to decode 32 or more out of 40 have achieved the expected standard, and the percentage doing so is another key measure. As for EYFSP, school results are presented in the Inspection dashboard but are not available in the public domain.

Key Stage 1 (KS1)
At the end of year 2 pupils receive a teacher assessment in reading, writing, maths, and science. In science pupils are simply deemed to have met or not met expected standards (EXS or HNM). In other subjects the majority of pupils are assessed as either working towards (WTS), working at the expected standard (EXS) or working at greater depth (GDS). Pre-key stage assessment frameworks are available for those pupils that are working below the standard of the curriculum. Pupils take tests to inform the overall teacher assessment in reading and maths; there are no tests for writing and science. A grammar, punctuation and spelling test is provided but it is non-statutory and no data is collected. The DfE collect pupils’ overall teacher assessment in each subject – they do not collect the test scores – and the percentage achieving expected standards and greater depth in reading, writing and maths are the key measures. Pupils’ KS1 results also act as the baseline for primary school progress measures but this may change in future if the DfE implement a baseline at the start of reception. A school’s KS1 results are shown in the inspection dashboard but are not available in the public domain. 

Key Stage 2 (KS2)
At the end of Year 6, pupils sit tests in reading, maths, and grammar, punctuation and spelling. 
Achieving a score of 100 or more indicates that the pupil has met the expected standard and a score of 110 is deemed to be a high score. There are no tests for writing and science so pupils receive a teacher assessment based on the KS2 teacher assessment frameworks*, and these mirror the format of KS1 with a binary result for science (HNM or EXS), and more differentiated outcomes in writing (WTS, EXS, GDS). Pre-key stage frameworks are used to assess pupils working below the curriculum standard, with more pre-key stage categories than at KS1. The DfE collect test scores and teacher assessments but headline measures are mainly based on scaled scores in reading and maths, and teacher assessment in writing.  Headline measures include:
  • % achieving expected standards in reading, writing and maths combined (1 measure)
  • % achieving high standards in reading and maths, and greater depth in writing (1 measure)
  • Average scores in reading and maths (2 separate measures)
  • Average progress in reading, writing and maths (3 separate measures)
There are floor standards linked to these results with an attainment floor set at 65% achieving expected standards in reading, writing and maths combined, and progress floor thresholds that change each year. Schools' results are available in the public domain as well as in the inspection dashboard.

Note that only reading and maths scores are used for Progress 8 baselines.

*Teachers also make an assessment of reading and maths. This data is collected but is not used in headline measures.

Links:
Performance tables: https://www.compare-school-performance.service.gov.uk/
DfE Statistics: https://www.gov.uk/government/organisations/department-for-
#education/about/statistics
Key guidance: https://www.gov.uk/education/school-curriculum

Tuesday, 6 June 2017

The Data Burden

In implementing a new approach to assessment and tracking we must first weigh up our desire for data against any impact on workload. Sometimes those seemingly minor tweaks can produce a butterfly effect, placing huge demands on teachers' time that are most likely disproportionate to any benefit gained from the data. Before embarking on a new assessment journey always start by asking: who is this data for? what impact will it have on learning?

Many schools have implemented systems with a small number of key learning objectives in each subject, encouraging teachers to have the confidence to make judgements on pupils' competence within a broad framework. Some schools however have broken these key objectives down into many smaller steps - perhaps to enable progress to be measured over shorter periods - and this is having an adverse impact both on teacher workload and possibly on learning itself as we seek to assess and record everything a pupil does. This may be done to improve accuracy but such micromanagement of assessment is undermining professional judgement and is unnecessary, demoralising and counterproductive. There is a great irony in procedures put in place supposedly for the purposes of school improvement that actually take teachers' time away from teaching. It is hardly surprising that some headteachers are experiencing a backlash. An anti-tracking rebellion is underway in many schools.

I recently visited a school to help them set up tracking for the reception year. Initially the school wanted to track against development matters statements in each of the areas of the early years foundation stage profile, making an assessment every half term. After discussion, they decided that they would just focus on reading, writing and maths. I took a look at the list of statements - there were around 60 for each subject across the various month bands. I got out my calculator:

60 statements

3 subjects

6 terms

45 pupils

That comes to 48,600 assessments that the Early Years teacher would have to make across the course of year.

Let's say each assessment takes 5 seconds to enter onto the system. This works out as 9 days per year for one teacher.

Perhaps the teacher wouldn't have to update each assessment each term but you take the point. It's a potentially huge burden. Thankfully the headteacher and deputy went a bit pale and decided to go back to the drawing board. Their tracking system now requires the teacher to record an assessment against each of the 17 areas of the foundation stage three times per year. Somewhat more manageable and the early years teacher was certainly happier. They can spend more time doing more important stuff, like teaching.

But yesterday I had a lengthy twitter chat with awesome @misterunwin who challenged my thinking further. In his school they do no objective level tracking at all in their system. No box ticking, no rag rating statements, no assessment by numbers. Teachers make an overall judgement of where each pupil is at each half term and this produces all the data the school needs for analysis and reporting purposes. Teachers therefore spend very little time on data entry, freeing them up to do the stuff that has real impact: planning lessons and teaching children. Detailed pupil progress reviews are used to reveal gaps, inform next steps and identify where further support is needed. Staff are very happy with this arrangement, and improved staff morale will obviously benefit the school in many ways.

I have always tried to encourage schools to keep lists of objectives to an absolute minimum; to strike that balance between impact on learning and impact on workload. However, I completely understand why schools will want to reject any approach to tracking that takes teachers' time away from teaching, which is why some schools are choosing not to track in this way at all. I doubt that we are about to see schools ditching tracking systems en masse - I'm certainly not advocating that, by the way - but we do need to be mindful of the how these approaches can eat into teachers' precious time and the adverse impact this can have.

Start by getting your calculator out.

The beginning of the end

The DfE's primary assessment consultation, which closes on the 22nd June, focuses on a number of aspects of statutory assessment but it really centres on the baseline, the point from which pupils' progress to the end of KS2 will be measured. The DfE's preferred option is to set the baseline at the beginning of the reception year, and this has been supported by the likes of the NAHT (with caveats), the Headteachers Roundtable, as well as numerous headteachers and senior leaders that I've spoken to over the past year or so. I do, however, recognise that this is a contentious and emotive subject with many in the profession being understandably concerned and opposed to the idea of assessing children at such a young age. Is it possible to assess 4 year olds with any degree of accuracy? What about the pupils' month of birth, which can make such a big difference at that age? And can assessment be disruptive and stressful for children if they are not fully settled in to school life?

These are valid concerns but most primary schools already assess children on entry into reception so this is not new. No doubt the primary reason for carrying out such assessments is to support children in their learning but all too often schools are seeking to establish a baseline from which to measure progress. Pupils are therefore shoehorned into various bands with associated point scores in order to count steps of learning, or plotted onto RAISE-style progress matrices, which show them moving from one band to another. Those that progress, for example, from the 30-50 low band on entry to meet expected standards at KS1 are deemed to have made 'above expected progress', and are colour coded green or purple for good measure. Simple stuff that usually satisfies the demands of governors, the LA advisor and even an Ofsted Inspector.

The problem is that this data is fairly meaningless. On-entry assessments, carried out for the purpose of teaching and learning, are being commandeered for progress measures in order to respond to the increasing pressures of accountability. Such conflicting aims result in perverse incentives, which inevitably skew the data.  It is therefore no surprise that most pupils, according to schools' own data, are below average on entry and appear to make good progress across key stage 1. The reality, if the data remained true to its intended purpose, may look somewhat different and would probably be more informative, too.

The other issue is that many of these assessment practices are immensely time consuming, and it is often in the reception year that tracking is at its most excessive. Teachers dutifully tick off numerous development matters statements, in order to 'level' a child and supposedly measure their progress, despite this being contrary to the purpose of the assessment. It is, in short, a colossal waste of time. Would it not, therefore, be preferable to have a dedicated and universal, standardised baseline assessment, which would afford more robust comparisons of pupils, cohorts and schools, instil greater confidence in progress measures, and free other forms of assessment from the damaging influence of perverse incentives?

However, this is clearly an unpopular opinion. Someone recently suggested I was confusing assessment with accountability, of losing sight of, or perhaps never really understanding, the true purpose of assessment. As a data analyst I admit I am more focussed on accountability and performance measures - that's my job - but I do understand that the main purpose of assessment is to understand what pupils have learnt, to identify gaps and barriers, and inform next steps; and that these principles are put at risk by accountability. However, doesn't accountability in education require some form of assessment? Aren't they inextricably linked? Or am I being naïve or narrow minded? Perhaps this is more about confusion over the purpose of assessment: formative or summative, low or high stakes, for teaching and learning or monitoring school standards. Can assessment be all these things without getting wrenched apart in a tug-of-war between such opposing forces? It would appear not. We only have to look at how the Foundation Stage Profile is being put at risk as pupils' development in specific early learning goals is used to establish prior attainment groups for key stage 1 measures in the Inspection dashboard. Schools are, for the first time, concerned about having too many 'exceeding' pupils, fearing the impact on future headline meaures. And concerns about the validity of key stage 1 assessment, used as a baseline for key stage 2 progress measures, are nothing new.

A baseline therefore, whether taken in the reception year or at the end of key stage 1, is most robust if it has a single purpose: to act as a start point for future progress measures. There may be some formative by-product but that's not the main reason for carrying it out. Perhaps the reason why one particular assessment - one rooted in the principles of the foundation stage profile - became so dominant the first time round, was because we lost site of the main purpose of a baseline assessment, or never truly understood it in the first place. Either that or it was a protest vote from a profession concerned about yet another accountability measure. But let's face it, the purpose was never very well explained; that the baseline was required to produce a standardised score, which would be used to construct prior attainment groups for a future VA measure. Pupils' scores at key stage 2 would then be compared against the average score of pupils nationally with the same baseline score. That's pretty much it.

Accountability measures are not going anywhere soon, so we have to consider whether we want an accountability system based on attainment or progress. Most would probably go for the latter and so we need to work out how this is best achieved. It makes sense to measure progress from the earliest point possible but this doesn't necessarily have to be the beginning of reception year. It could be from the end of reception year, which would mean modifying the EYFSP, or from a separate assessment at the beginning of year 1. Whatever happens, it is obviously fairer to judge school performance on the basis of the progress pupils make and we need to recognise that the current process of measuring progress from key stage 1 to 2 is flawed, inaccurate and not fit for purpose. Future measures need to be far more robust, more standardised, and take account of as much of pupils' journey through school as possible.


Either that or scrap the entire system and start again.

Sunday, 21 May 2017

This is a low

I often use a race analogy to explain value added, to help people understand how we can measure progress without levels and why we don't need to have data in the same format at either end. Here's a simple example:

Imagine you enter a 10k race, which is part of a series of 10k races being held across the country on the same day. When you register you are asked what pace group you’d like to run in: slow, medium, fast. A keen runner, you choose to go in the fast group and you're handed a green vest to wear. Obviously, the medium pace runners get orange vests, and the slower group wear red (everyone loves a RAG rating system). You feel good that day, having trained hard, and run your race in 41 minutes. You’re thrilled because you’ve run a PB and you’re 10 minutes faster than the average time for your race that day. Even better, you find out you are 12 minutes faster than the national average time for the whole series. Unfortunately that’s not what the race organisers are interested in; they’re interested in how your time compares against the national average time for the green vest group, which happens to be 37 minutes. Despite being way faster than the overall average time, you are 4 minutes down on the average time for your group. Your value added score is therefore -4.

This is how VA works: it involves comparing one result against the average result of those in the same start group nationally. Here we have a start defined by a colour and a result in a time format; for KS1-2 measures we currently have a start defined by a sub level and a result in scaled score format. Same thing. 

Whilst I much prefer value added to the old levels of progress measure - it's rooted in some form of reality after all - it does have one serious flaw: SEN and EAL pupils are often expected to cross the same line in the same time. 

The issue is that many EAL and SEN pupils have comparably low start points, and are therefore placed into the same prior attainment groups, effectively treating them as similar pupils. And this means they will be compared against the same benchmarks at KS2. As we know, many EAL pupils make rapid progress and score well in their KS2 tests, whereas SEN pupils do less well. The end of KS2 estimates against which each pupil in the prior attainment group is compared, being an average of the performance of SEN and EAL pupils, tend to be too high for the former and easily attainable for the latter. The issue is exacerbated by the current system of low nominal scores assigned to the pre-key stage assessments, which almost guarantees that SEN pupils can only obtain negative progress scores whilst EAL pupils excel against their benchmarks.

We can return to our race analogy to illustrate this issue further. Imagine our pace groups are defined by how many steps runners could take when they were 18 months old. The runners wearing red vests were those that couldn't walk at that age. But perhaps some of those have gone on to be fast runners whilst others have continued to have difficulty walking. On race day they are in the same group, in the same vest, and each of their times will be compared against the overall average time for group. Hardly fair.

This issue needs resolving somehow. Introducing some form of CVA is an obvious answer - a measure that recognises the difference between SEN and EAL pupils - but is likely to lead to a proliferation of SEN pupils and a corresponding decline in those registered as EAL. Removal of pre-key stage pupils from progress measures is also a possibility but that may result in a big increase in pre-key stage pupils as schools seek to get certain pupils discounted. 

I'm not sure what the answer is but it needs serious thought because as it stands, schools are hammered if they have SEN pupils, especially if they are pre-key stage. 

The DfE stated that they wanted measures to reflect the progress made by all pupils.

Time to make good on that.