Wednesday, 17 December 2014
Wednesday, 10 December 2014
Wednesday, 5 November 2014
Wednesday, 29 October 2014
Wednesday, 8 October 2014
I recently sent out a tweet asking for thoughts and experiences relating to small schools, particularly regarding data. Now, I could have written a lengthy, and probably rather dull, blog on the subject of data and small schools but instead I thought I'd just publish the following email, which perfectly and succinctly summarises the problems facing these settings when it comes to accountability measures and inspection. Regrettably, the author wishes to remain anonymous but you know who you are. Thank you so much for the following contribution:
I saw your tweet about wanting to hear about the experiences of small schools on specific data issues and thought I’d rather email than tweet.
I write from the perspective of being a governor and volunteer data handler for a small school (96 pupils).
Whilst small pupil numbers are a real benefit in terms of the school being able to concentrate on looking at each pupil as an individual, noting each child’s attainment and progress and the next steps required I think small schools have particular problems when it comes to handling cohort and group data. No doubt you’ll be familiar with these – though perhaps no. 4 is only just becoming an issue with more focus on this in inspections.
1. Small pupil numbers mean that we don’t have sufficient sample sizes to be confident that our collective data is telling us much at all. Our results have to be very good (or very bad!) to be classed as statistically significant on RAISEonline. Even worse if trying to analyse groups.
2. It’s very hard to spot trends because we need results over several years to collect sufficient sample sizes. By the time we can be confident of a trend through data analysis, things will have moved on anyway.
3. I have no confidence that the people who judge us by our data will necessarily be competent statisticians. In my experience some people start off their analysis with a general rider about small cohort numbers requiring caution and then throw caution to the wind and proceed to read far more into the data than it can possibly support.
4. I particularly worry about proving to Ofsted that “From each different starting point, the proportions of pupils ... exceeding expected progress in English and in mathematics are high compared with national figures” and hope that any inspector who visits us understands that if we only have 1 pupil at a particular starting point, say 2b in reading, and that child fails to reach a Level 5, our score of 0% making more than expected progress from that starting point is actually in line with a national percentage of 29%. I do hope inspectors are being briefed to read the data properly and would love to access an authoritatively written statement on this that properly explains the situation.
5. Small schools (often in underfunded ‘leafy’ areas with little or no pupil premium) often do not have the budget to be able to afford or justify what seem like expensive data handling packages – especially given the question of how much use they will really be when sample sizes are so small. Even FFT is hard to justify now the LA no longer subscribe.
6. In small schools where the head is also a class teacher and probably the premises & catering manager and there is no deputy head and perhaps just a single school secretary/administrator, he/she will have to balance data management along with all their other tasks. Where do they get time in the day to really master this subject let alone source useful software and keep abreast of new government initiatives?
7. Headteachers find it hard to discuss the progress of disadvantaged pupils with governors if there are so few pupils receiving the pupil premium that discussion will lead to identification.
Hope the above is of some help and thanks for all your work shared over Twitter,
As stated above, I think this email perfectly encapsulates the issues experienced by small schools when it comes to data, particularly that presented in RAISE, the Ofsted Dashboard and Performance Tables. There is some provision for this in the Ofsted Handbook, which headteachers of small schools should be aware of:
'Where numbers of pupils are small and achievement fluctuates considerably from year to year, inspectors should take into account individual circumstances when comparing with national figures, and should consider any available data of aggregate performance for consecutive cohorts' (Ofsted Inspection Handbook, p.20, section 60).
This gives small schools the go ahead to merge cohort data and produce aggregated attainment and progress figures for, say, the last three Y6 cohorts, which will result in a more viable data set and more meaningful analyses. On a number of occasions I've calculated 3 year aggregated VA (using my VA calculator - see other blog), which has proved to be very useful indeed and made a positive contribution to the overall outcome of inspections.
Also, small schools should access their FFT data, which provides 3 year average VA and CVA, and trend indicators. In addition, the new FFT Aspire system will allow schools to toggle between single year and 3 year averages. Potentially a very useful feature, especially when dealing with small cohorts or groups. If you are wondering whether or not inspectors will be interested in or accepting of FFT data, take note of the following:
'Evidence gathered by inspectors during the course of the inspection should include: any analysis of robust progress data presented by the school, including information provided by external organisations' (Ofsted Inspection Handbook, p.66, section 195)
I'm sure the issues outlined above resonate with staff in many if not all small schools up and down the country; and there are clearly specific challenges facing those schools. But there are ways round them and I hope that this has given you some ideas. Please get in touch if you have any questions or comments.
Tuesday, 23 September 2014
Tuesday, 16 September 2014
Friday, 5 September 2014
Wednesday, 27 August 2014
and we'll leave those cute little Awols alone.
Wednesday, 20 August 2014
I'm going off topic.
Sometime ago, a junior school head came to visit me in my office. She wanted to discuss the issue of KS1 data accuracy (obviously). I pushed my jar of pound coins towards her, strategically placed a climbing gear catalogue within line of sight, and prepared myself for some proper headteacher ranting. But this head didn't want to rant; she wanted to take some action. She wanted to do stuff. She wanted data. Which is always nice.
So, after some discussion we hatched a plan: to carry out CAT tests on entry in as many Junior schools as possible. We had no idea if this project would be of any use and what we would do with the data when we got it but it sounded like positive action and we thought it would be pretty neat, too. In the end after numerous meetings and emails, 13 out of the 20 junior schools in Gloucestershire got involved and a date in early October was set for their new Year 3 intakes to do the tests. Exciting!
The test itself is known as a PreA test and is specifically designed to be carried out early in year 3. If you'd like to learn more about these and other CAT tests, please contact GL Assessment.
I said above that we didn't know what we would do with the data, which is really true. I had a sort of, kind of idea. A CAT test provides scores for the pupils verbal, non-verbal and quantitative reasoning; it does not generate a level or sublevel that can be directly compared with the pupil's KS1 results. However, like other CAT tests, the PreA test would provide an English and Maths estimate for the end of KS2 in the form of a sublevel. I thought it would be interesting to compare these estimates with those generated using RAISE VA methodology. Not exactly a perfect solution, but compelling, in a data-ery sort of way.
So, once the junior schools had carried out the PreA tests in October last year, they sent me the data. I then converted each pupil's KS2 sublevel estimates generated by the tests, into points scores (by the way, I don't like using the term 'APS' here because they're not averages. I'm pedantic like that). Next I put each pupil's KS1 results into my VA calculator (more information on that here) to generate end of KS2 estimates using RAISE VA methodology, and took estimated point scores for each pupil. I now had two point score estimates for the end of KS2 for each Y3 pupil in the 13 junior schools taking part: one based on the CAT PreA test; the other based on their KS1 results. Neat! now all I had to do was subtract the CAT estimate from the RAISE VA estimate (the former from the latter) to find which one was highest. Positive figures would indicate that the estimate derived from the CAT tests was in advance of those derived from KS1 results; negative figures would indicate the opposite. 'So what?' I hear you shout. Fair question, but bear in mind that it's the RAISE VA estimate that the pupil's progress is measured against (well, sort of, because, actually, their estimates won't really be calculated until they've done their KS2 SATS, but we're trying here, OK?). And if the RAISE VA estimate (i.e. that based on KS1) is always higher that the CAT estimate then this could be rather worrying as it may indicate that the future VA bar will be set unrealistically high for those pupils.
So what was the outcome?
Well, the estimates based on KS1 results were higher than the those based on the CAT test in pretty much every case. I'm writing this at home without the full dataset in front of me but we're talking about approximately 600 pupils here. It was quite startling. Wanna see some data? Course you do.
So, that's it really. Deeply flawed I know, but interesting and a worthwhile exercise (the data was used by one school as part of their evidence base for inspection and proved very useful). The lack of control group is an obvious issue here and needs to be addressed in future. Ideally we'd like to get 10 primary schools to take part at some point. Traditionally schools have carried out CAT testing in Year 5 but more schools are considering alternatives. I actually think it's worth doing them earlier as you have more time to act on the data, so perhaps more primary schools would be interested in testing in year 3. Many of the junior schools heads involved in this project intend to continue using the tests as it gave them a alternative and rich source of information on pupils strengths and weaknesses, which they didn't have previously. This is a positive thing.
Anyway, that's the end of this blog. Hope it was useful, or at least interesting.
Oh, and by the way, I am now a governor of a junior school and now own new pair of climbing shoes.