Originally published for Assembly Data Trends series: https://assembly.education/how-to-analyse-exam-results/
It’s been three whole years since we’ve had exam results in England, and what three years it has been! This issue of Data Trends provides 7 top tips on how to analyse and report on your school or MAT exam results this Summer, from our former Vice Principal, Dan Stucke.
Exam Results 2022
Last week saw the release of A-Level results across the country, and a bit of digging shows some intriguing trends.
The CAGs and TAGs process through 2020 and 2021 saw huge grade inflation at both A-Level and in GCSEs. Ofqual set a goal this year of reducing those gains by half, then returning to 2019 levels by next year. Largely they seem to have been successful in this aim. The aim is for the same to happen to GCSE results later this week.
Sam Freedman has written a sensible article on the grade inflation that took place, the return to exams and why, love them or hate them, they are the fairest system we have. A level 2022 results: Exams may be an imperfect system – but it’s the fairest one we have (tes.com)
Professor Lindsey Macmillan has analysed the overall change in grades this year and dug into the differences between different centre types. An interesting read, particularly around the differences in how much different institution types have or have not reduced their previous gains – Turbulence on the glide path: A-level results 2022 | UCL Centre for Education Policy and Equalising Opportunities (CEPEO)
FFT have analysed the differences in each subject in some depth – A-Level results 2022: The main trends in grades and entries – FFT Education Datalab
I delved deeper into the trends at a regional level and found some quite unusual patterns. Full read here: A-Level Results 2022 – Regional Analysis | LinkedIn.
How To Analyse Your Exam Results
This is the first year in the past 12 where I haven’t spent the last week of August and first week of September analysing exam results for schools and writing various reports for Heads, Governors, Executives etc..
We know that this year in particular will see some considerable variations in results, whatever the national picture on Thursday, here are some of my top tips for school and MAT leaders conducting their own analyses.
1. Know Your Audience
Who is your analysis for? What key information will they need to know to be able to do their role effectively? What format is best for them to receive the information and keep it to hand through the year ahead?
I used to create several different styles of reports for different audiences, a detailed report for SLT, modified version for governors and a very visual 2-page only summary for staff and other stakeholders.
2. Tell Your Story
Data analytics is about telling a story with numbers. There will be a narrative that goes with your results, with the impact of Covid and return to exams after TAGs & CAGs this could be a crucial tale to tell this year. Work with the wider leadership team to establish what the story is, then ensure the data you present and the narrative that goes with it objectively tells this tale to the audience.
3. Identify Your Key Metrics
What are the key performance indicators? Some of these will be obvious and set by the audience of your reporting.
This year is not like others! Attainment & Progress 8 will not be available until later in Sept/Oct when the DfE publish the mapping tables. Schools/MATs should not be attempting to calculate a P8 figure using methodology and tables from previous years, these will be inherently very inaccurate. This is the first year of exam results where students taking them completed the reformed SATs in Year 6, so the first time we have a proper table mapping a KS2 standardised score to an expected Attainment 8 score. Combining this with the unknown impact of Covid and attempts to redress previous grade inflation mean that the tables will be completely different to anything available from previous years. Your initial analysis will have to focus on more traditional attainment figures such as average grade, percentage of 4+/5+/7+ grades etc..
4. Contextualise Your Results
Linked closely to telling a story, it is important that your analysis doesn’t just focus on the grade outcomes. Tell your audience the context of this year group. What proportion were disadvantaged? SEND? EAL? What was the breakdown of their attainment at KS2?
Crucially this year, how were they affected by Covid? What was their attendance? How did this compare to years gone by? What was their teachers’ attendance? What does the wider pastoral data tell you about their journey through secondary school, and what impact could this have had on their results?
Where appropriate report on trends over time. You will need to use your judgement this year on where to do this and where not. If you are including results over the past 4 years (there is logic in including 2019/20/21/22 data as this spans the pre/’post’ covid period), make sure your audience know how to interpret this. Make it crystal clear in your commentary where comparison can be fairly made, and where they can’t.
6. Actionable Summaries
A report for reporting’s sake is not a good use of time and energy. Your analysis should focus on lessons learned, that can be applied to the learning of future year groups. What were your successes? What were your areas for improvement? What insights does the data tell you about these strengths and weaknesses? Which of these need further analysis to get to the bottom of? If your report isn’t focused on the questions of “So what?” and “What next?” then you might as well save the time and energy and just send out a link to the performance tables when they are published later this year!
7. Automate & Standardise Your Reporting
Reporting takes time to do well, usually the time of senior staff, so it is expensive. If your systems are poorly set up it can be incredibly inefficient. I standardised my reporting throughout the year, so that interim reports were of the same format as end of year reports (sometimes abbreviated but still all in the same format using the same systems). This made them easier to produce and ensured the audience were well versed in the style, language and data used.
I wish I had had one system to pull all the data together automatically. Sadly, my data used to be siloed in various systems, I streamlined the integration of this as best I could but it still involved multiple systems, spreadsheets, screenshots and powerpoint to pull together manually.
Needless to say, this is where Assembly can help. Assembly Pro can automatically collate your data from your MIS, assessment packages, safeguarding systems etc., each and every night. You can create bespoke dashboards based on our reporting tools. This means you can multiply end of year dashboards ready for different audiences. The collation and presentation will be done for you in advance, all you will need to do is analyse and tell the story of the data.
If you are asking others to analyse data for their area, Subject Leads for example, provide the data to them ready formatted and presented. Their job should be analysis and story telling too, not number crunching and graph drawing. Again, look to use the same formats and systems all year long so this is habitual and efficient.
If you would like to know more about our data analytics services, please don’t hesitate to get in contact with us. If you are a CEO who is presented with a beautiful data analysis from one of your team later this term, speak to them and find out how much time and effort it took from them and their team. Resources are tighter than ever, could you invest in better systems to give these important leaders some time back this coming year?
We wish you and your students the very best for Year 11 results later this week. This group of students have achieved these grades in the most exceptional of circumstances. This needs celebrating up and down the country. They couldn’t have worked harder, and despite everything Covid had to throw at them they should all be incredibly proud.