Ofsted must re-think the way it collects and uses evidence during inspection

Please note: the ATL website is no longer being updated and will be taken down soon.

Visit the new NEU website

Blog
20 February 2015 by ATL
Mary Bousted blogged well last year about the inadequacies of Ofsted quality assurance systems. I'd like to look at one of the fundamental problems with Ofsted’s methodology for collecting and using evidence during inspection.

The Ofsted inspection framework is essentially standards-based: individual inspectors are required to assemble evidence to show how well an organisation is performing in relation to standards laid down by Ofsted.

The processes set out in the inspection handbook have been designed with the purpose of enabling all inspectors to base all the evaluations they make on clear and rigorous evidence, so that all final judgments are objective, impartial, fair and reliable.

According to Mark Schweizer, there are two ways in which inspection protocols can determine how evidence is utilised during inspection: as it is collected, using a step-by-step approach, or after all evidence has been collected - the end-of process approach.

Ofsted currently evaluates evidence in a step-by-step fashion. Organisations are selected for inspection on the basis of publicly available evidence; evidence which is also used to tailor an inspection programme.

Once an inspection gets underway, the inspectors evaluate new evidence each day and their evaluations are fed into end-of-day meetings. Consequently, their final judgements are likely to be based on an accumulation of evaluations - rather than on evaluations of accumulated evidence.

Despite the best intentions of the designers of this methodology, the result is an inspection process that, since its inception almost a quarter of a century ago, has been able to do very little to guarantee objectivity, impartiality, fairness and reliability. Step-by-step evaluation of evidence is simply too open to the prejudgement, personal interpretation, unsubstantiated reasoning and premature conclusions of individual inspectors.

It is hardly surprising that the inadequacies of this way of working have been manifested in so many reports of inconsistencies in practice, fickle expectations from inspectors, contestable findings and questionable judgements.

It has taken Ofsted a long time to publicly acknowledge these basic errors in inspection practice, and make a commitment to addressing issues of reliability in inspections.

How Ofsted interprets 'reliability' in relation to inspection, and what steps it takes to ensure it, remain to be seen. Nevertheless, if it is to address the problems it faces with the trustworthiness of its work, it would gain much from a switch to end-of-process use of evidence: a move which would entail a profound shift in the way that inspectors are trained and deployed.

End-of-process evaluation requires individual inspectors to resist the temptation to make judgements about what they see, hear and read during the early stages of inspection. Instead it requires them to devote their attention entirely to ensuring the robustness of the evidence they collect.

Inspection will always call for inspectors to make decisions under pressure, with incomplete information for some aspects of performance and an overload of information for others. There will be times when inspectors are faced with contradictory and ambiguous information. This means each individual should be adept at deciding what evidence should be considered in the unique circumstances of each organisation.

When inspectors focus on feeding substantive and good quality evidence into the final evaluation, the resulting collectively generated judgements are more likely to be founded on what worked and what didn’t work in the distinctive circumstances of the organisation inspected.

It remains doubtful that inspection judgements will ever be totally objective, that is to say completely value-free. It is simply not possible to eliminate bias altogether from inspection methodology, but it is possible to control it and reduce it.

Incorporating an end-of-process approach to the use of evidence during inspection offers the very real prospect of generating inspection decisions that are better informed and more trustworthy, thereby increasing significantly the confidence in those judgements. Surely this makes the investment needed to change the way evidence is collected and used during inspection all the more necessary and worthwhile.

By Terry Pearson.

Terry Pearson is a former FE senior manager. He now works as an independent education consultant. He can be tweeted @TPLTD or emailed at terrypearsonltd@gmail.com

Tagged with: 
Ofsted

Comments

I personally don't believe that Ofsted can make a judgement on the overall quality of a provider during the short time frame of an inspection -- it is only a quick snapshot, as is a lesson observation.