The UK’s Serious Fraud Office (SFO) are in the news again, following the collapse of another prosecution due to disclosure failings. The case concerned allegations against three former executives of G4S related to electronic monitoring contracts, and follows the collapse of a similar case against former executives of Serco, along with a number of other high profile collapsed or overturned cases in the past few years.

I wanted to find out more about the disclosure failings in the G4S and Serco cases. There isn’t much in the public arena about the G4S case yet, but following the collapse of the Serco case, the SFO commissioned Brian Altman KC and Rebecca Chalkley to write a report into those failings which was published in July 2022 (and can be found here).

Whilst some aspects of the Report deal more specifically with the SFO’s position as a public body, with significant resourcing and staffing constraints, and operating under the Criminal rather than Civil disclosure rules, there is much in the report that makes sobering reading for any eDiscovery professional.

Among the failings detailed in the Report are: 

• Failure to disclose documents supportive of the Defence’s arguments
• Failure to provide documents specifically requested by the Defence
• Sets of documents, some key, completely overlooked for review
• Documents recorded by the SFO as disclosed but which the Defence claimed not to have received

Here is my perspective on some of the key areas highlighted in the Report.

Coding Panel

The Report helpfully includes screenshots as well as detailed analysis of the Coding Layout used by the SFO.

As well as being cumbersome to work through, the Layout required reviewers to make tagging decisions alongside rewriting a Description text box incompletely prepopulated by metadata, and to complete one or more text boxes giving reasons for various tagging decisions.

Asking reviewers to complete multiple tasks may well be unavoidable, but it usually comes at the cost of speed and accuracy.

Reviewers were asked to complete a number of addition tags (with reasons) around ‘disclosability’ as well as relevance, but as will be seen, some reviewers claim they were unaware that they had to review for disclosability.

The Coding Panel was not designed in a way that could have distinguished between a reviewer deciding that a document was not disclosable and a reviewer failing to make any determination on disclosability (i.e., there is no specific ‘not disclosable’ tag).

The system wasn’t set up with logic which would have prevented reviewers moving on without completing all necessary tags.

In short, the Layout could have been much better designed to create a more efficient flow through its various parts, and to ensure that all necessary decisions were made, with the reviewer unable to move on until they had.

I also wonder if some of the text boxes where common re-occurring themes would be inserted could have been converted into tags, for efficiency and consistency. Tags could also have been designed to work alongside the metadata to automatically generate more accurate descriptions (in much the same way that tags are used to generate US privilege logs). And finally a more pertinent set of tags could have proved a much better way to understand the documents rather than relying on the text in the Description field.

Review Material

The Report is particularly critical about the review materials available to the team:

“We question how much of it really assisted them. We think they had far too much disparate, detailed and voluminous internal guidance documentation over time, risking a loss of important messaging, particularly as regards focus on the real issues in the case. The Document Review Guidance document was of little practical utility and offered contradictory guidance.”

There was no version control when instructions changed, and no audit to check reviewer had read and understood new material.

When Defence Statements were served, bringing into scope important new issues, the team were just sent the Statements and were expected to work out for themselves how they affected the review, with no new guidance or Defence specific tags.

It also seems that when instructions to the team changed, they did so on a “don’t go back” basis, but with no plan about how to deal with previously tagged documents.

Perhaps most significantly, at least two reviewers failed to tag documents as disclosable because, they say, they had though their role was only to tag for relevance, not also for ‘disclosability’.

Quality Control

The Report criticises the lack of any consistent Quality Control (QC) and Performance Monitoring processes.

QC was mostly by way of occasional random dip samples, and even then only one category of “bucket” was sampled at any one time. The dip samples were small by comparison to the number of documents reviewed, and were discontinued due to low overturn rates.

There was no consistent ongoing QC, and no targeted QC, even when there were concerns about the very low number of relevant documents being tagged as disclosable.

The Report notes the missed opportunity for early intervention that could have identified and corrected misunderstandings, such as those reviewers who had believed they were only reviewing for relevance, not for relevance and disclosability.

Reviewers appeared to have adopted some bad habits, such as mass or family group tagging before reviewing the full group of documents first, or without realising that mass tagging would overwrite previous tagging.

The platform that the SFO were using did not allow for separate QC Coding Layouts to be created, so overturns were by way of overwriting original tagging with no audit or record kept.

A good QC process should start as soon as the review does, and evolve as the review progresses, using what is learned in the review to adopt better targeted searches and methodology.  It should also be fully traceable and recorded, and used to identify team and individual reviewer issues, so that they can be resolved and improved swiftly.

Review Management

Commenting on the two reviewers who said they thought they were only required to tag for relevance, the Report says that this:

If accurate, suggests serious systemic failures of communication, tasking, training, guidance and/or oversight and monitoring

One of the more unusual aspects of the Report is that it delves in detail (albeit anonymised) into a handful of reviewers, particularly the ones who missed critical documents. The Report looks at their experience, pay rates, and asks for their side of their experience working on the Review.

Meetings with reviewers were held once a fortnight (far too infrequently in my view), and appear to have mostly focussed on review speeds, putting pressure on reviewers but without investigating why speeds were low or what could be done to help improve them.

It is so important to encourage regular two way dialogue with reviewers, to understand what they are seeing. This gives an opportunity for the review manager to feedback to the team any issues identified in QC, but it also allows reviewers to feed back on the challenges they face, leading to improvements in process such as adapting the Coding Layout, adding new tags, and clarifying instructions and changes.

Project Management

One of the Report’s recommendations is the engagement of dedicated PMs.

As well as organising and co-ordinating (the report notes confusion over individual’s roles and responsibilities and people having to carry out work which they were not skilled to do), a PM could have taken ownership of investigating the technological issues coming up in review: 

• Documents for both G4S and Serco were held in the same database, contrary to best practice;
• Searches would produce different results, even when the same search was re-run immediately with nothing in the database having changed
• Sets of documents were missed out of the workflows identifying documents for review
• Reviewers complained about very slow loading times
• New incoming documents would not be loaded onto the system for months, and by the time they had been passwords had expired.

Conclusion

In response to criticism following the collapse of the cases, Lisa Osofsky, director of the SFO argued that the Criminal disclosure rules are no longer fit to cope with the volume of data on these cases, citing the need for manual review. Whilst the rules are demanding, they are no less so than I have seen on Public Inquiry reviews and commercial fraud reviews, and I suspect in reality that the biggest challenge is that of resourcing.

As the Report notes, improved processes are no guarantee that mistakes will not be made. In a review of 1.8 million documents, there will be tagging errors. It is essential to have robust defensible processes (including QC processes) in place to minimise and help identify errors.

In talking about the need for all documents to undergo manual review, I also wonder if Lisa Osofsky has misunderstood the role that technology can play. AI based approaches, from processes such as clustering and categorization and/ or use of predictive coding / active learning / seeding might have been used to front load potentially relevant and / or disclosable documents to accelerate the review, as well as helping the review and QC process move forward in a more targeted and efficient way.

Finally, the Report demonstrates the benefits that dedicated and experienced review managers and project managers can bring, leveraging experience from the wider eDiscovery industry and utilising best practices.

About the author:

Robert Gradel leads FORCYD’s Document Review Services in the UK.

Robert has over 10 years’ experience in eDiscovery, assisting legal teams and corporates on complex document and data review projects, including matters related to banking finance, fraud, regulatory activities and commercial litigation.