eDiscovery and Document Review processes have again come under judicial scrutiny in the long running Post Office scandal, now the subject of a Public Inquiry.

The scandal is generally considered to be the most widespread miscarriage of justice in British legal history, spanning a period of over twenty years; and even today many victims still await justice.

By way of brief background, the scandal related to a flawed point of sale IT system called Horizon that the Post Office rolled out to the individual post office shops around the country, run by individual sub-postmasters and sub-postmistresses (“SPMs”). But Horizon was plagued by bugs, errors and defects, which led to discrepancies and shortfalls in the accounts of the individual post offices. Thousands of SPMs lost huge sums of money as they were pressured into making up the mysterious shortfalls. Between 2000 and 2014, the Post Office prosecuted 736 SPMs, an average of one a week. Some went to prison following convictions for false accounting and theft. Many SPM’s lives were ruined. More than 60 SPMs have since died without receiving any justice. At least four committed suicide as a direct result of the scandal.

The conduct of the Post Office throughout and still to this day has been almost beyond comprehension. There is a huge amount of information about this online, but for this article I am only going to look at issues around disclosure.

Part of the purpose of the current ongoing Public Inquiry is to investigate what the Court of Appeal described (in reference to the criminal prosecutions) as “pervasive failures in disclosure for over a decade”, failures “so egregious that a prosecution in any of the Horizon cases was an affront to the conscience of the court”. The Court of Appeal further said that the Post Office’s “approach to investigation and disclosure [was] being driven by what the Post Office considered to be in its best interests rather than that which the law required”.

If that isn’t remarkable enough, the Post Office and its advisors have been criticised by the Inquiry for continuing failures to make disclosure to the Inquiry itself.

“I see no reason to alter the view I have expressed on more than one occasion that the failures of disclosure which have come to light are properly described as grossly unsatisfactory.”

– Sir Wyn Williams, chair of the Inquiry – 5 September 2023

In particular, a document, referred to “Appendix 6” or the “Identification Codes document”, which the PO now accepts should have been disclosed to the Inquiry pursuant to requests made in February and June 2022, came to light following a Freedom of Information Request made by an Investigative Journalist in May 2023. As part of the internal documents forming the PO’s internal prosecution process, individual SPMs were to be categorised according to various racial slurs as set out in the Identification Codes document.  Suffice it to say that it is a horrific document.

Investigation of why Appendix 6 was not disclosed has uncovered a range of issues with the eDiscovery processes adopted by the PO and its advisers. They all overlap, but broadly consist of issues around:

• Use of Search Terms;
• Approach to Family Documents;
• De-deduplication;
• Data Governance and Database Set-up
• Use of law graduates to conduct the review;
• Issues around accountability and processes.

Let’s look at each of these in turn.

Search terms

When a request from the Inquiry was received, the approach adopted was to use a very narrow set of search terms.

For example in response to a request for

“Copy of [Post Office] Investigations Policy (together with all iterations of the same since 1999 that are within [Post Office’s] custody and control)”

the search terms used were:

“Policy’ AND (‘Investigat*’ or ‘Prosecut*’ or ‘Whistle’).”

(for anyone unfamiliar with search term syntax, the ‘*’ would bring in any words that start with the preceding letters, so eg Investigat* would capture investigate, investigation, investigations etc).

Using “policy” rather than “polic*” would miss documents containing the word policies if they didn’t also say policy. Similarly, for another request, the term “guidelines” was used with the consequence that documents just containing the word “guide” were missed.

Families

Documents were batched out for first and second level review based on search term hits only. Family members were not added. Reviewers apparently had “discretion” whether to look at family members via the Family View pane but there do not appear to be any documented or consistently applied principles as to when to look at families and when not to. So if the document which had the search term hit was an attachment to an email, along with other attachments, neither the email or the other attachments would necessarily be looked at.

Further, it appears the QC process only looked at documents with hits, not the other family members.

De-duplication

An item level deduplication approach was taken. Let’s say that a document which had a search term hit appeared as attachments to two different emails, and each email had a different set of attachments to it, and only this document hit on the terms. Item level deduplication would mean that only one of the sets of emails and attachments would be accessible to the reviewers. The other email and its attachments would never be looked at.

It now seems accepted by everyone on the PO side that this is not good practice and should not have happened, although there are differing accounts as to the circumstances that led to it.

Data Governance and Database Set-up

The data collected in response to the various historic and current legal processes is vast, and disparate. There are over 54 million documents held across at least four databases in Relativity. As a consequence, some databases and datasets were missed when search terms were run. This structure also meant that it would not have been possible to follow threading or near duplicate methodology to identify every iteration of a key document if it were found by reviewers, as those other documents might not be in the same database.

Issues around accountability and processes

At the hearing on 5 September 2023, there was what Sir Wyn Williams, chair of the Inquiry, described as an “unappealing wrangle” between the PO’s technology vendor and main law firm about responsibility for the de-duplication methodology. The extent to which different parts of this ecosystem had authority to act seems vague. There was disagreement as to whether there was a clear audit trail around the decision to deduplicate on an item level.

Commentary

“document providers must put in place systems and processes to facilitate that disclosure. Responding to a request for documents or information is about more than just devising search terms – it requires a common-sense approach and, in some circumstances, an understanding of a document provider’s corporate history, key personnel and document archives, and considering documents in their full context.”

– Sir Wyn Williams, chair of the Inquiry

In one sense, all eDiscovery work entails a need to balance issues of reasonableness, proportionality and defensibility, and ultimately it will be for the Inquiry to come to its own conclusions about what has happened.

But I think it is still useful to extrapolate some key principles and approaches around what best practice might look like in a scenario such as this.

1. Search term reports can only ever be a “way in” to a dataset, one of many possible approaches. But it is always an iterative process that should develop and improve as review progresses. Ther should be a feedback loop and continuous improvement to processes.
2. The point of search terms and these other approaches is to find the quickest and most accurate way to get to the relevant documents. It is natural where a search term brings up a large amount of documents to look for ways to reduce the potential review pool, by refining the searches so they are more accurate. The aim should never be simply to reduce the number of documents slated for review. It may just be that there are a huge amount of documents that need review.
3. I would always strongly recommend that all family documents be reviewed, and if a client insisted that they will review hits only, I would make sure that a very strong warning would be documented.
4. If a client still insisted on reviewing hits only, I would then suggest at least adding the covering email to the review batches, so that reviewers have a clear understanding of the context, timeline, senders and recipients involved, and what other attachments there are to that email (rather than having to go searching via the Family pane to find them).

   • “So what has happened here, then? Because we’ve got an email… to a whole bunch of people … saying “You need to comply with this compliance document”. That’s important, isn’t it, because the email shows who was distributing it, the email shows to whom it was distributed. The content of the email shows an instruction. “You must comply with this, and you’re going to be audited for your compliance”. They’re all relevant things that we get from the email that we don’t get from the [underlying document]”

   • – Mr Jason Beer KC, counsel to the Inquiry.

5. Similarly with deduplication, any process must be very carefully designed to minimise the possibility of relevant documents being overlooked, and if ultimately a client insists on a process such as item level deduplication, then the instruction needs to be interrogated and the risks need to be set out clearly in writing.
6. Again the purpose of deduplication is efficiency, it should not be used purely to reduce the number of documents for review to an arbitary number.
7. Of course cost is a significant factor in any project, but at Forcyd we are very resistant to the idea of using graduates straight out of university for review. That’s why we continue to invest in developing our network of highly experienced and highly skilled reviewers, and why we also believe that reviewers should be remunerated fairly for the work that they do (a view also echoed in the various reports around the disclosure failures at the SFO).
8. Whilst reviewers should always be encouraged to follow their own lines of enquiry, this must be managed and focussed on achieving greater efficiency and accuracy.
9. A process whereby reviewers choose whether or not to review family documents would make a review unmanageable. One of the key parts of being a review manager is understanding how many documents are due to be reviewed, how many have been reviewed and how many are yet to be reviewed. Without this, it is impossible to manage or project cost and timelines
10. There is a fundamental need to adopt a project management mindset, including:

    • Identifying clear lines of responsibility and accountability;
    • Ensuring a joined up partnerial approach between all elements of the project;
    • Ensuring clear audit trails of decisions and steps taken;
    • In the context of document review, ensuring that instructions to reviewers are complete precise and clearly documented in writing.

11. Whilst a good service provider would always be happy to make recommendations and suggestions to a client, especially a client that is not familiar with eDiscovery and Document Review, it is important that this does not creep into making decisions that ultimately only the client should make.
12. Aside from deduplication, there does not appear to have been any use of specialist eDiscovery technology, not just Technology Assisted Review or Continuous Active Learning, but longstanding methodologies such as clustering, categorisation and concept searching and the newer methodologies around sentiment analysis and communication visualisation. The primary issues in the project were the volume and variety of data, and concerns around cost and time; these techniques would surely have helped with that?

Final Thoughts

Ultimately eDiscovery and Document Review have to be seen as a solution to the very specific challenges in any particular case. The key is to design the right combination of processes and workflows to quickly efficiently and accurately get to the key data faster. Each project is a complex organism, that requires its various component parts to be working in harmony towards a common goal.

There is always the potential for things to go wrong, and no project can ever be perfect, but robust, defensible and well documented processes give assurance to the client and to other involved parties whether regulators, Inquiries, or other parties to litigation.

In terms of the Inquiry, Sir Wyn Williams has made it clear that he will continue to monitor disclosure closely moving forward. Given that there was no commitment during the hearings to modify the approach to Search Terms and Families for future document requests (as opposed to the remediation steps being taken for past failures), I suspect this is not the last we will hear of these issues.

About the author:

Robert Gradel leads FORCYD’s Document Review Services in the UK.

Robert has over 10 years’ experience in eDiscovery, assisting legal teams and corporates on complex document and data review projects, including public inquiries, big ticket commercial litigation, fraud and competition law related matters.

References and Resources

The account of the processes given by the PO and its advisers are taken from the oral hearings held by the Inquiry on 4th July 2023 and 5th September 2023,  the Directions issues by Sir Wyn Williams on 6th July 2023, 14 July 2023 and 5th September. All direct quotations are taken from these.