A lot of regulatory investigators are lawyers or come from a legal background. I would argue that a...
Are we there yet? When have you reviewed enough evidence to move on?
In regulatory investigations, the worst thing you can do is spend a long time focusing on the wrong things. Resource constrained and time-poor regulators need to allocate their teams of experts wisely and proportionately – not only within each case but also across their whole portfolio of cases.
The key question, when you’re reviewing thousands upon thousands of items, is how do you know when to stop, or when enough is enough?
Reviewing too much evidence can be exhausting, but how do you know when to stop? Photo by Andrea Piacquadio
In litigation, courts apply the concept of proportionality to decide whether a discovery or disclosure exercise is worthwhile. But this assessment tends to focus exclusively on quantity – how much data you need to review, how long it would it take, and what that would cost, relative to the issues at stake in the case.
For regulators, proportionality considerations are often more nuanced. There is more emphasis and awareness on the quality of evidence, not just the quantity. It's important to avoid the temptation to double down on a particular case to try to review every evidence record you have collected.
Keeping a team of lawyers on a review long after you’ve identified the most compelling evidence could amount to a law of diminishing returns. It can be hard to justify that cost when there are other cases waiting with tranches of freshly collected data waiting to be reviewed.
In criminal cases where there is a different evidential burden, it may be necessary to conduct more exhaustive searches. However, in most regulatory cases, the question may come down to whether or not it’s an effective return on investment to keep a legal team reviewing items in a case when there is enough evidence to proceed to litigation or negotiate a settlement. In this situation, the pursuit of (perceived) perfection simply isn’t worth it.
Making informed decisions about evidence quality
Our white paper, The art of elusion: How statistical sampling techniques can help regulators manage a deluge of data, delves into some common ways regulators can use statistical sampling to make more efficient use of their resources.
Many will argue that the only way to be absolutely certain a data set contains (or doesn’t contain) evidence relevant to a case is to review every item. (That’s assuming your human reviewers don’t make any mistakes.)
However, our paper shows that you only need to sample and review a few hundred or maybe a few thousand items to make informed judgments about the relevance of a produced (but as yet unreviewed) data set.
If you were reviewing the prevalence of responsive material in a produced data set, for example, you could take a small sample and use those numbers to estimate the proportion of responsive material in the remaining population of items. You might estimate that fewer than 1,000 items in a production – among the millions you received – were responsive to the production order.
You won’t know for sure, but if you understand statistical sampling, you can calculate how likely your estimate is to be wrong and how wrong it’s likely to be. For example, you might estimate with 95% certainty that the production contains between 950 and 1,050 responsive items.
Use your elusion
Perhaps a more compelling opportunity for regulators who have doubts about the quality of a produced dataset is elusion sampling – asking for a small sample of the data that wasn’t included in the production. This provides a court-recognised method to validate the effectiveness of a production and confirm the prevalence of the data that what was left behind and not produced.
Our paper shows how to efficiently use elusion sampling to build confidence in a production and know that relevant evidence was not left behind in the discarded data.
Enough is enough?
These statistical techniques can help you answer the pressing question that regulatory investigators often face: have we reviewed enough items to proceed to the next stage of our investigation?
Rather than relying on exhaustive reviews or gut instinct, you can use your knowledge of sampling to understand, for example, the likely prevalence of responsive items among the items you haven’t yet seen. Then the question becomes, is it worth reviewing another 10,000 or 100,000 evidential records if there are probably only 10 to 20 additional responsive items among them?
In a situation where you have limited resources (and that’s most situations for regulators), this let you make an informed decision about whether to go down the path of exhaustive review and diminishing returns or to obtain a better return on investment by working with the evidence you have already uncovered and move the investigators or the legal review team on to the next case.
Download our white paper
If this sounds like something that would help you complete more investigations, more efficiently, download our white paper today.