Deploying Trainers in Technology Assisted Review (TAR) without “Spoiling the Broth”

Leaving little room for interpretation, the court in Coquina Investments v. Rothstein, stated that the defendants’ litany of ediscovery project management pitfalls (which involved over 200 attorneys across two firms) culminated into a “case of too many cooks spoiling the broth.” While Coquina Investments involved format of production issues, the same rationale applies when deploying trainers in technology-assisted review (TAR) —too many trainers can lead to inconsistency and poor machine learning.

Taking Control of the Technology-Assisted Review Kitchen

Using TAR in litigation is strikingly similar to working in a professional kitchen. There are many parts moving on parallel tracks. Just like a pastry chef may begin working on dessert while a grill chef prepares the main dish, you may have reviewers allocated to train a recently found hard drive while a sub-team performs corrective training on a production set. And above all else, in either scenario, nothing leaves the kitchen without a taste test (quality control). But perhaps the most difficult task involves assigning appropriate roles to a diverse cast of employees during the stages of machine training.

• Lead Attorney: The Chef de Cuisine—in charge of all things related to the kitchen. This role involves making executive decisions like when to stop review, how to provide additional training and who will train the machine.

• Subject Matter Experts (SMEs): The Sous-Chefs—second-in-command to the Chef de Cuisine. These are attorneys that have a firm knowledge of the nature of the case and the issues involved. They are capable of making high-level decisions and have an expansive knowledge of the dispute.

• Contract Attorneys: The Chefs de partie—line cooks responsible for certain areas of production. These are attorneys who are comfortable and trained on the issue at hand, but do not have the level of knowledge possessed by Subject Matter Experts.

Choose Your Recipe

The Chef de Cuisine works closely with the Sous-Chefs to ensure that everyone clearly understands the basics of the recipe so that when the Chef de Cuisine (the Lead Attorney) is out of the “kitchen” the quality of the output remains constant.

When it comes to dedicating a team of SMEs to train the system, the adage “less is more” carries the day. As discussed in a document produced by the TREC 2008 Legal Track, determining whether a document is responsive or not responsive is a deceptively subjective process.  Lawyers “draw lines”—often at different places—across a number of determinations like “the nature of the risk posed by production, the party requesting the information” and the willingness of the production party to face a challenge for underproduction. Because the risk of inconsistencies in deciding responsiveness is exacerbated by the introduction of more trainers, rarely will you want more than five SMEs training the system. The restaurant owner mutters, “but my project is big, there is no way that I can rely on only five reviewers.”  Generally, two to five reviewers can handle the targeted review load for even a very large project. The total amount of training documents will vary depending on if you plan to “seed” the system (and how much “seeding” you plan to do), the number of documents in your data set and your desired confidence level. Ultimately, responsiveness decisions made on this fraction of documents will be extrapolated to all remaining documents in the data set; it becomes critical that the SMEs are in sync with the goals and structure of the case.

Reduce and Stir

While the ideal structure for deploying this handful of SMEs is still up for debate, there is common consensus that there must be some process in place to arbitrate consistency when responsiveness disputes arise. I’ve seen some interesting hierarchical training structures over the years designed to handle training disputes. These are some of the most common:

Finally: Tasting the Broth

An effective document review and an efficient kitchen both rely upon QC measures to ensure quality and consistency of output. A well-designed plan for validating the automated technology-assisted review output is key to knowing when to stop training for quality and when the documents are ready for consumption at the next stage of the case. Where the Chef de Cuisine is responsible for ensuring that only quality dishes leave her kitchen, the Lead Attorney is also responsible for the quality of the data in her case. Only when quality control measurements reflect defensible levels of recall and precision will a Lead Attorney be in a position to move beyond first-pass review and plate the production for the requesting party—Bon Appetit!

To gain hands-on TAR experience, register now for the newest educational course offered by Kroll Ontrack, TAR Learning Labs.  The next Learning Lab is coming up in Minneapolis, MN, in early June.  Sign up soon, space is limited!