Technology Assisted Review: Trick or Treat? Ralph Losey , Esq., Jackson Lewis
description
Transcript of Technology Assisted Review: Trick or Treat? Ralph Losey , Esq., Jackson Lewis
Technology Assisted Review: Trick or Treat?Ralph Losey, Esq., Jackson Lewis
1
Ralph Losey, Esq.
2
Partner, National e-Discovery Counsel, Jackson Lewis
Adjunct Professor of Law, University of Florida Active member, The Sedona Conference Author of numerous books and law review
articles on e-discovery Founder, Electronic Discovery Best Practices
(EDBP.com) Lawyer, writer, predictive coding search designer,
and trainer behind the e-Discovery Team blog (e-discoveryteam.com)
Co-founder with son, Adam Losey, of IT-Lex.org, a non-profit educational for law students and young lawyers
Discussion Overview
3
What is Technology Assisted Review (TAR) aka Computer Assisted Review (CAR)?
Document Evaluation Putting TAR into Practice Conclusion
What is Technology Assisted Review?
4
Why Discuss Alternative Document Review Solutions?
Document review is routinely the most expensive part of the discovery process. Saving time and reducing costs will result in satisfied clients.
Traditional/LinearPaper-BasedDocument Review
Online Review
TechnologyAssisted Review
5
Information retrieval effectiveness can be evaluated with metrics
Fraction of relevant documents within retrieved results – a measure of exactness
Precision
Fraction of retrieved relevant documents within the total relevant documents – a measure of completeness
Harmonic mean of precision and recall
Recall
F-Measure HotNot
All documents
Bobbing for Apples: Defining an effective search
Information retrieval effectiveness can be evaluated with metrics
Fraction of relevant documents within retrieved results – a measure of exactness
Precision
Fraction of retrieved relevant documents within the total relevant documents – a measure of completeness
Harmonic mean of precision and recall
Recall
F-Measure
1) Perfect Recall; Low precision
Bobbing for Apples: Defining an effective search
HotNot
Information retrieval effectiveness can be evaluated with metrics
Fraction of relevant documents within retrieved results – a measure of exactness
Precision
Fraction of retrieved relevant documents within the total relevant documents – a measure of completeness
Harmonic mean of precision and recall
Recall
F-Measure
2) Low Recall; Perfect Precision
Bobbing for Apples: Defining an effective search
HotNot
Information retrieval effectiveness can be evaluated with metrics
Fraction of relevant documents within retrieved results – a measure of exactness
Precision
Fraction of retrieved relevant documents within the total relevant documents – a measure of completeness
Harmonic mean of precision and recall
Recall
F-Measure
3) Arguably Good Recall and Precision
Bobbing for Apples: Defining an effective search
HotNot
Key Word Search Key word searches are used throughout discovery However, they are not particularly effective
» Blair and Maron - Lawyers believed their manual search retrieved 75% of relevant documents, when only 20% were retrieved
It is very difficult to craft a key word search that isn’t under-inclusive or over-inclusive
Key word search should be viewed as a component of a hybrid multimodal search strategy
Go fish!
10
Where are we?
What Is Technology Assisted Review (TAR)?
12
13
Classification Effectiveness
Any binary classification can be summarized in a 2x2 table Test on sample of n documents for which we know answer
» A + B+ D + E = n
14
Classification Effectiveness
Recall = A / (A+D)» Proportion of interesting stuff that the classifier actually found
High recall of interest to both producing and receiving party
15
Classification Effectiveness
Precision = A / (A+B) High precision of particular interest to producing party: cost
reduction!
How precise were you in culling out from your bag of 10,000 and ?
16
Sampling and Quality Control Want to know effectiveness
without manually reviewing everything. So:» Randomly sample the documents» Manually classify the sample» Estimate effectiveness on full set
based on sample
Sampling is well-understood» Common in expert testimony in
range of disciplines
Sample size = 370 (Confidence Interval: 5; Confidence Level: 95%)
300
370
Precision: 81%
Annual event examining document review methods
17
TREC 2011
[T]he results show that the technology-assisted review efforts of several participants achieve recall scores that
are about as high as might reasonably be measured using current evaluation methodologies. These efforts require human review of only a fraction of the entire collection,
with the consequence that they are far more cost-effective than manual review.
-Overview of the TREC 2011 Legal Track
18
Putting TAR into Practice
TAR or CAR? A Multimodal Process
Must… have…
humans!
19
The Judiciary’s Stance
Da Silva Moore v. Publicis Groupe» Court okayed parties’ agreement to use TAR; parties disputed
implementation protocol (3.3 million documents)
Kleen Products v. Packaging Corp. of Am.» Plaintiffs abandoned arguments in favor of TAR and moved forward with
Boolean search
Global Aerospace Inc. v. Landow Aviation, L.P. » Court blessed defendant’s use of TAR over plaintiff’s objections (2 million
documents)
In re Actos (Pioglitazone) Products Liability Litigation» Court affirmatively approved the use of TAR for review and production
EORHB, Inc., et al v. HOA Holdings, LLC» Court orders parties to use TAR and share common ediscovery provider
Must address risks associated with seed set disclosure
Must have nuanced expert judgment of experienced attorneys
Must have validation and QC steps to ensure accuracy
21
TAR/CAR:
Tricks Treats TAR can reduce time spent on
review and administration TAR can reduce number of
documents reviewed, depending on the solution and strategy
TAR can increase accuracy and consistency of category decisions (vs. unaided human review)
TAR can identify the most important documents more quickly
&
TAR AccuracyTAR must be as accurate as a traditional review
Studies show that computer-aided review is as effective as a manual review (if not more so)
Remember: Court standard is reasonableness, not perfection:• “[T]he idea is not to make it
perfect, it’s not going to be perfect. The idea is to make it significantly better than the alternative without as much cost.”
-U.S. Magistrate Judge Andrew Peck in Da Silva Moore
22
23
Conclusion
24
Parting Thoughts
Automated review technology helps lawyers focus on resolution – not discovery – through available metrics» Complements human review, but will not replace the need for
skillful human analysis and advocacy
Search adequacy is defined in terms of reasonableness, not whether all relevant documents were found
TAR can be a treat, but only when implemented correctly» Reconsider, but do not abandon, the role of:
» Concept search» Keyword search» Attorney review
25
Q & A
26