The Inaugural Study
Litigants
on both sides of the “v.”, as well as judges and academics, have long
been interested in obtaining independent studies comparing the time,
cost and accuracy of traditional, manual document review processes with
the use of electronic document review tools. If it is shown through
such studies that electronic applications can reliably improve the
results of manual document review (or are at least as good), litigants
stand to save millions of dollars in review costs, with better results.
The largest component of discovery for litigation or investigation involves the process of humans reviewing documents to determine if they are (a) relevant to the matter and (b) subject to a privilege that would protect them from disclosure. In recent years, several services and software vendors have developed sophisticated processes and tools for automating document reviews. To date, however, there have been no objective third-party studies to prove how these processes and tools compare to human review as to quality, cost, and timeliness.
The Institute's inaugural study compares modern review technologies with a manual review process completed to respond to a Department of Justice document request.
The Data
The Data Provider: A Fortune 20 US Company
Data Provider, seeking to acquire a Fortune 100 competitor in 2005, responded to a Department of Justice Second Request for Documents by performing a traditional human review of a large volume of data involving nearly 80 custodians:
Starting point:
- 1 .5 million email messages
- .3 million loose files
- .6 million scanned documents
Eliminating duplicates, humans reviewed over 1.9 million items
Human document review time involved 150 people over 10 elapsed weeks
Resulting contract lawyer costs: over $4 million
Produced to DOJ: over 150,000 items
Processing Vendor costs for supporting the document review: over $3 million
The Institute's inaugural study compares modern review technologies with a manual review process that was prepared in response to a Department of Justice document request.
June 2009 - Results Submitted for Publication
Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review
Herbert L Roitblat |
Anne Kershaw |
Patrick Oot |
Abstract of Forthcoming Journal Article
In litigation in the US , the parties are obligated to produce to one another, when requested, those documents that are potentially relevant to issues and facts of the litigation (called "discovery"). As the volume of electronic documents continues to grow, the expense of dealing with this obligation threatens to surpass the amounts at issue and the time to identify these relevant documents can delay a case for months or years. The same holds true for government investigations and third-parties served with subpoenas. As a result, litigants are looking for ways to reduce the time and expense of discovery. One approach is to supplant or reduce the traditional means, having people, usually attorneys, read each document, with automated procedures that use information retrieval and machine categorization to identify the relevant documents. This study compared an original categorization, obtained as part of a Department of Justice Request and produced by having 225 attorneys review each document with automated categorization provided by two legal service providers. The goal was to determine whether the automated systems could categorize documents at least as well as human reviewers could, thereby saving time and expense. The results support the hypothesis that machine categorization is no less accurate than employing a team of reviewers at identifying relevant/responsive documents. Based on these results, it would appear that using machine categorization can be a reasonable substitute for human review.