Michael Lissner
Note: This is the third in the series of posts explaining the work
that we did to release the data donation from Lawbox LLC. This is a
very technical post exploring and documenting the process we use for
extracting meta data and merging it with our current collection. If
you’re not technically-inclined (or at least curious), you may want to
scoot along.
Working with legal data is hard. We all know that, but this post serves
to document the many reasons why that’s the case and then delves deeply
into the ways we dealt with the problems we encountered while importing
the Lawbox donation. The data we received from Lawbox contains about
1.6M HTML files and we’ve spent the past several months working with
them to extract good meta data and then merge it with our current
corpus. This post is a long and technical one and below I’ve broken it
into two sections explaining this process: Extraction and
Merging.
Extraction
Extraction is a difficult process when working with legal data because
it’s inevitably quite dirty: Terms aren’t used consistently, there are
no reliable identifiers, formats vary across jurisdictions, and the data
was made by humans, with typos galore. To overcome these issues we use a
number of approaches ranging from hundreds of regular expressions to
clever heuristics.
The first step we take is to convert the HTML files into an in-memory
tree that we can traverse and that we can query using XPath, a variable
that contains only the text of the opinion (for later analysis), and a
variable that contains simplified versions of the HTML with any headers
or other junk stripped out.
From there, the tree, text and simplified tree get sent into various
functions that extract the following pieces of meta data:
- Citations
- Case name
- Case date
- Docket number
- Judge(s)
- Jurisdiction
Of these, jurisdiction and citations are by far the hardest. The others
are fairly straightforward, though dates are often missing and must be
laboriously looked up.
Citations are extracted using our standard citation finder. We’ve
described how it works in the past
(pdf), but the basic idea
is to tokenize text into valid words and then find valid reporters
within the tokens. Whenever a valid reporter is found, you then inch
backwards and forwards from it, identifying the volume, page number,
year, and any other related information.
Finding the jurisdiction relies on a collection of about 500 regular
expressions,
each designed to find a specific court. Since the data provided by
Lawbox is rather dirty, you can see that these regular expressions do a
lot work. Unfortunately this approach isn’t enough for many
jurisdictions, and for the hard ones we go a step further.
If the regular expressions fail, the next step we take is to use the
citation information as a clue towards the jurisdiction. In many cases
it works! It’s often enough to know that a case is in the California
Appellate Reporter or the U.S. reporter. Using that information alone,
we can often figure out the hard cases.
But sometimes they’re really hard to figure out.
The really hard cases in the Lawbox collection describe their
jurisdiction like so: “United States District Court, D. Alabama”.
Doesn’t look hard, but, well, Alabama currently has three district
courts, the Middle, Northern and Southern, but it doesn’t have a generic
“D. Alabama” (at least not since 1824). For the rare case like this, we
developed a clever solution: We use the judge information in the case to
determine the jurisdiction. Since most judges don’t move too much
between courts, before we began importing anything we extracted all the
judges and made tallies of where they worked. Then, when we encountered
a case like the above, we said, “OK, who’s the judge in this case, and
where does he work?” In almost every case this worked very well, but in
some cases it didn’t and for those, we simply put the information in
manually.
For the remainder of the meta data fields listed, we employed similar
tricks, but these were the hardest examples. For the remainder of our
approach, you can inspect the code
itself.
Just be careful of hairballs.
Merging
Once all of the meta data is properly extracted, the next step is to
merge it with our existing corpus, identifying duplicates and merging
them, or simply adding new cases if no duplicate was found.
The merging process takes one of three main avenues:
- Cases for which there cannot be a duplicate.
- Cases for which there is exactly one duplicate.
- Cases for which there are multiple duplicates.
For the vast majority of the Lawbox donation, we were able to simply add
the case to our collection without further ado. We determined this by
comparing the date and jurisdiction of the new opinion to our collection
and seeing if we had any cases from that jurisdiction during that time.
If there weren’t any cases from that place and time, bingo, the new case
couldn’t be a duplicate and we could add it straightaway.
For the opinions that might have duplicates, we developed a
duplicate-detecting algorithm. The process for this algorithm is as
follows.
1: Create a set of candidate documents that might be duplicates by
searching our existing corpus. First search it for cases in the same
jurisdiction within 15 days of when it was issued and which have the
same words in their name. Since names can vary greatly, the last word of
the plaintiff and the first word in the defendant are used as queries,
but only if those words:
- aren’t uppercase and less than three letters long (indicating an
abbreviation); - aren’t words that occurs very frequently (indicating a stop word);
- don’t contain punctuation or numbers (indicating something out of
the norm); and - aren’t less than two letters long (indicating they’re an
abbreviation).
Once this query returns, if it has results, we continue to step 2, but
if not, we try a new query using the docket number instead of the case
name. This often works, but if it fails we try one final time using the
citations. Unfortunately we can’t use the citations for all queries
because prior to this donation we did not have a good collection of
citation information.
2: Once we have some cases that our new one might be a duplicate of, we
attempt to match up the duplicates by docket number. This often works,
but if it doesn’t, we gather statistics about the items our new document
might be a duplicate of. Specifically, we gather:
- The edit
distance
between our new case and each of the candidates; - The edit distance between the text of our case and each of the
candidates; - The difference in length between our case and each of the
candidates; and - The cosine
similarity between
our case and each of the candidates.
Once that’s gathered, we set it aside and move on to step three.
3: At this point, we compare the case names to see if any of them are
good matches. We assume that if we have one candidate, if it has all the
same words in its case name as does our new document, and if they’re in
the right order, it must be a duplicate. So for example, Lissner v.
Carver is a duplicate of Michael Lissner v. Brian Carver, but not of
Carver v. Lissner (right words but wrong order).
4: If this approach fails, our next step is to attempt a similarity test
based on the docket number instead of the case name. This often works,
but when it doesn’t, we have another approach, using the statistics
generated above.
5: Our last approach is the statistical approach. Of the statistics
generated above, the cosine similarity is very accurate and the others
seem flawed in various ways. Cosine similarity takes all of the words in
each case, counts up how many times each one occurs, then plots all of
the words in a multi-dimensional vector space. Once we have a vector
space for each case, we determine the difference between the new case
and each candidate. If the two cases are very similar, they get a high
similarity rating. If not, they get a low one. In our experience a good
duplicate has a similarity of about 98%, and a dissent to the same case
has usually has a slightly lower similarity, generally around 97%.
Anything below 90% is unrelated. The extent to which this approach works
is remarkable, but it is slow and can lose accuracy if cases have
additional words, say, as part of the header information.
6: After all this is done, if we still haven’t determined if we have any
duplicates, our final approach is to send the new case and all its
candidates to a human reviewer, who looks at their contents and makes a
determination. Fortunately, this only happens rarely.
The Final Step
Once the meta data is extracted and any duplicates have been found, we
take the best parts of each document, merge them together and save it to
the index. Once this is complete the document shows up in search, in our
bulk files and everywhere else.
This approach took many month’s development, and it will receive another
round of polish the next time we add a batch of cases. Until then, we
hope this post has been educational, and that it can serve as a
reference for any data merging projects.
Source link
