COLIEE 2019: Applying AI to legal reasoning

Amii is proud to once again sponsor the Competition on Legal Information Extraction and Entailment (COLIEE). By building a community of practice regarding legal information processing and textual entailment, COLIEE has been able to steadily advance the understanding of the process of applying AI to legal reasoning. Now in its sixth year, COLIEE takes place on June 21, 2019 in Montreal, and is run in association with the International Conference on Artificial Intelligence and Law (ICAIL) 2019.

Origins of COLIEE

How many competitions have begun in response to a failed bar exam? At least one. COLIEE was founded by long-time colleagues Randy Goebel (Amii Fellow and Professor at the University of Alberta), and his colleague Ken Satoh (Professor at the National Institute for Informatics).

“Ken and I had always worked on all kinds of applications of AI to reasoning in any domain. Legal reasoning was an early target for AI […] because it’s a domain in which there’s an attempt to be precise about how you write things in natural language,” says Goebel. “And Ken was so keen on this, that nine years ago […] he got very frustrated with trying to take the next step in applying AI to legal reasoning, so he took a law degree.”

Satoh completed a law degree at the University of Tokyo, while at the same time maintaining his position as a full-time professor at one of the most prestigious computer science departments in Japan. Once completed, he took the Japanese bar exams and failed – five times.

“COLIEE was born out of his frustration at the way the bar exams were administered,” explains Goebel. “He and I decided we could stage a competition. We could use Japanese bar law exams as examples with statute law in Japan and mount a competition to see how people could use AI to answer bar law exam questions.”

Since the competition was first staged, Satoh has passed the Japanese bar exams. Meanwhile, Goebel has accomplished the same, albeit with AI; he led a team which designed a program that passed the exams in 2017. But COLIEE still happens every year, growing in challenge and ambition. This year, Goebel and Satoh are just two of six coordinators, and a total of 18 teams are competing from 13 different countries, from the US to Botswana; Argentina to Germany.

The Tasks at Hand

Since its inception, the challenges have steadily increased in complexity; now, the competition includes two categories classified by legal concepts in statute law and case law. Those two categories each have two tasks: an information retrieval task and an entailment task. Teams have the option of submitting for one task, all four tasks, or any number in between.

The information retrieval tasks challenge a program to take a given test case and retrieve the related statutes or cases. The entailment tasks take it a step further.

“Entailment is really just saying: if this is a statute and this is a question, does the statute entail the question or not? [Editor’s note: the Dictionary.com definition of entail is “to cause or involve by necessity or as a consequence: e.g. a loss entailing no regret”] It’s the legal reasoning required to say, if you are wearing an expensive kimono and a bystander pushes you out of the way of a car that’s about the hit you […] is he liable for damages to the kimono?” says Goebel. “So we want to build computer programs that […] retrieve the appropriate statutes, then they have to find a connection.”

In other words, an entailment task requires a program to perform information retrieval, then create a yes/no argument for the test case based on the returned statutes or cases. In the example above, a program would be required to retrieve statutes related to property damage and liability, then determine whether or not the bystander would be liable for the damaged kimono.

The structure of the competition has accelerated research in this area in a unique way. COLIEE has built a community of people who are tackling similar challenges, and as a result, its participants have naturally developed a vocabulary that has quickened the process of exchanging information and ideas.

“It’s almost to that point at the COLIEE competition workshops […] people talk without even specifying. They say, ‘and here’s our approach to task one’. They don’t describe task one anymore.”

Explaining explainability

Explainability – the ability to determine how a model arrived at an answer – plays a large role in the entailment portion of the competition. Goebel’s lab, the Explainable AI (XAI) Lab out of the University of Alberta, is dedicated to this exact concept. Goebel’s lab is competing in COLIEE, and in this aspect, he has an advantage.

“Our lab has been tackling these two areas of medical reasoning and legal reasoning; we want to drive the science of AI forward. So we have an advantage to all of the people on the planet who, in an ad hoc way, apply learning of any kind to these domains, because we’ve been doing them for longer, and our focus is on making them explainable.”

Many machine learning models are challenged by the inability to explain how they arrive at decisions. For example, a model can take points of data and group them together based on a similarity it noticed, but it will not be able to articulate the similarity. Many refer to this issue in metaphor, calling them “black box” models.

Black box machine learning models can perform brilliant tasks – but in application, they can have serious consequences. If a model is assisting a doctor to recommend treatment, a judge to recommend sentencing, or a hiring manager to choose resumes, it is important to be able to determine why it is arriving at decisions.

“Explainability is as simple as saying: when I tell you something, please explain,” says Goebel. “The background that I have comes from formal philosophy and building systems to create hypotheses about data. That’s what scientists do. In there lies all of the mechanisms you need to do explanation.”

The 6th Competition on Legal Information Extraction and Entailment (COLIEE) takes place on June 21, 2019, and is run in association with ICAIL in Montreal, Quebec. Visit https://sites.ualberta.ca/~rabelo/COLIEE2019/ for more information.