COLIEE 2022: working towards faster, more accessible legal systems with artificial intelligence

Could artificial intelligence help build a faster, more accessible legal system? In June, Amii Fellow Randy Goebel hosted an academic workshop in Japan to present the results from the Competition on Legal Information Extraction/Entailment (COLIEE) and show how researchers are advancing research into using AI to search through legal information.

This year, 25 teams from universities and companies around the world took part in the competition, designing artificial intelligence models that could do legal research. COLIEE began nearly a decade ago as a partnership between Goebel and colleague Ken Satoh, then based out of Japan's National Institute of Informatics. Goebel, who is a professor at the University of Alberta, says it felt like a natural extension of the work done at the university’s Explainable AI lab, which advances models that support explaining how artificial intelligence systems behave. AI assistance could make legal research more precise and help lawyers and other professionals be sure they have found all the information they need. And it could cut down on the sometimes lengthy delays that can cause legal matters to stretch on for years, he says.

"In Explainable AI, when it comes to health and law — those are two areas where the application of AI can actually have an impact on the rest of society. More precise, faster legal judgements are something we can have a measurable impact with," Goebel says.

Democratizing justice

Legal systems can be complicated and confusing, even for those with years of training. Lawyers, judges and other professionals spend countless hours reviewing documents, looking for relevant legislation and cases that might apply to whatever matter they are working on. By helping automate the most tedious and time-consuming parts of that task, Goebel is hopeful that AI assistance can allow legal professionals to focus their time elsewhere while potentially leading to faster and more precise judgements.

Using artificial intelligence to help search through statutes and cases can break down some barriers that keep the legal system inaccessible to the layperson. He says, for example, a machine-learning algorithm could potentially be built into a website, which would provide answers to simple, straightforward legal questions. While it would be no substitute for human expertise, it could be helpful to find answers that don't require meeting with a lawyer.

Cracking open case law

When it started in 2014, COLIEE gave teams two tasks to complete; competitors would design their programs and train them on a dataset made up of a broad sampling of civil statutes — the laws as written by Japanese legislators. The models are then tested with a yes-or-no question pulled from the notoriously-difficult Japanese bar exam. For instance, Goebel says, a question might pose a situation where a man pushes a woman out of the way of a speeding car but, in the process, ends up damaging the expensive kimono she is wearing. Is the man liable for the damage to the kimono? The first task involves the model finding all the relevant statues. And the second task asks the model to use those statutes to answer the question, liable or not liable.

In 2018, COLIEE added two new tasks covering different kinds of legal knowledge. Instead of Japanese statutes, the new challenges involved training data from Canadian case law. Competitors receive a copy of a judgement stripped of references to relevant cases and must identify the cases that the judge cited. It's no easy task: competitors are looking for a handful of cases among thousands of possibilities.

In the final challenge, teams get a case that relies on precedence from another judgement. Their model must go through the relevant case and find exactly which section supports the conclusion the judge made in the later decision.

The COLIEE competition tests different skills that an AI legal tool would need. Not only do models have to be able to effectively search through hundreds of thousands of pages of dense information, but they also must digest the legalese and determine its meaning. Finally, teams must find a way to create a model that can make connections between relevant cases and apply reasoning to what it finds.

This year's COLIEE competition saw teams from the University of Alberta, NeuralMind, Hokkaido University and Shizuoka University win one of the four tasks.

Learn more about how artificial intelligence and machine learning can be used to solve real-world problems. Check out our Industry Solutions page to see how Amii helps companies adopt AI breakthroughs.