These days, there is ample evidence of the numbers underlying that discomfort. AI models that predict appellate court outcomes with an accuracy of roughly 70 to 71% have been developed recently; some models have achieved an accuracy of 70.2% across nearly two centuries of U.S. Supreme Court decisions. Attorneys and legal scholars have taken notice of SCOTUSbot, an unofficial tool designed to predict the court’s decisions, and are attempting to determine what to do with this capability. AI was able to accurately predict 88% of prosecution decisions and 82% of asylum outcomes using American legal data, according to a 2018 study by Professor Elliott Ash of the University of Warwick. At the time, those figures were startling. Since then, they have only climbed.

Seventy-one percent seems like a lot. Indeed, it is. However, it’s what the accuracy suggests rather than the accuracy itself that is unsettling. An AI must have discovered recurring patterns in judges’ reasoning and decision-making in order to forecast court outcomes at that rate. patterns that are trustworthy enough to model. patterns that are consistent enough for a machine to pick them up, draw conclusions from them, and use them in the future. The unsettling interpretation of that figure is that, at least at the appellate level, judicial decision-making may be more influenced by past conduct than most people, including the judges themselves, would be willing to acknowledge.

All of this has a bias issue, which researchers are generally honest about. Case data from the past is used to train these AI systems. Human biases can be found in historical rulings on bail, sentencing, immigration, and property; some are evident, some are systemic, and some operate covertly over decades. A model is learning more than just legal reasoning when it gains the ability to forecast results from that data. It’s also about understanding the prejudice and error patterns that influenced those results. Any automated decision system trained on biased data will likewise be biased, as Professor Ash stated clearly in his seminal work. In other words, what is anticipated isn’t always what justice looks like in theory. Because of its unique history, this system has a tendency to produce it.

It’s instructive to see how the legal system is handling all of this. There is a discernible drift but no consensus. AI was the focus of Chief Justice John Roberts’ entire 2023 year-end report. During last week’s oral arguments, Justice Samuel Alito stopped the proceedings to ask a well-known proponent of artificial intelligence if the court should just ask Claude to make the decision. The attorney politely declined. However, if it was a joke, it revealed a truth. The notion is no longer ridiculous enough to be considered wholly ludicrous.

It’s important to focus on what Sotomayor actually said rather than just the headline figure because her remarks at Alabama were among her longest public statements on the topic. AI was not rejected by her. She tells students that all new associates at the large law firms where her former clerks now work are expected to use AI, and she encourages the class to become proficient in it before graduating. She uses it, or at least acknowledges its existence. Her worry was more focused and, in a sense, more intriguing. AI, according to her, is “a sophisticated human” that receives all of its inputs from people and carries “the very best in us and the very worst in us.” Although it may appear to be a dystopian warning, that framing is not. This observation relates to inheritance. What we create is a reflection of who we are.

It’s still unclear if a 71% prediction rate is a sign of judicial rigidity, a victory for machine learning, or both. There seems to be a good argument for reading it both ways. Judges are either applying principled consistency, which is arguably what law should look like, or they are acting automatically in ways that prevent the sincere reconsideration that complex cases occasionally require if the patterns are real and teachable. You can’t tell which of those two possibilities is true just by looking at the number; they feel so different. That’s probably the most truthful way to describe it. A statistic is not self-explanatory. It simply remains at 71%, posing a query that the legal system is still unsure how to respond to.

The post A New Study Found That AI Predicts Appellate Court Outcomes With 71% Accuracy. That Is Terrifying appeared first on Creative Learning Guild.