In its landmark ruling on AI-generated fake legal precedents by the National Company Law Tribunal (NCLT) on the legal system, the Supreme Court on Thursday severely criticized artificial intelligence-based artificial intelligence and warned that human control over judicial decision making must remain “total and absolute” at every stage of adjudication.
The court has also set down an NCLT order that allowed Essel Infraprojects to be incorporated in the corporate insolvency resolution process after finding that the tribunal had cited non-existent and “hallucinated” judgments made by artificial intelligence as legal precedents.
A Bench comprised Justices P.S. Narasimha and Alok Aradhe expressed concern over the incident as they consider it a danger to the integrity of the justice delivery system.
"This is yet again a case where the Tribunal relied on non-existent, fake and hallucinated material, generated through Artificial Intelligence (AI), as if it were a precedent in support of its judgment," the court observed.
In one of the strongest remarks on AI's misuse in judicial proceedings, the Bench compared the spread of AI-generated fake legal material to the release of methyl isocyanate, the toxic gas responsible for the Bhopal gas tragedy.
"This byproduct... is like the release of methyl isocyanate in the province of law and justice. Invisible, insidious, and catastrophic. It doesn’t just contaminate but takes away the very lifeline of judicial determination; it’s by the time anyone notices,” the judgment said.
And the Supreme Court made it clear that while artificial intelligence is helpful to judges and lawyers in research and administrative tasks, it cannot replace human reasoning or judicial discretion.
The Bench emphasized that courts that deploy AI technologies must ensure that a “human in the loop” remains responsible for every part of the decision-making process.
"What is significant for our decision making is our resolve to adopt artificial intelligence technology in the aid of adjudication, while asserting and declaring total and absolute control over adjudications to the human in the loop, at every stage," the judgment said.
The court said AI is increasingly capable of performing both routine and intellectual tasks and that the workload is increasing and professionals will adopt technology to be more efficient.
As a prime example of AI's rapid development, the Bench referred to Garfield Law Limited (GLA), which was the first AI-driven law firm in the United Kingdom to receive regulatory approval for providing certain legal services.
But the judges said that too much reliance on AI could lead to a weakening in independent legal reasoning.
"The result of AI can be gratifying, even inspiring; but if left unregulated, Artificial Intelligence may invade our intellectual work ethic and before long, make us dependent on its vast capabilities," the court observed.
The Bench also distinguished AI from previous technological innovations adopted by the judiciary, stating that AI could, unlike conventional digital tools, affect thinking, reasoning and even decision-making itself.
Recognising the implications for the legal profession, the Supreme Court directed the Bar Council of India (BCI) to form a committee to investigate the issues in detail and to develop guidelines for the proper use of artificial intelligence.
The case stemmed from the suspension of the suspended director Pooja Ramesh Singh on appeal against NCLT order and later on appeal against the National Company Law Appellate Tribunal (NCLAT) which had accepted Essel Infraprojects Ltd. into insolvency proceedings under section 7 of the Insolvency and Bankruptcy Code on a petition filed by Jammu and Kashmir Bank Ltd.
The insolvency proceedings arise from a default of ₹200 crore credit facility which was secured by a corporate guarantee from Essel Infraprojects and a mortgage over land in Gorai, Borivali, Mumbai. The NCLT had admitted the insolvency petition on the basis of the alleged debt of ₹87.43 crore.
But in reviewing the case, the Supreme Court found that the tribunal had used fictitious judicial precedents generated by AI rather than genuine legal authorities. Based on the fact that reliance on such precedents undermined the credibility and fairness of judicial proceedings, the court set aside both NCLT and NCLAT orders and told the tribunals to reconsider the matter based solely on verified facts, evidence and applicable law.
It is expected to set a great precedent for the Indian judiciary’s growing use of artificial intelligence and its need to keep the law and justice at the heart of AI, and human judges will still be able to do so.
