Artificial Intelligence (AI) is rapidly transforming the legal profession, from document review to drafting contracts and conducting legal research. It offers speed, efficiency and a competitive edge for lawyers. However, the recent misuse of AI in the courtroom has raised significant ethical and procedural concerns.
While many are excited about what AI can do, it is crucial to exercise caution when using such tools. AI tools, especially generative models, do not always differentiate between fact and fiction. This becomes a serious issue when incorrect outputs are accepted at face value.
Recent High Court Revelation
In a matter raising an important issue concerning the integrity of the judicial decision-making process in the era of Artificial Intelligence (AI), a Division Bench of Pamidighantam Sri Narasimha and Alok Aradhe, JJ., took cognisance of an Andhra Pradesh trial court order that relied on four fabricated judgments and issued notice to the Attorney General for India, the Solicitor General of India and the Bar Council of India to examine the systemic implications of reliance on AI-generated, non-existent or synthetic precedents.
The instant dispute originated from a civil suit filed by the respondents seeking an injunction in respect of certain property. During the pendency of the suit, the trial court appointed an Advocate Commissioner to note the physical features of the property. After the Commissioner submitted the report, the petitioners (defendants in the suit) raised objections to it.
The Andhra Pradesh case does not stand alone; rather, it represents the latest in a series of AI-related judicial failures.
In October 2025, the Bombay High Court quashed a ₹27.91 crore income tax assessment after discovering that the National Faceless Assessment Centre had relied on three non-existent judicial decisions. The Court observed: “In this era of Artificial Intelligence, there is a tendency to place undue reliance on system-generated results. However, when exercising quasi-judicial functions, such results must not be relied upon blindly.”
Moreover, in January 2025, a trial court judge in Karnataka cited incorrect precedents generated by ChatGPT, with both parties stating they had never cited those cases. Meanwhile, a Punjab and Haryana High Court judge publicly admitted to using ChatGPT to research bail jurisprudence.
Early Cases and International Spread
The problem first came to prominence in the US case of Mata v Avianca 22-cv-1461 (PKC) SDNY 2023, which was widely discussed among legal professionals. Approximately six months later, the UK’s First-tier Tribunal (Tax Chamber) encountered a similar incident in Harber v HMRC [2023] UKFTT 1007 (TC), where a litigant provided nine fictitious AI-generated decisions.
The tribunal endorsed Judge Castel’s warnings from Mata, emphasising the “many harms that flow” from AI hallucinations, including wasted resources, damage to judicial reputation, the undermining of authentic precedents, and increased cynicism towards the legal system.
Since then, international incidents involving AI hallucinations in court proceedings have increasingly surfaced on blogs and social media, attracting significant attention from legal professionals observing their rapid spread across jurisdictions.
It was against this backdrop that the High Court in Ayinde emphasised the need for lawyers to clearly understand their responsibilities in the evolving context of AI. While the facts of Ayinde are well known and need not be repeated here, the Court specifically considered suspected instances of generative AI producing fictitious case law, fabricated citations and inaccurate statements of law.
The Court acknowledged AI’s powerful benefits in litigation (both criminal and civil) but stressed the significant associated risks. The administration of justice, the Court explained, depends absolutely on the integrity of practitioners. It emphasised lawyers’ duty to verify all materials, equivalent to checking research conducted by junior colleagues.
How AI-fabricated Legal Citations Occur
This troubling phenomenon stems from a technical flaw known as “hallucination” in AI. In simple terms, hallucination refers to the generation of plausible-sounding but incorrect or imaginary information by AI models.
These hallucinations occur because generative AI, like ChatGPT, predicts words based on patterns rather than factual accuracy. If asked to provide a citation supporting a legal argument, it may invent one that appears legitimate, even referencing fabricated case names, judges and court dates.
While the Supreme Court has declined to issue directions on AI in a recent public interest litigation, it has recognised the dangers of its use within the judicial system. The White Paper on AI and the Judiciary (“White Paper”) published by the Supreme Court sheds light on the approach adopted towards AI in India’s judicial system.
The White Paper identifies artificial intelligence as a crucial tool for addressing India’s judicial backlog of over five crore cases. It highlights AI’s role in improving case management, legal research and transparency. At the same time, it emphasises that AI is intended to support, not replace, human judgment.
The Supreme Court has introduced indigenous tools, including the Supreme Court Portal for Assistance in Court’s Efficiency (SUPACE) for analysing case records, the Supreme Court Vidhik Anuvaad Software (SUVAS) for translating judgments into 19 languages, and Technology Enabled Resolution (TERES) for real-time transcription.
The Court warns that premature AI adoption could compromise judicial integrity. Risks include inaccurate outputs, hallucinations (such as false citations) and algorithmic bias that may perpetuate social hierarchies. The White Paper outlines potential AI applications in courts and provides guidelines for responsible use tailored to judges, lawyers and clerks. Core principles include mandatory human verification and strict adherence to confidentiality.
A landmark step by the judiciary came through the AI policy issued by the Kerala High Court in July 2025. The policy, titled “Policy on the Use of Artificial Intelligence Tools in the District Judiciary”, seeks to ensure the responsible use of AI as an assistive tool rather than a substitute for legal reasoning. Its scope is not limited to generative AI but extends to all AI systems, including databases providing access to case law and statutes.
Impact on Legal Integrity and Justice
The use of false citations goes far beyond mere embarrassment. It threatens the sanctity of legal arguments, delays proceedings and may potentially damage client interests. Worse still, it erodes public trust in the legal system.
When legal professionals rely on tools that introduce misinformation, it places an undue burden on judges and opposing counsel to identify and correct these errors. In a high-stakes environment such as litigation, even minor lapses can have serious ramifications.
Professional bodies such as the Solicitors Regulation Authority (SRA) and the Bar Council are taking notice. There is growing discussion about updating codes of conduct to explicitly address the use of AI in legal practice. Ethically, lawyers have a duty to maintain client trust and uphold justice; misuse of AI, even unintentionally, may breach this duty.
The question of accountability remains complex. Is it the AI developer’s fault or the lawyer’s? Most experts agree that the onus remains on the legal professional. AI does not relieve a solicitor or barrister of their ethical obligations. Proper use of technology requires vigilance, training and professional judgment.
Future of AI in Legal Practice
Despite these challenges, AI continues to offer significant potential for positive transformation within the legal profession. With proper use and thoughtful oversight, it can reduce administrative burdens, improve access to justice and streamline legal research.
Persistent issues such as backlog, language barriers and unequal access to legal information may be alleviated through AI tools. However, it cannot be ignored that reliance on opaque algorithms in judicial processes may reduce accountability.
For now, India’s judiciary appears determined to keep humans firmly in control, treating AI as an assistant rather than an oracle. How long this balance endures will determine not only the efficiency of justice delivery but also the level of public trust in the system. The legal fraternity must proceed cautiously to ensure that technology supports, rather than undermines, sound legal practice.
(Dr S Krishnan is an associate professor in Seedling School of Law and Governance, Jaipur National University. Tanisha Choudhary and Shubham Choudhary are law students at the same university)
