[Sara Singh is a third-year student at Dr Ram Manohar Lohiya National Law University, Lucknow. In this piece, the author examines the legal and professional implications of using AI in court submissions, analysing recent Supreme Court precedents and the judiciary’s shift from treating unverified AI hallucinations as mere negligence to treating them as professional misconduct. The piece argues that while the duty of legal professionals to verify their submissions remains non-negotiable, true accountability requires a multi-stakeholder framework that also extends liability to AI developers and regulatory bodies.]
Introduction
Imagine arguing your case before a Supreme Court bench, and you confidently refer to a landmark decision that supports your client’s position. At first, the judges seem confused and then stern. After a few minutes of searching their databases, they say, “The case you are citing doesn’t even exist.” It’s a “hallucination,” a fake citation created by the Generative-AI you used to prepare your arguments.
A similar instance occurred on February 17, 2026, during a PIL hearing in the SC: a bench led by CJI Surya Kant came across a citation titled ‘Mercy v. Mankind,’ a case that does not exists in the annals of Indian Jurisprudence, as Justice B.V. Nagarathna herself pointed out, while the CJI noted that a “series of such judgments” had similarly been cited before. At this point, who is to blame for this professional crisis? As more and more legal professionals, we face a growing “liability gap” in which technology provides significant efficiency gains, yet sometimes fails to meet the standard of accuracy and verification required in the legal domain.
This piece argues that while the use of AI in legal professions has been inevitable, and yes, the advocates bear a non-delegable duty to verify their work before presenting it in court, this duty may not be the end of the debate. It further suggests that the chain of responsibility should extend to developers who design the tools and regulatory bodies (such as the Ministry of Electronics and Information Technology (MeitY) and the Bar Council of India), which shape both the technical standards and the professional environment in which these tools are used.
The paper, therefore, presents a multi-stakeholder approach to close the gap. Since the judiciary has shifted in treating AI mistakes from merely an error to professional misconduct, and while this is necessary, a multi-stakeholder framework would hold advocates, developers and regulatory bodies to account at the same time, rather than substituting for the other.
Current Judicial Stance: From Error to Misconduct
Further, B.V. Nagarathna had noted in the oral observations in the wake of the Mercy v. Mankind incident that, even when real judgments are cited, AI often attributes “fake quotes” to them, which places a burden on the judges to act as fact-checkers. Additionally, beyond mere inaccuracies, the Court has expressed concern over ‘synthetic’ judgments and a resulting ‘decline in the art of drafting,’ where unverified AI-generated blocks of text replace original legal articulation. This has made the judiciary shift from curiosity to concern, further adding that this wastes judicial time because judges must verify incorrect or fake citations.
The most significant shift, however, occurred in February 2026. In the Gummadi Usha Rani Case, the Supreme Court observed that citing AI-generated fake precedents is a professional misconduct and not just an ‘error in decision-making’. [Gummadi Usha Rani v. Sure Mallikarjuna Rao, 2026 SCC OnLine SC 341, ¶ 7 (P.S. Narasimha and Alok Aradhe, JJ).] Here, by moving the act from negligence to misconduct, the Court has signalled that the ultimate responsibility for verifying whatever is presented and cited before the court remains non-delegable and rests with the advocate.
Since the legal frameworks governing the use of AI are still evolving, and the Bar Council of India’s (BCI) Standards of Professional Conduct and Etiquette [Part VI, Chapter II] were framed long before the emergence of large language models, they have been silent on the use of generative AI. The SC has tried to address this issue recently in Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd., where the court made strong remarks about the NCLT Bench, which passed an order relying on an AI-generated hallucinated judgment. [Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., (2026) SCC OnLine SC 668, ¶¶ 6–7 (P.S. Narasimha and Alok Aradhe, JJ).] The court has outlined a draft for “Regulations for Use of Artificial Intelligence in Courts, 2026”, which includes many salient aspects of how AI would be governed, such as mandating acceptable/prohibited uses, transparency, proportional use with human oversight, a dedicated regulatory body, rigorous auditing/training protocols, and stringent safeguards against AI hallucinations.
However, the draft regulations themselves leave a gap that matters for this piece’s argument. They have imposed a human-in-loop requirement for significant AI actions, regulating how courts use AI institutionally, but they do not regulate the extent or responsibility of advocates who are using the tools. Moreover, judges who relied on these AI-hallucinated cases and cited judgments fall outside the draft’s prohibited-use categories, leaving no clear disciplinary route, as in Pooja Ramesh Singh v. J&K Bank Ltd., where it was the NCLT, not the counsel, that relied on the hallucination.
The Dichotomy of Accountability: Whom to Blame?
There could be two intuitive models for allocating responsibility in human–AI collaboration.
Firstly, the Gatekeeper model: Under this model, professionals bear ultimate responsibility as decision-makers. While the data shows that when advocates and firms use AI daily or weekly, they experience significant benefits, most notably improved efficiency, but this efficiency does not absolve them of their duty of candour to the Court.
As seen recently, the Bombay High Court penalised an advocate for filing AI-generated content without verification. The issue was that a litigant filed written submissions generated through AI that cited a non-existent judgment. The court could not find the case in any legal database and imposed a fine of Rs. 50,000. The main reasons are professional responsibility and the accuracy of legal submissions. While AI is welcome as a research aid, it can produce ‘hallucinations’ such as fake case law or non-existent quotations. Therefore, the ultimate responsibility for verifying these remains with the advocate.
Secondly, the joint responsibility model, which this piece argues is not a replacement for Gatekeeper but a necessary supplement. This model ensures that the liability is shared across the chain, from the developers who design the guardrails to the institutions that integrate them. For example, if a professional-grade legal AI is marketed as a reliable tool for advocates but fails to “ground” its answers in actual law, the developer has arguably failed in its duty of care.
It is pertinent to note that if a professional AI tool is making mistakes like hallucination of cases or citing fake case laws, then that could be classified as “deficiency of services” under the ambit of Section 2(11) of the Consumer Protection Act, 2019. The liability lies not just with the legal professionals but also with the AI tool because it has violated its statutory duty to process and respond to the information correctly.
This reveals that we don’t lag in respect of the legal framework but in respect of its implementation. Additionally, until the joint responsibility model is invoked, we are bound to abide by the gatekeeping model. However, the notion behind this is not that it is better or universally accepted; rather, we don’t have a better alternative, or this is the only way anyone has used to date.
Beyond India: The Same Gap, Everywhere
India is not the only one facing this challenge. The Mata V. Avianca case in the US was the first to bring global attention to this issue. A New York attorney had used ChatGPT to research a personal injury case. ChatGPT ended up providing court decisions that neither the airline’s lawyer nor the judges themselves could find, and ultimately, sanctions were imposed on it.
Additionally, the California appellate court imposed USD 10,000 in sanctions against an attorney who submitted AI-generated case citations in September, 2025 (Presiding Justice Lee Smalley authored the opinion in Noland v. Land of the Free, L.P.). This became the state’s first published ruling on this issue. Then there’s Singapore’s Supreme Court, which has permitted AI for court filings with a condition that court users should personally verify all the output. The lawyers will take full responsibility for its accuracy. In July 2025, the UK published its AI Action Plan for Justice, which embodies principles of transparency and governance for the use of AI across its courts and tribunals. These instances around the world are not mere facts but reveal a pattern of such incidents and raise the question of accountability, which, in these cases, rests with the advocates.
China (Hangzhou Internet Court), being an exception in 2025, clarified that liability for AI errors is based on a fault-based standard, and thereby it recognises AI as a service rather than a product. But the developer was cleared, as the plaintiff could not show the required fault or actual harm, implying that even if one system asks a developer-liability question, the answer so far has been no. Hence, every jurisdiction surveyed here has held that the accountability behind the submissions should rest with the professionals and has been silent on the developer’s question, which is the global version of the same liability gap that exists in the Indian Context.
Bridging the Liability Gap: A Multi-Stakeholder Approach
Today, the argument is not against the use of AI in the legal profession, but it is important to understand that AI is just a tool which should act like a secondary reviewer, and not a primary one. And therefore, a multi-stakeholder framework needs each stakeholder to share the responsibility and run in parallel rather than substituting for each other.
For advocates, the duty to verify what they are presenting before the court is non-negotiable, and they must act as a primary reviewer. For developers, regulators need to treat “deficiency in service” under CPA, 2019, as a remedy rather than a theory. Moving forward with the help of the IT Amendment Rules, 2026, the government is regulating deepfakes and AI-generated content. The Ministry of Electronics and Information Technology (Meity) has introduced the India AI Governance Guidelines, which adopt an “innovation over restraint” philosophy. The said philosophy aims at updating the existing IT Act rather than strictly imposing restrictions.
For courts, their duty is to maintain what was outlined in Pooja Ramesh Singh and Gummadi Usha Rani, i.e., to monitor usage and to issue guidelines where necessary. The Kerala High Court’s recently published policy is a major initiative concerning the use of AI in the district courts. It is the first formally documented set of guidelines regarding the use of AI tools in a court of law. This has explicitly provided established protocols and also given a blueprint for the other courts to follow. From these, it is evident that these three absorb the other two. BCI issuing guidelines would not make Section 2(11) self-executing, and these claims against the developer would not absolve the advocates of their obligation to verify what they submit.
Conclusion
The recent case of Mercy v. Mankind was not just an anomaly; rather, it must be taken as a warning by legal professionals. On the question of accountability, the court has classified it as misconduct rather than negligence. Around the world, from California’s sanctions to Singapore’s verification mandate, accountability for AI-assisted mistakes rests with the advocate. In India, that is necessary but may not be sufficient. What is missing is the three obligations set out above, operating together rather than one absorbing the other two: the Bar Council issuing a specific verification obligation for advocates, invoking Section 2(11) against developers, and courts continuing to hold the line on professional misconduct. Until all three are doing their part, the advocate’s non-delegable duty will keep doing the work of all three, which is exactly the liability gap this piece set out to name.