[This article is authored by Aditya Mani Tripathi, a second-year B.A. LL.B. student at Lloyd Law College, Greater Noida. In this piece, the author examines the emerging challenges posed by Artificial Intelligence (AI) in the mental health sector, focusing on the risks arising from unregulated AI companions and the inadequacy of current Indian legal frameworks such as the IT Act, 2000 and the Mental Healthcare Act, 2017. The author argues that India must urgently develop a comprehensive regulatory framework that restricts AI systems from providing unverified or potentially harmful mental health advice, while ensuring human oversight, ethical compliance, and privacy safeguards in line with global best practices.]
Introduction: The Growing Role and Risks of AI in Mental Health
Artificial Intelligence (AI) has experienced significant and pervasive growth among the public, especially amongst the younger populations. More than 70% of teenagers today have used AI, and half continue to use AI companions, which definitely have their benefits but also pose a concern that needs to be addressed immediately and critically. This article advocates for the creation of regulatory frameworks to restrict AI companions from providing unsolicited and medically inaccurate mental health advice that can influence and potentially harm a person, amid the clear dichotomy between AI’s positive potential for mental health solutions and the risks arising from its unregulated use. This concern was tragically demonstrated through a recent documented instance of a tragic outcome wherein a 16-year-old teenager committed suicide, a mishap that was allegedly reinforced by an AI chatbot. The AI chatbot allegedly failed to prioritise and correctly implement suicide prevention methods; conversely, the AI chatbot allegedly offered advice to the teenager on how to move forward with his plan. In response to this, the owners of the said AI chatbot recognised the safeguard limitations in their product and showed commitment to assuage these limitations.
More and more people are now relying on AI as a companion for their emotional needs, even though the leading AI chatbots are inconsistent with their replies on suicide and hence pose a serious safety concern for people, especially the youngsters. Moreover, it is pertinent to note that there has been a massive surge of AI mental health chatbots in contemporary times. According to a study, AI-enabled mental health chatbots may offer dangerous responses to users; they may also fall short of human care and bring risks of reinforcing stigma with them. Thus, it is of critical importancethat a central regulatory body establishes clear guidelines that reasonably restrict the deployment of these AI products in the mental health sector in India to uphold essential safeguards like ethical compliance and data security, while also enabling modern solutions to mental health issues.
Current Legal Framework Governing AI and Mental Health in India
Section 43A of the Information Technology Act,2000 states that a body corporate will be liable for compensation if it is negligent in implementing and maintaining reasonable security practices regarding one’s sensitive personal data or information by a resource owned by the body corporate and thus causes any wrongful loss or gain to a person. In continuation, Rule 3 of the Information Technology (Reasonable security practices and procedures and sensitive personal data or information) Rules, 2011 clearly states that any information related to an individual’s mental health condition is a part of sensitive personal data or information. Hence, through parity of reasoning, organisations involved in the creation of AI services need to ensure that proper security practices are followed when faced with queries regarding an individual’s mental health. Moreover, Rule 8 of the Information Technology (Reasonable security practices and procedures and sensitive personal data or information) Rules, 2011 outlines reasonable security practices as a comprehensive document information security programme and information security policies that contain operational, technical, managerial and physical security control measures. Compliance with recognised standards like IS/ISO/IEC 27001, followed by regular audits is also outlined in the said legislation.
The Mental Healthcare Act,2017 (MHA), is the cornerstone for addressing the issues related to mental illness and its treatment. Section 2(r) of the MHA defines a ‘mental health professional’ wherein AI is not mentioned as a recognised mental health professional and thus it is correct to presume that AI should not be used for mental health-related issues by people, hence advocating towards reasonable restrictions on AI systems. Moreover, as of now, AI does not seem to possess the ability to make fair, equitable and compassionate decisions; ungoverned AI systems might also arrive at different conclusions for different individuals without the presence of any genuine rational basis, which further advocates for the creation of reasonable restrictions on AI systems. These restrictions can include preventing AI systems from providing unauthorised and unregulated clinical functions like diagnosis, counselling and therapy regarding an individual’s mental health without professional human supervision.
Section 20(2)(d) of the MHA firmly provides privacy rights to every person who is undergoing treatment for any mental illness. Various AI systems that engage in mental health issues have been found to violate these privacy guidelines and hence are in no capacity to offer unregulated mental health advice to the people. This further advocates for the creation of specific legislations that minimise unrestricted AI responses on an individual’s mental health.
Constitutional Perspectives on Privacy and Mental Health Rights
The Hon’ble Supreme Court of India in K.S. Puttaswamy (Privacy-9J.) v. Union of India (2017) 10 SCC 1, has clearly stated that the right to privacy is a fundamental right under the ambit of Article 21 of the Constitution of India. From the aforementioned, it has been aptly described how AI systems can induce violations of privacy laws. Any case of breach of privacy of an individual affected by mental health issues can aggravate the individual’s mental illness, increase the feeling of mistrust and consequently exacerbate mental health symptoms. In the context of professional and qualified mental health services, a major drawback of AI systems is that they may provide harmful or incorrect responses to vulnerable individuals. The behaviour of these AI chatbots can trigger alerts surrounding safety concerns; this, coupled with AI’s inability to provide pragmatic and suitable response in extreme scenarios and a lack of safe responses, poses a precarious threat to public safety; in a crucial study, it was found that AI chatbots failed to provide appropriate clinical solutions to the people in issues involving hallucinations, suicidal thoughts and delusions for about 20% of the time. Complete reliance on AI systems can lead to people underestimating the restrictions of these systems and overestimating their ability to provide proper treatment. This phenomenon is widely known as therapeutic misconception (TM), which can intensely exacerbate a person’s mental health.
The Hon’ble Supreme Court of India in C.S.E.C v. Subhas Chandra Bose (1992) 1 SCC 461 has asserted that health can be defined as a state of complete social, physical and mental well-being and is not merely recognised by the absence of disease. In Vincent Panikurlangara v. Union of India (1987) 2 SCC 165, the Hon’ble Supreme Court pronounced that in a welfare state, the obligation to create and ensure the creation and sustenance of conditions congenial to good health rests on the State; moreover, it was also stated that the right to maintenance and improvement of public health is included as a part of Article 21 of the Constitution of India. Thus, the detrimental effects of AI’s inability to disseminate proper mental health treatment and advice can gravely affect the health of the people, and hence the state is obligated to assuage this issue through implementing reasonable restrictions through creation of a comprehensive and independent legislation which should include qualified human oversight, enforce ethical guidelines, and defined liability guidelines for regulating the use of AI in healthcare and hence uphold the fundamental rights enshrined in the Constitution of India.
Global Regulatory Developments: Lessons from the EU and USA
Article 6(2) read with Annexure III of the EU AI ACT, 2024 has classified AI systems involved in healthcare to be in the list of ‘high-risk AI systems’, the requirements for which include human oversight, transparency of information to users, compliance with robustness, cybersecurity and accuracy guidelines, proper data governance, proper documentation and record keeping, along with risk and quality management systems. This act effectively recognises the need to properly regulate the exponentially growing AI systems and ensures reasonable restrictions to protect people’s rights.
The state of Illinois, which is a part of the United States of America, has recently passed legislation that is known as the Wellness and Oversight for Psychological Resources Act which prohibits AI systems from engaging in therapy and mental health assessment without proper supervision by a competent licensed professional. This legislation aims to ensure that AI is used in complementary work like scheduling appointments, analysing data, etc., but prevents AI from undertaking the duties of licensed medical professionals. This act demonstrates a clear legislative intent to rectify the gap that allowed AI systems to provide professional clinical advice and therapy with very little regard to the potential outcomes which could further exacerbate an individual’s mental health. This legislation piece acts as a working model regarding the reasonable restrictions that ought to be imposed on AI systems to restrict their role in the mental healthcare sector and to prevent any mishaps. Thus, it is pertinent to observe and derive from this legislation, in the context of Indian mental healthcare, to ensure the simultaneous development of a scientific temperament towards technological advancement while also protecting the rights of individuals who can be affected by the unregulated presence of AI in the mental healthcare sector.
Policy Recommendations for Regulating AI in Mental Health
A series of well-reasoned recommendations have been formulated to address the role of AI in mental health issues, ensuring appropriate oversight and safety which include: firstly, the creation of a regulatory body to ensure proper human oversight which includes psychiatrists, psychologists, and other competent professionals. This is required to ensure that the people suffering from any unfavourable scenarios do not receive detrimental advice. Secondly, mandatory proceduresshould be implemented for crisis detection and escalation that identify some habitual triggers indicating self-harm, aggravated mental health issues etc., and defer them to the concerned professionals promptly.
Thirdly, the creation of legislation to ensure stringent compliance with the data privacy and security laws like the Digital Personal Data Protection Act, 2023 for mental health-related scenarios especially. Fourthly, the establishment of clear liability guidelines followed by compliance towards various ethical concerns should be prioritised as determining liability will be crucial in case of any adverse or unprecedented scenarios.
Conclusion: Balancing Innovation with Protection in AI Mental Health Services
The world is undoubtedly going through a paradigm shift towards a newer, advanced and automated era. Artificial Intelligence, through its widespread application and utility, holds a critical role in this shift; but it is imperative that the role of artificial intelligence in segments like mental healthcare which hold a considerably sensitive place in society, is optimally regulated. The article has shed light on how unsolicited, inaccurate and medically unfeasible responses have been documented from artificial intelligence systems in contemporary times which can further lead to a magnanimous crisis among the vulnerable people who are influenced by this advice. The need for firm, clear and efficient laws regarding how artificial intelligence responds to sensitive questions related to mental health issues is of pivotal importance in contemporary times. This is further reinforced by the State’s obligation to protect and uphold critical fundamental rights issues. Hence, the need for the creation of reasonable restriction laws on artificial intelligence regarding sensitive issues like mental health is the need of the hour to ensure that innovations are hand-in-hand with new protective measures.