AI is becoming an input into decisions across India’s public health system. Between April 2023 and November 2025, 28.2 crore consultations through eSanjeevani, India’s national telemedicine service, benefited from its AI-enabled Clinical Decision Support System. The National Health Authority (NHA) is also developing AI-assisted claims adjudication under the Ayushman Bharat Pradhan Mantri Jan Arogya Yojana (AB-PMJAY), with AI systems being tested to interpret claim documents, assess treatment-guideline compliance and detect fraud. AI-generated outputs can therefore enter both clinical decisions about care and administrative decisions about the use of public resources.
The governance question is not simply whether AI should be used in these settings. It is what role its output should be allowed to play in a consequential decision. A clinical recommendation may inform a doctor’s diagnosis or treatment; a fraud-risk assessment may trigger additional scrutiny of a legitimate claim or delay payment. The greater the consequence attached to an algorithmic output, the stronger the case for evidence that it performs reliably and for safeguards governing how that output is used.
When AI Enters Public Decisions
AB-PMJAY had authorised 12.69 crore hospital admissions worth ₹1.92 lakh crore by June 2026. Its anti-fraud systems use AI and machine learning to identify suspicious transactions and assign risk scores to hospitals and claims. In May 2026, NHA also showcased AI solutions for claims adjudication, extending AI’s role from identifying suspicious claims to evaluating them.
NHA’s anti-fraud framework recognises the possibility of false positives, where a legitimate claim or provider is incorrectly flagged as suspicious, and subjects such outcomes to further audit. Such a flag can subject a legitimate claim to additional scrutiny, with the potential to delay payment and impose costs on the hospital concerned.
But the weight given to an algorithmic assessment also depends on the evidence behind it. BODH, the government’s Benchmarking Open Data Platform for Health AI, provides a mechanism for testing and validating AI solutions using diverse, anonymised real-world health datasets before population-scale deployment. It is designed to assess performance, robustness and generalisability.
A 2026 study describing BODH reports preliminary evaluation of 12 AI models across five Indian hospital networks. It found that performance on BODH’s diverse datasets was lower than suggested by conventional, single-metric evaluation. The study estimated that vendor-reported evaluation overestimated model accuracy by 18–34 percent, while also identifying performance disparities across demographic groups that were not evident in those evaluations.
The implication goes beyond whether a model performs well in testing. Evidence generated in a development or validation environment may not establish how a system will perform across the populations and conditions in which it is deployed. The consequence of an error also depends on the authority given to its output: a recommendation that informs professional judgement carries a different risk from one that triggers scrutiny, delays payment or becomes the basis for a decision. Assurance must therefore address both model performance and the role its output plays in the decision process.
Making Assurance Follow the Consequence
India needs a risk-based assurance layer for consequential administrative and claims algorithms. The level of assurance should increase with the consequence an AI output is permitted to produce. A system that provides information, one that flags a case for scrutiny and one whose output can affect payment should not face identical requirements.
Higher-impact systems should require stronger evidence of performance, meaningful human oversight, auditability, avenues for review or appeal, and continuing monitoring. Human oversight should require that the responsible decision-maker retains authority to assess and depart from an algorithmic recommendation. Material changes to models should trigger reassessment.
This does not need to be built from scratch. The Central Drugs Standard Control Organisation (CDSCO)’s 2026 guidance under the Medical Devices Rules, 2017 provides a regulatory pathway for qualifying AI-enabled medical devices, including requirements around validation, risk analysis, clinical evidence and quality management. The Strategy for Artificial Intelligence in Healthcare for India (SAHI) provides a broader framework for safe, ethical, evidence-based and inclusive healthcare AI, while BODH provides a further component of the emerging assurance architecture.
The gap lies in algorithms whose outputs can influence administrative decisions without falling within the medical-device framework.
Government procurement can provide a practical lever for applying these requirements to such systems. Contracts can specify intended use, the decisions an AI output may inform, performance standards, human oversight, reporting, reassessment after material changes and conditions for withdrawal. This would make assurance requirements part of the conditions under which public authorities adopt and continue to use these systems.
India’s next step is to make assurance follow the consequences an AI system can carry.
Leave a Reply