Google’s AI wake-up call: Why healthcare governance must lead innovation

Examining AI risk, regulation and real-world harm to understand why governance, not technology, determines whether healthcare innovation is safe

“Dangerous and alarming.”

That was how The Guardian described the health advice generated by Google’s AI Overviews, after multiple responses were withdrawn for being inaccurate and potentially harmful. Users searching for explanations of blood test results were presented with misleading information that, in a healthcare context, could influence clinical decisions, delay care, or cause harm.

This was not a minor error. It was a patient safety risk.

If one of the world’s most sophisticated technology companies, Google, with vast resources, elite engineering capability, and deep experience in machine learning, cannot reliably prevent unsafe health advice from being generated, it forces a more uncomfortable question: what does this mean for smaller providers, startups, and care organisations adopting AI tools without robust governance, validation, and accountability frameworks in place?

This was not simply a technical glitch. It was a governance failure. And it follows a pattern healthcare leaders have seen before.

In this blog, Dr Richard Dune examines how the Google AI Overviews incident exposes wider governance failures and the risks of deploying AI in healthcare without proper oversight.

A familiar pattern: Deployment before safeguards

The Google AI Overviews incident is not occurring in isolation. It echoes earlier controversies where innovation advanced faster than governance.

In 2017, DeepMind (owned by Google) was granted access to 1.6 million NHS patient records at the Royal Free London NHS Foundation Trust to develop a diagnostic app. Patients were not explicitly informed. The UK Information Commissioner later concluded that the data sharing lacked a proper legal basis and failed transparency requirements.

At the time, the debate focused on consent, data protection, and lawful processing. Nearly a decade later, the same underlying issue has re-emerged in a different form. This time, the concern is accuracy, reliability, and direct clinical risk.

The common thread is not malicious intent. It is the repeated deployment of AI into health contexts before governance, oversight, and accountability mechanisms are fully established.

The scale of the risk

This concern extends far beyond a single product or company.

Research consistently shows that healthcare leaders recognise AI risk, but often adopt tools faster than their governance structures can mature. Around 72% of healthcare executives cite data privacy as the leading risk in AI adoption, while nearly 69% worry that AI will exacerbate data security and confidentiality challenges.

Healthcare also remains the most expensive sector globally for data breaches, with average costs exceeding £7 million per incident. Meanwhile, public trust is fragile. Surveys suggest that only about 41% of people in the United States trust AI, below the global average.

These figures reflect lived organisational experience. They point to systems being introduced before organisations are ready to govern them safely.

Why this matters now

This debate is no longer theoretical. In the UK, the Medicines and Healthcare products Regulatory Agency (MHRA) is actively gathering evidence to inform future regulation of AI in healthcare. Incidents like Google’s AI Overviews strengthen the argument for clear, enforceable frameworks that prioritise patient safety before widespread deployment.

Internationally, the World Health Organization has set out six core principles for AI in health:

• Human rights
• Safety
• Transparency
• Accountability
• Equity
• Sustainability. 

In the United States, the Coalition for Health AI and The Joint Commission have developed the RUAIH framework to guide the responsible use of AI in health systems, particularly at the bedside.

The frameworks exist. The challenge is not a lack of guidance. It is whether organisations adopt these principles proactively, or wait until harm, litigation, or regulatory intervention forces their hand.

AI governance is a leadership issue, not an IT task

One of the most persistent misconceptions about AI in healthcare is that governance sits primarily with IT teams. It does not.

AI governance is fundamentally a board-level and executive responsibility. It intersects with patient safety, clinical risk, data protection, professional accountability, and organisational reputation. Treating AI adoption as a procurement decision or technical upgrade is a serious category error.

For providers, boards, and senior leaders, several fundamentals must be in place before deploying any AI system in health or social care.

Human oversight must be explicit, not implied

AI can support clinical judgement, but it cannot replace it.

Healthcare organisations must define, in writing and in practice, where human authority sits. Clinicians and professionals need clear guidance on when AI outputs may be used, when they must be questioned, and when they must be ignored.

Without explicit oversight models, responsibility becomes blurred, particularly when time pressure, staffing shortages, or automation bias encourage over-reliance on AI outputs.

Accountability must be resolved before deployment

When an AI system generates unsafe advice, who is responsible? Is it the developer? Is it the organisation deploying the tool? Is it the clinician relying on its output?

These questions cannot be answered after harm has occurred. They must be resolved contractually, operationally, and ethically before deployment. Without clear accountability, AI introduces risk that no one truly owns, until something goes wrong.

Validation must occur in real clinical workflows

Performance in controlled testing environments does not guarantee safety in practice.

AI systems must be validated in the real-world contexts where they will be used: busy wards, overstretched primary care settings, community services, and social care environments. This includes testing how tools perform under pressure, with incomplete data, and across diverse patient populations.

Clinical safety cases should not be static documents. They must be living artefacts, reviewed as systems evolve.

Transparency is a safety requirement, not a courtesy

Clinicians and patients deserve clarity about what AI systems can and cannot do. This includes transparency about known limitations, confidence thresholds, bias risks, and failure modes. Black-box decision support undermines professional judgement and erodes trust, particularly when outputs appear authoritative but are poorly understood.

Transparency is not about overwhelming users with technical detail. It is about enabling informed, safe use.

Continuous monitoring is essential

Unlike traditional software, AI systems can degrade over time. Changes in data patterns, population health trends, or clinical practice can lead to algorithmic drift, where performance worsens without obvious warning. Ongoing monitoring, audit, and re-validation are therefore essential components of safe deployment.

AI assurance is not a one-off exercise. It is a continuous governance process.

The path forward: Sequencing innovation properly

None of this is an argument against innovation. AI has genuine potential to support diagnostics, workforce planning, population health management, and operational efficiency. But innovation in healthcare only earns trust when it demonstrably serves patients, professionals, and systems safely.

Google had every conceivable advantage and still failed to prevent unsafe health advice from being generated. That should act as a warning to organisations operating with fewer resources, weaker governance, and less regulatory exposure.

The answer is not to abandon AI. It is to sequence innovation properly, embedding governance, assurance, and accountability from the outset, not retrofitting them after harm occurs.

Why this matters for health and social care leaders

For leaders across health and social care, the implications are clear.

• AI adoption strategies must sit alongside governance frameworks
• Patient safety must be prioritised over speed to deployment
• Clinical oversight must be meaningful, not symbolic
• Accountability must be explicit, not assumed. 

When AI fails in healthcare, the consequences are not abstract. They are human.

Continuing the conversation

These questions sit at the heart of ongoing work explored through the HSC Innovation Observatory, which examines innovation at the intersection of governance, regulation, and real-world practice.

The aim is not to resist change, but to support leaders, practitioners, and organisations to engage with innovation critically, responsibly, and with patient safety at the centre.

References

• The Guardian (2026) – ‘Dangerous and alarming’: Google removes some of its AI summaries after users’ health put at risk. 
• MHRA (2025) – Call for Evidence on Regulation of AI in Healthcare.