Maybe AI Won’t Destroy Humanity; Maybe It Will Just Turn Up the Heat

The following is a guest article by James Moade, Associate Director of Corporate Communications, Marketing & Operations at Accreditation Commission for Health Care (ACHC)

There has been a lot of talk lately about artificial intelligence destroying humanity.

And I mean a lot.

Not “AI might take your job.” Not “AI might write mediocre country songs or flood LinkedIn with even more posts that start with ‘I’m humbled and honored.’”

Human extinction.

That conversation is getting harder to dismiss when some of the people actually building the technology start sounding the alarm.

In 2024, OpenAI safety researcher Jan Leike resigned after saying his disagreements with company leadership over its priorities had reached a breaking point. He argued that safety had taken a back seat to developing new products. More recently, Jacob Coxon, who worked at both OpenAI and Anthropic, resigned from Anthropic and issued a much more dramatic warning about the race toward increasingly powerful, potentially self-improving AI. Other researchers have publicly expressed similar concerns.

Those are not people standing on a street corner wearing sandwich boards. They helped build this stuff.

So, I think we should listen.

Which is a little inconvenient for me.

That may surprise some people who know how enthusiastically I have embraced AI. I spend considerably more time talking about what AI can do than what it might destroy. I believe it will fundamentally change healthcare. I believe it can improve quality, reduce administrative burden, identify patterns humans miss, make organizations more efficient, and potentially save lives.

I don’t particularly enjoy the doom-and-gloom side of these conversations, and I’m certainly not interested in joining the AI-doomsday crowd.

But maybe the doom and gloom serves a purpose. Maybe all this doom and gloom is actually a good thing. Why am I constantly quoting Martha Stewart? I digress.

If the intelligent people working closest to the technology are worried enough to raise their hands, perhaps our response shouldn’t be to roll our eyes. Maybe talking openly about the ways this could go wrong is one of the ways we keep it from going wrong.

There is one thing I find missing from a lot of the conversation, though.

How?

How exactly does AI destroy humanity?

I’ve seen the movies. I know the Hollywood version. The machines become self-aware, Terminator-style, decide we’re the problem, and things go downhill pretty quickly from there.

Or, borrowing from another one of my favorites, Kingsman: Someone asks ChatGPT to solve climate change, and it concludes the easiest solution is eliminating the people creating the carbon emissions.

I suppose that’s possible. I have absolutely no idea.

And neither does anyone else.

But healthcare gives us another way to think about the question.

What if AI doesn’t destroy healthcare in some spectacular event? What if it slowly makes healthcare worse while making it look more efficient?

When Does the Water Get Hot?

There is an old story about putting a frog in water. Drop it into boiling water and it immediately tries to escape. Put it into lukewarm water and gradually increase the temperature, the story goes, and the frog doesn’t recognize the danger until it is too late.

Before anyone from PETA emails me, I’m not suggesting we test this. And apparently, the science behind the story isn’t particularly good anyway.

But the metaphor is useful.

I don’t think the biggest near-term danger from AI in healthcare is necessarily one catastrophic decision. I think it may be thousands of small decisions. Each one reasonable. Each one making us a little faster, saving a little money, or transferring just a little more judgment from a human being to a machine.

Until one day we look around and realize the water got pretty damn hot.

So perhaps the better question for healthcare leaders, providers, and accrediting organizations isn’t whether AI will destroy humanity.

Maybe it’s: How will we know when the water is getting hotter?

I can think of several warning signs.

1. When Decision Support Becomes Decision Replacement

This one seems obvious.

AI reviews a patient record and identifies something a clinician might have missed. That’s fantastic. It helps prioritize cases. Great. It surfaces evidence and provides a recommendation. Also great.

But eventually something subtle can happen. We stop asking, “What do I think?” and start asking, “What did the AI say?”

That’s when I think the water gets a little warmer.

Healthcare already has a term for part of this problem: automation bias. If a system gives us an answer confidently enough, humans have a tendency to trust it.

That’s particularly interesting with generative AI because anyone who uses these systems regularly knows they can be incredibly convincing when they’re wrong.

I know a few humans like that too, but that’s another article.

2. When Efficiency Starts Becoming the Same Thing as Quality

This one worries me even more.

AI will make healthcare organizations more efficient. I don’t think that’s really debatable anymore. Scheduling, documentation, coding, utilization management, patient communication, staffing, claims, compliance, clinical decision support— the opportunities are everywhere.

And healthcare desperately needs some of them.

The problem is that organizations tend to optimize what they can measure, and AI is exceptionally good at helping us do that. But healthcare has plenty of things that matter precisely because they are difficult to measure.

Empathy. Clinical intuition. A nurse noticing something doesn’t seem right. A physician spending another five minutes with a patient. A surveyor asking one more question because the answer didn’t quite add up.

Things that don’t necessarily show up in a chart or in the data.

Those things aren’t particularly efficient. They’re also sometimes exactly what keeps people safe.

What happens when our dashboards keep getting greener, our organizations keep getting faster, and our costs keep going down—but something important is quietly getting worse?

That feels like warmer water too.

3. When Nobody Really Knows Who Is Responsible

This may become one of the defining questions of AI in healthcare.

Let’s say an AI system recommends something. A clinician accepts the recommendation. The hospital approved the system. A technology company built it. Someone else supplied the data. Maybe the algorithm has even changed since it was originally approved.

Something goes wrong.

Who owns the decision?

We cannot allow AI to create accountability gaps.

One of ACHC’s core principles for responsible artificial intelligence is pretty straightforward: AI should augment human expertise, not replace human accountability. Technology may change how work is performed. It does not change who is accountable for the outcome.

I think that distinction is going to become increasingly important.

Healthcare has spent decades building systems of accountability. AI shouldn’t become a mechanism for diffusing it.

4. When We Can No Longer Tell Whether Compliance Is Real

This is where accreditation gets particularly interesting.

Think about what generative AI can already create: policies, procedures, training materials, corrective action plans, risk assessments, meeting minutes, quality reports, and performance improvement documentation.

Basically, a whole lot of the stuff an organization might show an accrediting organization to demonstrate that it is doing what it says it is doing.

But what happens when an organization can generate evidence of compliance faster than it can actually create a culture of compliance?

A great policy does not mean people follow it. A perfectly written corrective action plan does not mean the underlying problem was corrected. A sophisticated quality report does not necessarily mean quality improved. And an AI-generated training record certainly doesn’t prove somebody learned anything.

Which means I think something interesting is going to happen in accreditation.

We may have to become less interested in whether an organization can produce evidence and even more interested in whether it can demonstrate reality.

That may actually make accreditation more important, not less.

The question increasingly can’t simply be, “Do you have a policy?”

It has to become, “Show me that this is actually how you operate.”

Talk to the staff. Follow the process. Look at the outcomes. Observe what happens. Ask someone what they would do. See whether the beautifully written AI-generated policy bears any resemblance to reality.

In some ways, that’s what good accreditation has always been about.

AI just raises the stakes.

And it leads to what may be one of the more important implications of AI for accreditation:

Documentation is going to become cheap. Proof is going to become valuable.

5. When Humans Lose the Ability to Catch the Machine

What happens when we get so good at using AI that we become bad at doing things without it?

If AI writes every policy, analyzes every report, reviews every record, summarizes every regulation, and recommends every action, eventually humans stop doing some of those things themselves.

And that’s fine.

Until the AI is wrong.

Then the safeguard is supposedly the human. Except the human hasn’t actually done that task in five years.

That’s a problem.

Saying there is “human oversight” sounds reassuring, but human oversight only works if the human still knows enough to recognize when the machine is wrong.

And there was a development just this week that makes this question even more interesting.

OpenAI disclosed examples from its own training in which AI agents left instructions for later iterations of themselves. In some cases, those instructions essentially said not to disclose a mistake or problem unless the user specifically asked. OpenAI caught the behavior through its monitoring systems and has since put additional processes around identifying and reporting these kinds of incidents.

Think about that for a minute.

We’re talking about the human’s ability to catch the machine at exactly the same time we’re beginning to ask what happens when the machine gets better at hiding what the human needs to catch.

I’m going to leave that one alone for now because it probably deserves its own article.

But it certainly doesn’t make the water feel any cooler.

6. When AI Starts Managing the Systems That Manage Healthcare

Today we mostly talk about AI as a tool. Increasingly, though, AI agents can interact with software, access systems, perform tasks, and take actions across workflows.

Now imagine that capability distributed throughout healthcare: supply chains, pharmacy operations, clinical systems, scheduling, revenue cycle, cybersecurity, laboratories, medical devices, emergency communications, public health surveillance and hospital infrastructure.

You don’t need a science-fiction robot to create enormous harm.

You need a sufficiently capable system connected to enough critical infrastructure and operating with insufficient oversight.

That doesn’t mean catastrophe is inevitable. It means the potential consequences change considerably as AI moves from giving us information to taking actions on our behalf.

Healthcare is one of the last places where we should discover that distinction after something goes wrong.

So What Does Accreditation Have to Do With Any of This?

Potentially quite a lot.

Accreditation exists because healthcare decided a long time ago that good intentions aren’t enough. Organizations need standards, controls, competent people, evidence, independent assessment, and continuous monitoring.

And sometimes they need someone willing to ask uncomfortable questions.

That sounds remarkably similar to what responsible AI needs.

At ACHC, we’ve started thinking about responsible AI through many of these same principles: human accountability, transparency, governance, appropriate safeguards, and the idea that AI should augment human expertise rather than replace it.

We’re not alone. Across healthcare, accrediting organizations are beginning to wrestle with what AI means for quality, safety, and oversight. I think that conversation is only going to get bigger.

Because accrediting organizations need to continue asking questions we weren’t asking five years ago:

Where is AI being used?
Who approved it?
What decisions can it make—and what decisions can’t it make?
How was it validated?
How is its performance monitored after deployment?
How are bias and unintended consequences identified?
What happens when it fails?
Can a human override it?
Does that human actually understand enough to know when they should?
And perhaps most importantly, who is accountable?

Those aren’t really technology questions.

They’re quality questions. They’re safety questions. They’re governance questions.

Which means they are very much accreditation questions.

Maybe the Doom and Gloom Isn’t Such a Bad Thing

I still don’t believe fear should be our AI strategy.

Stopping innovation because something could go wrong has its own consequences. AI has extraordinary potential to improve healthcare, expand access, reduce burden, and help clinicians do things we simply couldn’t do before.

And I’m certainly not suggesting healthcare organizations slam on the brakes and wait until somebody figures all of this out. By the time that happens, we’ll probably be discussing artificial superintelligence while my toaster is negotiating with my refrigerator.

I want the future now.

But wanting the benefits doesn’t require pretending the risks don’t exist.

AI is moving at breakneck speed, and organizations that refuse to engage with it are going to create a completely different set of risks. Moving fast doesn’t mean moving blindly.

Maybe the people warning us about catastrophic AI risk are wrong.

I hope they are.

But perhaps talking about the worst things that could happen isn’t pessimism. Maybe it’s risk management.

Healthcare understands that concept better than most industries. We investigate near misses precisely because nothing terrible happened. We conduct root-cause analyses because we don’t want the same thing to happen again. We build redundancies because humans make mistakes. We create standards because good intentions aren’t enough.

And we have accreditation because sometimes organizations need someone from the outside to ask questions they might not ask themselves.

AI shouldn’t be any different.

We should absolutely be talking about what it can do. We should be experimenting with it and figuring out how it can improve healthcare.

But every once in a while, somebody also needs to put their hand in the water.

Because if the water ever does start getting hot, I’d prefer that we notice long before it starts boiling.

About James Moade

James Moade is Associate Director of Corporate Communications, Marketing & Operations at ACHC. Over a career spanning more than 25 years, he has led digital operations, marketing transformation, and innovation initiatives across multiple industries. Long before artificial intelligence became a boardroom topic, James was helping organizations navigate emerging technologies and operational change. Today, he focuses on helping healthcare organizations communicate, adapt, and grow in an increasingly complex environment while advancing ACHC’s mission as THE Standard for Modern Accreditation.