Keeping Healthcare AI Safe and Effective

The following is a guest article by Steve Frank, Founder and CTO at Med*A-Eye Technologies

Hardly a day passes without a new study comparing the performance of artificial intelligence to human expertise, often with the computer having the edge. Doomsayers fear the loss of human expertise altogether, eliminating not only livelihoods but the evolving knowledge necessary to judge and improve the AI systems themselves.

Don’t worry, say the doom doubters. AI isn’t really thinking — powerful chatbots are just trained to represent relationships between words and phrases. They create humanlike responses to your queries by guessing the next word based on the query itself and the answer it progressively builds. Therefore, say the doubters, human thinkers will always be essential in critical arenas like healthcare.

Maybe.

Deep Learning, Not Deep Thinking

While the principles underlying large language models may be simple, human thought is also simple when reduced to its biological mechanics. Trained on a significant slice of accumulated human knowledge, chatbots like GPT-4 exhibit — or convincingly simulate — humanlike reasoning abilities and vast contextual awareness. If the underlying algorithms designed to mimic human conversation don’t actually reason, they make a darn good show of it. Perhaps most unnerving, AI can evince creativity — the ability to invent new ideas rather than merely synthesizing existing ones, arguably the touchstone of intelligence.

But having the answer to every question doesn’t make you smart, just handy. AI learns only what it’s told to learn; it can’t enter the problem-rich domain of humans. Chatbot aside, AI is best suited to and usually developed to handle a specific task. “Deep learning” uses multiple layers of human brainlike organization to detect patterns and extract features from a complex input. This approach has given rise to diverse applications that analyze medical images, identify a disease, predict its course, and flag hospital patients at risk for conditions such as sepsis.

These applications share a common ancestral paradigm — the “perceptron” — that arose during World War II to classify objects. In those early days and for decades to come, no one would confuse the perceptron’s learning ability with an understanding of the objects it could classify. Today’s neural networks, fantastically more complex successors to the single-neuron perceptron, can detect very subtle patterns — subtle enough to elude human recognition and enable the computer to rival or outperform human experts. AI gives off a “ghost in the machine” vibe as it performs tasks previously entrusted to people with specialized skills and knowledge.

AI’s Big Challenge: Lack of Expert Knowledge

The problem? AI does not genuinely perform as an expert — it just recognizes patterns at an expert’s level. Humans can identify the image regions on which the AI bases its judgment but not the basis for the decision — at least not yet. Efforts to develop “explainable” AI have foundered on the increasing complexity of AI models. Knowing nothing about the world or its narrow specialty within it, AI isn’t smart about what it’s looking at. It’s too dumb (in both senses) to provide the basis for its decisions.

As a result, our trust in AI derives almost entirely from its record of success. Once that record reaches a high enough level, the temptation to unthinkingly rely on its judgment may be irresistible. This mindless trust in machine competence can lead to the erosion of expertise on the part of humans (“deskilling”) and overreliance on AI support (“automation complacency”).

In healthcare, overreliance on AI exacerbates the problem. Clinicians faced with questionable AI output may lose the clinical knowledge necessary to correct it or fail to notice the error because of automation complacency. No AI system is perfect. The possibility of error will always exist — and with it, the possibility of overreliance.

In a recent high-profile example, Ex-Trump lawyer Michael Cohen, seeking to have his probation cut short, identified cases supporting his position and sent them to his lawyers, who included them in the brief they filed. The judge blew his stack when he discovered the cases did not exist. Cohen found the “cases” using a chatbot, which made them up. So, two tiers of lawyers regurgitated the output of a notoriously unreliable AI without discharging that most basic of lawyerly responsibilities — checking the facts.

Humans Must Effectively Harness — Not Overuse — AI, Especially in Healthcare

We, as humans, have to remain more intelligent than our computers and not let them dumb us down or make us lazy. Healthcare AI systems don’t make things up but can make mistakes. Moreover, their training can become stale without our knowing it. Predicting sepsis is a good example. In April 2020, the University of Michigan deactivated its sepsis-alerting AI system because its predictions had become dangerously unreliable. The culprit was the coronavirus pandemic, which fundamentally altered the relationship between fevers and bacterial sepsis.

Healthcare AI promises to reduce medical errors, ease practitioner workloads, and improve the availability and equity of care. Despite its immense value, the advancement cannot be overused.

If we don’t actually understand AI — if we fully comprehend its low-level organization but can’t know the basis for a particular high-level output — we must not abandon our human faculties, based on experience and domain understanding, to think for ourselves and know better. Especially in healthcare, the role of AI should be decision support, never decision-making.

About Steve Frank

A machine-learning innovator and entrepreneur, Steve Frank is the founder of Med*A-Eye Technologies, which develops AI-driven software that helps doctors spot disease in medical images. Before that, he was a patent lawyer and partner at Morgan, Lewis & Bockius LLC.

The following is a guest article by Steve Frank, Founder and CTO at Med*A-Eye Technologies

Hardly a day passes without a new study comparing the performance of artificial intelligence to human expertise, often with the computer having the edge. Doomsayers fear the loss of human expertise altogether, eliminating not only livelihoods but the evolving knowledge necessary to judge and improve the AI systems themselves.

Don’t worry, say the doom doubters. AI isn’t really thinking — powerful chatbots are just trained to represent relationships between words and phrases. They create humanlike responses to your queries by guessing the next word based on the query itself and the answer it progressively builds. Therefore, say the doubters, human thinkers will always be essential in critical arenas like healthcare.

Maybe.

Deep Learning, Not Deep Thinking

While the principles underlying large language models may be simple, human thought is also simple when reduced to its biological mechanics. Trained on a significant slice of accumulated human knowledge, chatbots like GPT-4 exhibit — or convincingly simulate — humanlike reasoning abilities and vast contextual awareness. If the underlying algorithms designed to mimic human conversation don’t actually reason, they make a darn good show of it. Perhaps most unnerving, AI can evince creativity — the ability to invent new ideas rather than merely synthesizing existing ones, arguably the touchstone of intelligence.

But having the answer to every question doesn’t make you smart, just handy. AI learns only what it’s told to learn; it can’t enter the problem-rich domain of humans. Chatbot aside, AI is best suited to and usually developed to handle a specific task. “Deep learning” uses multiple layers of human brainlike organization to detect patterns and extract features from a complex input. This approach has given rise to diverse applications that analyze medical images, identify a disease, predict its course, and flag hospital patients at risk for conditions such as sepsis.

These applications share a common ancestral paradigm — the “perceptron” — that arose during World War II to classify objects. In those early days and for decades to come, no one would confuse the perceptron’s learning ability with an understanding of the objects it could classify. Today’s neural networks, fantastically more complex successors to the single-neuron perceptron, can detect very subtle patterns — subtle enough to elude human recognition and enable the computer to rival or outperform human experts. AI gives off a “ghost in the machine” vibe as it performs tasks previously entrusted to people with specialized skills and knowledge.

AI’s Big Challenge: Lack of Expert Knowledge

The problem? AI does not genuinely perform as an expert — it just recognizes patterns at an expert’s level. Humans can identify the image regions on which the AI bases its judgment but not the basis for the decision — at least not yet. Efforts to develop “explainable” AI have foundered on the increasing complexity of AI models. Knowing nothing about the world or its narrow specialty within it, AI isn’t smart about what it’s looking at. It’s too dumb (in both senses) to provide the basis for its decisions.

As a result, our trust in AI derives almost entirely from its record of success. Once that record reaches a high enough level, the temptation to unthinkingly rely on its judgment may be irresistible. This mindless trust in machine competence can lead to the erosion of expertise on the part of humans (“deskilling”) and overreliance on AI support (“automation complacency”).

In healthcare, overreliance on AI exacerbates the problem. Clinicians faced with questionable AI output may lose the clinical knowledge necessary to correct it or fail to notice the error because of automation complacency. No AI system is perfect. The possibility of error will always exist — and with it, the possibility of overreliance.

In a recent high-profile example, Ex-Trump lawyer Michael Cohen, seeking to have his probation cut short, identified cases supporting his position and sent them to his lawyers, who included them in the brief they filed. The judge blew his stack when he discovered the cases did not exist. Cohen found the “cases” using a chatbot, which made them up. So, two tiers of lawyers regurgitated the output of a notoriously unreliable AI without discharging that most basic of lawyerly responsibilities — checking the facts.

Humans Must Effectively Harness — Not Overuse — AI, Especially in Healthcare

We, as humans, have to remain more intelligent than our computers and not let them dumb us down or make us lazy. Healthcare AI systems don’t make things up but can make mistakes. Moreover, their training can become stale without our knowing it. Predicting sepsis is a good example. In April 2020, the University of Michigan deactivated its sepsis-alerting AI system because its predictions had become dangerously unreliable. The culprit was the coronavirus pandemic, which fundamentally altered the relationship between fevers and bacterial sepsis.

Healthcare AI promises to reduce medical errors, ease practitioner workloads, and improve the availability and equity of care. Despite its immense value, the advancement cannot be overused.

If we don’t actually understand AI — if we fully comprehend its low-level organization but can’t know the basis for a particular high-level output — we must not abandon our human faculties, based on experience and domain understanding, to think for ourselves and know better. Especially in healthcare, the role of AI should be decision support, never decision-making.

About Steve Frank

A machine-learning innovator and entrepreneur, Steve Frank is the founder of Med*A-Eye Technologies, which develops AI-driven software that helps doctors spot disease in medical images. Before that, he was a patent lawyer and partner at Morgan, Lewis & Bockius LLC.