Michael Higgins
All writing

Designing AI products for regulated industries: lessons from healthcare

The best AI product safety training available is not at an AI company. It is inside industries where software has always been able to hurt someone.

Working draft, content pending

The most useful AI safety training I ever received did not come from an AI company. It came from healthcare.

For the past six years I have led AI product work inside America's largest healthcare company, building AI that 400,000 employees use to get real work done. Healthcare is a strange place to learn AI product management, and that is exactly why it works. Every instinct the industry beat into me turns out to be the instinct AI products need most.

Frontier AI companies treat safety as a research problem, and it is. But it is also a product problem, and on the product side the playbook already exists. Regulated industries have been running it for decades. Here is what it says.

Do no harm is a product principle, not a slogan

Healthcare's founding constraint is that your product can hurt someone. Not "degrade a metric." Hurt a person. When that possibility is real, you stop asking "does it work in the demo" and start asking "how does it fail, who does it fail for, and does anyone notice in time."

That posture transfers directly to AI. A model that confidently miscategorizes a request is not a rounding error; it is a person who does not get what they need, at scale, from a system that sounds sure of itself. Bring AI into a function like procurement and the hard design work is not the model. It is the failure choreography around the model: what the system does when it is uncertain, how a person overrides it, and how the override teaches the system instead of disappearing into a log.

The rule I hold my teams to: AI handles the triage and the lookups, people make the judgment calls, and the system fails visibly, never silently. That is "do no harm" translated into product language.

Regulation is a design material

Most product teams treat regulation as friction to be minimized. Healthcare taught me to treat it as a material to design with, the way an architect treats gravity.

HIPAA and its cousins force answers to questions every AI product will eventually be required to answer. What data is this system allowed to see? Who consented to what? Who is accountable when the system acts? Where does a decision get reviewed by a person? Teams that answer these questions early do not move slower. They move faster later, because clarity about data and accountability is exactly what unblocks enterprise adoption, procurement review, and security sign-off.

The AI industry is converging on this the hard way. Model cards, audit trails, human-in-the-loop requirements: these are healthcare-shaped ideas arriving under new names. Teams fluent in designing inside constraint will ship trustworthy AI faster than teams that experience constraint as insult.

Bias has a blast radius

In consumer software, a biased model shows the wrong ad. In healthcare, a biased model can route the wrong patient to the wrong level of care. The industry learned, painfully and publicly, that bias is not a model metric. It is a question about who a decision touches.

The discipline that follows is stakeholder tracing: for every automated decision, name the people it lands on, then check the failure rate for each group separately, never on average. Averages hide exactly the people who get hurt. When we built automated risk scoring, the useful question was never "what is the accuracy." It was "who gets flagged that should not be, who does not get flagged that should be, and what happens to each of them next."

AI product teams inherit this discipline whole. The blast radius of a biased foundation model is wider than anything healthcare ever shipped, and the tracing has to be just as concrete.

Skeptics are the adoption strategy

A workforce of 400,000 people contains every possible attitude toward AI: enthusiasts, pragmatists, and people certain it is coming for their jobs. The instinct is to build for the enthusiasts and wait for everyone else to come around. That instinct is wrong, and in healthcare it is dangerous, because the skeptics include many of your most experienced people, and their skepticism encodes real knowledge about how things fail.

The adoption framework that actually worked for us did not try to convert skeptics with enthusiasm. It earned them with recoverability and transparency: you can always see what the AI did, you can always undo it, and when it is unsure it says so and steps back. Skeptics do not adopt AI because they start trusting technology. They adopt it because the product stops asking for blind trust.

I now believe this is the general case. The AI products that win the enterprise will be the ones designed for their most skeptical user, because that user's bar, "prove I can recover when you are wrong," is simply the correct bar.

What this means for AI product teams

If I compressed six years of healthcare AI product work into a checklist for teams building AI anywhere, it would be short:

  • Design the failure, not just the feature. Decide what uncertainty looks like on screen before you decide what success looks like.
  • Treat every regulatory question as a preview of your roadmap. Data provenance, consent, accountability, and review are coming for every AI product; answering early is a speed advantage.
  • Trace every automated decision to the specific people it lands on, and measure failure by group, never by average.
  • Build for the skeptic. Recoverability, visibility, and honest uncertainty are what turn AI from a demo into infrastructure.

Healthcare did not teach me to slow AI down. It taught me what it takes for AI to be trusted with things that matter. That is the product bar now, in every industry, whether the regulators have arrived yet or not.

Contact

Let's build AI people actually use.

I'm interested in AI deployment, architecture, and product leadership roles where design judgment, enterprise scale, and human outcomes meet.