You run a health insurance company. Someone from a major consulting firm is in your conference room right now, showing you a slide deck.
The pitch: prior authorization is broken. Your reviewers are drowning in faxed clinical notes and mismatched forms. Decisions that should take hours take weeks. Patients are waiting. Physicians are furious. CMS is watching. The consulting firm has committed a billion dollars to build AI-powered “agentic systems” that will rebuild how your organization decides — faster, fairer, less wasteful. Clinical reasoning. Prior auth. Claims integrity. Care navigation. End to end. AI-first.
It sounds like the future.
They’ve already started hiring the engineers to build it. The job posting is public. Let’s read it together.
What the posting says
The role is called Agentic AI Engineer — Healthcare AI. The posting is thorough, specific, and honest in the way that job postings are uniquely honest: it is a company talking to insiders, not to you. It doesn’t think you’re reading.
Here is what it describes. A system of LLM-powered agents capable of multi-step reasoning, planning, tool use, and workflow execution against “complex, regulated operational processes.” Agents that handle prior authorization decisions, claims integrity review, and clinical reasoning. Agents that retrieve clinical criteria, assemble context from patient records, apply policy logic, and produce auditable rationales.
The posting also specifies — and this is where you should slow down — that engineers will need to build “recovery from compounding errors,” “planning under uncertainty,” “robust tool use when individual steps fail,” and “failure-handling to reduce hallucinations and unsafe actions.”
Read that again. The system needs built-in recovery from its own compounding errors. It needs guardrails to reduce — not eliminate, reduce — hallucinations. It needs robustness for when its own steps fail.
This is not a system that works and occasionally breaks. This is a system that routinely breaks and is engineered to recover. There is a difference.
You understand that difference.
The technical risk you already see
You’ve managed complex operational systems your entire career. You know what happens in what Charles Perrow called a tightly coupled system — one where each step feeds directly into the next, with no buffer, no independent redundancy, no window for a human to catch a problem between stages. Nuclear plants. Chemical processing. Space shuttles. When tightly coupled systems fail, they fail fast and they fail completely, because a local error propagates at machine speed into systemic collapse.
A multi-step agent system is tightly coupled by definition. Step one’s output is step two’s input. Step two’s output is step three’s input. If each step is 95% reliable — an optimistic estimate for current LLMs on complex reasoning — five steps in series gives you 77% end-to-end reliability. Ten steps: 60%. This is not speculation. Shubhra Mittal’s 2026 study at Microsoft AI, “How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making,” tested nine models across multiple task families and horizons and found that task success follows a geometric decay law — and that per-step reliability saturates well below 1.0 even for the strongest models, guaranteeing eventual collapse at sufficient chain length.
But the real danger isn’t the error rate. It’s the error signature. When a human reviewer makes a mistake, the output looks wrong — a missing field, a nonsensical approval, a file sent to the wrong queue. When an LLM makes a mistake, the output is confident, well-formatted, and grammatically perfect. The next agent in the chain has no reason to doubt it. The error doesn’t look like an error. It looks like a correct input.
The posting knows this. That’s why it asks engineers to build “self-correction” and “human-in-the-loop checkpoints.” But you recognize the irony — the one that automation researcher Lisanne Bainbridge identified in 1983: the more you automate a process, the harder it becomes for the human overseer to catch the moment the automation fails. The human is no longer doing the work that would have given them the context to notice something is wrong.
You automated the process to remove the human. Now you need the human back to watch the automation. But the human you need no longer exists — you automated away the job that would have trained them.
The actuarial risk you can calculate
This is where it gets worse. And this is your home field.
Your business model rests on one mathematical premise: independence of risk. The law of large numbers works because policyholder A’s claim has nothing to do with policyholder B’s claim. Your actuaries price risk on this assumption. Your reserves are built on this assumption. Your reinsurance contracts are structured on this assumption.
Now consider what an agent system introduces. When a human reviewer makes an error, it is an individual error — idiosyncratic, uncorrelated with other reviewers’ errors. A hundred reviewers make a hundred different mistakes on a hundred different cases. This is noise. Your models handle noise.
When an agent system has a flaw in its reasoning logic, it makes the same error on every case that matches a certain pattern. This is not noise. This is correlated failure — the one thing your actuarial models assume does not happen.
You know the precedent. You were in the industry in 2008. AIG sold credit default swaps priced on models that assumed mortgage defaults were independent events. They were not. Defaults were highly correlated — when one neighborhood went, the region went; when one region went, they all went. AIG’s model said the exposure was manageable. The actual exposure was infinite. A company whose entire purpose was to price risk was destroyed by a risk it mispriced.
Nassim Taleb has a phrase for this payoff structure: picking up pennies in front of a steamroller. You earn in small, steady increments through correct pricing. You lose catastrophically, without ceiling, when the model breaks. Your one protection is that the model is right. An agent system introduces a failure mode your model has never accounted for — systematic, correlated errors at the decision layer — and you cannot even estimate the probability distribution, because LLM failure modes are not understood well enough to model actuarially.
In Taleb’s framework, this is Knightian uncertainty: not risk you’ve mispriced, but risk you cannot price at all.
The legal exposure you cannot cap
And now it gets worse again.
You have lawyers. You know the difference. When an individual employee makes an error in a prior authorization decision, the liability theory is negligence. The plaintiff has to prove your employee did something wrong — failed to follow protocol, ignored relevant information, acted unreasonably. This is familiar ground. You carry insurance for this. You have decades of precedent. Damages are bounded.
When a product has a defect, the liability theory shifts to strict liability. The plaintiff does not need to prove you were careless. They only need to prove the product was unreasonably dangerous. The evidentiary burden is lower. The exposure is higher.
And if the defect is systematic — if it affects every case processed through a given pathway — the individual tort becomes a class action. One flawed rule in your agent’s logic, applied to ten thousand prior auth decisions, creates ten thousand plaintiffs with the same cause of action.
This is not hypothetical. In February 2026, the California Superior Court consolidated twelve product liability cases against OpenAI — In re: ChatGPT Product Liability Cases, JCCP No. 5431. Legal commentators are calling it the tobacco playbook. The asbestos playbook. The opioid playbook. The EU’s revised Product Liability Directive explicitly classifies software — including AI systems — as products, with member states transposing by December 2026. The Yale Law Journal published a framework for “nondeterministic torts” this year, specifically addressing agentic AI liability.
Plaintiffs’ attorneys will seek your internal safety evaluations, your model cards, your red-teaming records, your product roadmap communications. Every guardrail listed in that job posting — every “eval gate,” every “observability and tracing” requirement — is both your best engineering practice and your future Exhibit A. Having guardrails proves you knew the risks. Not having them proves you were reckless. Neither position protects you.
And remember the layer beneath: the errors are correlated. Every patient affected by the same systematic flaw belongs to the same class. Your actuaries cannot price it. Your lawyers cannot cap it.
The answer you already have
You’ve been in this business long enough to know what prior authorization actually is. It is not a reasoning problem. It is a decision tree.
Your clinical criteria manuals — InterQual, Milliman — are already structured as explicit rule sets. Diagnosis code. Procedure code. Prior conservative treatments attempted. Clinical thresholds met or not met. The logic is deterministic. It is already written down.
What makes the process slow is not the decision. It is the input. Your reviewers spend most of their time not applying criteria but wrangling information — reading faxed documents, parsing inconsistent clinical notes, chasing missing fields from incomplete submissions. They are not doing clinical reasoning. They are doing data entry under duress.
You already know the architecture. A rule-based engine handles the decision logic: deterministic, auditable, traceable, each output tied to a specific policy clause. AI handles the periphery: reading documents, extracting fields, flagging gaps. Each AI call is short, independent, and individually verifiable. A wrong extraction doesn’t automatically cascade into a wrong decision — it routes to a human for confirmation.
Call this deterministic spine, probabilistic periphery — a coinage, not an established term. The spine is certain. The periphery is where AI earns its keep without threatening to burn down the house.
This architecture doesn’t just reduce technical risk. It dissolves the actuarial problem: errors are independent again, so your models work again. It sidesteps the legal problem: a rule engine’s decision traces to a specific rule, which is a fixable bug, not a design-defect theory. Your lawyers can defend it. Your actuaries can price it. Your compliance team can audit it.
You knew this. You just haven’t heard anyone say it in the room where the billion-dollar pitch is happening.
Why no one will say it
Because this architecture has a problem. Not an engineering problem. A narrative problem.
“We built a reliable rule engine with AI-powered document parsing” is not a billion-dollar story. It is not agentic. It is not reasoning. It is not rebuilding the decision-making machinery of American healthcare. It is a smarter, safer version of what you already have — and smarter, safer versions of what you already have do not earn consulting firms billion-dollar commitments or keynote slots.
Technology historians call this technology push versus demand pull. Demand pull: you have a problem, you find the right tool. Technology push: you have a tool, you find the biggest problem you can attach it to. Agentic AI is the tool. Healthcare prior authorization is the problem — not because agents are the right solution, but because prior auth is the largest, most expensive, most politically visible problem available to absorb the investment thesis.
The job posting is the proof. It describes engineers who will build recovery mechanisms for compounding errors, guardrails against hallucinations, safety controls for unsafe actions. The entire engineering effort, stated plainly in the listing, is to make a probabilistic system behave as if it were deterministic.
You could start from deterministic instead.
Coda
I wrote previously that the failure modes emerging in AI agent systems are not new — that seventy-five years of automation research already catalogued them, and that we disabled the safeguards that once kept them in check. That was theory.
This is the specimen.
A well-funded, carefully designed project, backed by a billion dollars, staffed with talented engineers, building a system whose own job description enumerates the ways it will break — and tasking those engineers with building recovery mechanisms, rather than questioning whether the architecture is the problem.
The old fires are lit. Now you can see the smoke.
Scopiology 