The Rules Are Half the Physics

Regulated industries break world models in four specific ways: colliding clocks, feedback loops, legal fences, and your own output becoming your next input. They also hand you the one thing world-model builders everywhere else lack, which is a supply of free, dated, published interventions to test against.

In the first post I made a promise that sounded almost too convenient: in a regulated industry, you do not have to infer the rules from a pile of data, because the rules are already written down. Someone published them. You can build your world directly from them, the way a game builds a cave system from cave-generation rules instead of hand-placing ten thousand caves.

I still believe that. We spent the last stretch stress-testing it against a genuinely ugly real-world regulated system, one of the largest and most litigated in the country, and it held up.

It came back with a correction stapled to it, though, and the correction turned out to be more useful than the original idea.

The rules give you a skeleton, not a world. They are exact where they apply and completely silent everywhere else, and getting that boundary wrong in either direction is how these projects quietly fail.

And then something I did not expect at all. The same rulebook that makes this work so hard hands you a thing that world-model builders in every other domain would pay enormous money for: a free test set.

Let me show you, starting with taxis.

The Manhattan taxi problem

This is my favorite AI result of the last few years, and it is the drunk-cats story of world models.

A team of researchers trained AI models on taxi routes through Manhattan. Millions of trips, turn by turn. The models got very good. By the obvious measure they were nearly perfect: the next-turn predictions were valid close to 100 percent of the time, and the routes the models produced were legal, sensible routes between 96 and 99 percent of the time.

Then the researchers did something clever. They reconstructed the map these models had implicitly learned, by asking them a lot of questions and drawing the streets those answers implied.

The map was nonsense. Streets running in physically impossible directions. Roads crossing over other roads where no overpass exists. The models could navigate Manhattan beautifully while carrying around an internal picture of Manhattan that could not exist.

That alone would be a good story. Here is the part that changed how we work.

The researchers added detours. Close a street at random, force the model off its usual path, and see if it can still get you home.

The model trained on shortest paths went from 0.99 accuracy, to 0.69 when just one percent of trips hit a detour, to 0.08 at ten percent, to 0.00 at fifty percent. Not degraded. Gone. A second model trained on near-optimal routes did the same thing.

There was a third model. It had been trained on random walks: aimless, pointless wandering, the kind of trajectory no real taxi would ever drive. By every normal measure it was trained on worse data.

That one held at 0.99 through a ten percent detour rate and was still at 0.97 when half of all trips were detoured, sitting there working fine while both of the “good” models were at zero.

The model trained on the garbage data was the one that actually understood the city.

Once you see why, you cannot unsee it. The shortest-path models only ever saw optimal routes, so they never visited the streets a detour puts you on, and their internal picture had no reason to include them. They were not modeling Manhattan. They were modeling good trips through Manhattan, which is a much smaller thing that looks identical right up until the moment you need it not to be.

So the real question when you build a world model is not “what should it predict.” It is “where do I get detours.”

Hold that thought. It is the whole post.

What a world model actually needs

Four things. Only the fourth is hard.

1. A state. Something you can be in.
2. An action. Something you can do.
3. A transition. A rule that takes a state and an action and gives you the next state.
4. Coherence under intervention. When you do something the model never saw in training, it still behaves sensibly.

Most things sold as world models have the first three and quietly skip the fourth. A good number do not really have the third either.

There is a precise way to say this that I have found useful in actual meetings. Any model of a changing system has two halves: one that says how the world evolves, and one that says what your instruments see. Nearly every “world model” I have reviewed inside a regulated company is entirely the second half. It is a careful, expensive, genuinely valuable description of what the data means, at what grain, from what source, and how it lies to you.

That is an observation model. It is a good thing to have. It is not something you can run forward, and calling it a world model starts a conversation that always ends the same way, because the very next question is “so it can forecast?” and the honest answer is no.

Getting this distinction right is not pedantry. It tells you exactly what you are missing.

Four things that make this hard when there is a regulator in the room

One: nothing runs on the same clock, and you did not pick the clocks

This is the failure I now look for first, because it silently ruins more work than anything else on this list.

In a regulated system, different parts tick at wildly different rates, and most of the rates are set by law:

• Contracts renew once a year and are frozen in between.
• Eligibility changes annually, in a window.
• Transactions settle over weeks or months, then get revised.
• Some money lands two or three quarters after the event it belongs to, and lands as a lump at a completely different level of detail than the event.
• Scores get restated several times for a period that already ended.
• Policy can change any day at all, in the middle of a contract.

Now watch what happens when two of those collide.

Almost every performance number in a regulated business is a fraction. If the top counts events on one clock and the bottom is recalculated on a different clock, the fraction moves for reasons that have nothing to do with what it claims to measure. Drop your most expensive customers during the year, then compute the year’s rate against a population recalculated at the end, and your numbers improve. Nobody improved anything.

We looked at a case where a genuinely impressive published improvement landed in the exact same window as a government program that independently pushed behavior in the same direction, in overlapping regions. Two causes, one effect, no control group. The number is probably not wrong. It is uninterpretable, which is a different and more annoying problem, especially if you were planning to train something on it.

Most “model failures” in regulated industries are not model failures. They are two clocks colliding.

Before I believe any rate a regulated system reports about itself, I now ask three things:

• What is the denominator, and when was it frozen?
• What else changed inside the same window?
• Which date is this built on: when it happened, when it settled, or when it was last revised?

You do not need a model to ask any of those. You do usually need to be inside the company to answer them, which is itself worth knowing.

Two: the world reads your model

A hurricane does not read the forecast.

This is the deepest difference between modeling weather and modeling a regulated business, and almost nothing in the world-model literature has a category for it.

In a regulated system, the quantity you are modeling is produced by people who are paid according to the model. A scoring model that determines payment changes what gets written down. A price benchmark feeds the negotiations that produce next year’s prices, which are your next input. A rule about what gets approved changes what gets requested.

Goodhart’s law stops being a clever aphorism on a slide and becomes a force inside the dynamics.

My favorite evidence for this is not a paper. It is a regulator conceding the point inside its own arithmetic. One federal payment formula we worked through contains an explicit term that scales for the fact that people consume more when their coverage is more generous. The regulator wrote the feedback loop into the formula, because pretending it was not there would have been indefensible.

Two rules fall out of this, and both are cheap on day one and brutal to retrofit:

Some quantities are things to measure, never things to optimize. The moment a measurement becomes a target in a system where the measured party generates the measurement, you have built a machine for producing the measurement instead of the thing.

Provenance is a required field on every number, not metadata. If a value can arrive by several different routes, and some of those routes are contested, then any total you cannot break down by route is a total you cannot defend.

Three: the fence around your data is legal, not technical

This one surprises engineers every single time.

Inside a big regulated group you will find data the company unambiguously owns, at enormous scale, that the modeling team is not allowed to use. Not a pipeline problem. A governance firewall, a consent decree, or a statute.

Stated plainly: the usable data estate is smaller than the data estate, and very often the most distinctive asset in the building is the one you are least free to touch.

The compensation is a pattern we have now hit three separate times, and it has stopped feeling like coincidence. The compliance-safe design and the technically correct design keep turning out to be the same design.

Model groups rather than individuals, and you get the ability to answer “what if” questions without a single person’s record entering the system. Build only on things that were published under a disclosure rule, and your provenance story writes itself. Describe a distribution instead of emitting a recommendation, and you stay out of an entire category of liability while shipping the more honest object anyway.

When a constraint keeps pointing at the same architecture from three different directions, that is usually the architecture.

Four: if it works, everyone uses it, and then everyone is wrong together

Short, but it belongs here.

There is agent-based simulation work on the insurance industry showing that when too few distinct risk models are in use, default risk rises and industry profits fall. Model diversity is not just prudent. It measurably improves outcomes for the participants.

For anything benchmark-shaped there is a nastier version: your output becomes next year’s input. A widely adopted benchmark stops measuring an independent market and starts measuring its own past output, and the variation that made it worth having compresses toward whatever number you happened to publish.

That is not a distant risk. It is the success case.

The turn: the rulebook is also the fix

Now the good half, and it is better than I expected going in.

Gift one: some transitions are exact, and free

Buried in the statutes are transitions that fire on a calendar and reference nothing at all about the actual condition of the thing they act on. After a fixed number of months, responsibility moves from one party to another. A status ends a fixed period after a triggering event. A weight phases in at 33 percent, then 67, then 100.

These need no estimation, no data, and no confidence interval. They are already exact. They are pure “what if” structure, sitting in a public document, free.

The design move this implies is to split the state into factors that move for different reasons:

• a factor that moves by statute and calendar, which you compile rather than learn, with zero fitted parameters
• a factor that moves randomly, which is the only part you actually fit
• a factor that is simply observed

Then model the arrows between them, and pay particular attention to the arrow running from the rule-governed factor into the random one. That arrow is the feedback loop from hard part two, and every model fitted on raw history gets it wrong, because in raw history the rule change and the behavior change show up at the same moment and look like one event.

This is a refinement of “build the world from the rules”, not a retreat from it. The rules generate the skeleton. Controlled randomness fills in the rest. It is the same shape as writing cave rules instead of drawing caves, moved up one level.

Gift two: free detours

Remember the taxis. You need detours, and in most domains getting them is painful or impossible. You cannot close streets in a live business to find out whether your model understands it.

In a regulated industry, the regulator publishes the detours in advance, with dates on them.

They are everywhere once you start looking:

• Phase-ins. A weight moving 33, then 67, then 100 over three years is a dated shock to exactly one part of the system and nothing else.
• Partial mandates. A program that becomes compulsory in 188 named markets and applies nowhere else is very close to a randomized experiment somebody else paid for.
• Calendar transitions. A clock that changes which party pays after a fixed interval, with no reference to anyone’s actual situation.
• Definition changes, which are the sharpest tool in the box and get their own section below.

Hold these out of training. Then check whether the model predicts them. That is a real test of understanding: it needs no ground truth, it cannot be passed by training longer, and a model can ace every accuracy metric you own and fail it completely.

Show me: the definition-change test

Enough theory. Here is the one I would run first, in five steps, on whatever you are building.

Of all the free detours, the sharpest is a change in how something is measured, on a known date, where the underlying reality did not change at all.

We worked through a case where a clinical measurement formula was officially revised on a single day. A large number of people moved between severity categories immediately. Nothing about any of those people changed. The map moved. The territory did not.

That is a scalpel. A model that predicts a real change in the world at that boundary has confused the map with the territory, and you find out in one test.

Step 1. Find your date. Every regulated industry has several of these lying around: a reclassified category, a revised index, a changed threshold, a restated definition. Write down the date and the exact scope of what the change touched.

Step 2. Split on the date and train only on “before.” No peeking. The whole point is that the model has never seen the new definition.

Step 3. Pre-register your predictions. This is the step everyone skips and it is the one that makes the test honest. Before you run anything, write down two lists: what must move, and what must not.

change:  measurement formula revised, effective 2021-XX-XX
scope:   affects how the category is computed, not what is being measured

MUST MOVE (the map)
  - distribution across severity categories shifts on the date
  - shift size matches the published effect of the revision
  - the shift is a step, not a ramp

MUST NOT MOVE (the territory)
  - underlying event rates
  - downstream outcomes for the same people
  - anything measured by an instrument the revision did not touch

Step 4. Roll the model forward across the date. Do not retrain. Do not tune. Just let it run into the change the way it would in production.

Step 5. Score two numbers, not one.

map_score       did the measured quantity move as the rule says?
territory_score did the underlying quantity correctly stay put?

coherent  = map_score high AND territory_score high
confused  = map_score high AND territory_score low   <- the dangerous one
blind     = map_score low

The failure everybody assumes they will get is “blind”: the model misses the change. In our experience the common failure is confused. The model dutifully predicts that the world got worse on the day the ruler changed length, because in every year of training data, a rising category count really did mean a worsening population.

That model will pass your accuracy dashboard forever and be wrong every time a rule changes. Which, in a regulated industry, is constantly.

Run this test on one dated definition change this quarter. It is cheap, it is decisive, and I have not yet seen a team run it and learn nothing.

Where this lands in a real industry

I have kept this general on purpose, but abstraction is easy to nod along with and hard to act on. So here is the concrete version, in one of the most heavily regulated and most publicly scrutinized industries there is: US health insurance.

These companies are unusually open about where they are deploying AI. Below is a map from what they say they are doing to which hard part it runs into. Everything in the left column is a public statement by the company or a public report. I have marked what is a company claim, what is a regulator or congressional finding, and what is an unresolved allegation, because those are three different things and collapsing them is how commentary about this industry usually goes wrong.

This is not one company’s problem, which is what makes it urgent rather than academic. Across roughly 53 million coverage determinations in 2024, denial rates by parent company ranged from 4.2 percent at Elevance to 12.8 percent at UnitedHealth Group, with Centene at 12.3, Kaiser at 6.8 and Humana at 5.8. Of the denials that were appealed, 80.7 percent were overturned. In July 2026, senators sent bipartisan letters to UnitedHealthcare, Humana and CVS asking for records on AI and predictive technology in post-acute denials, and asking each to confirm whether it remains policy that a final denial cannot be made by AI.

Read that last sentence as a product requirement, because that is what it is.

Notice the shape of the whole table. Not one of these is a modeling problem. Every single one is a clock, a feedback loop, a legal boundary, or an attribution question. That is the argument of this post in miniature, and it is why we think the interesting research here is not bigger models.

Three things we are building next

Built in the open, as always.

1. A detour benchmark for regulated world models. Coherence under intervention has no standard test, so everyone reports accuracy and nobody reports robustness. We are assembling dated regulatory interventions into a public, versioned, held-out suite: phase-ins, partial mandates, calendar transitions, and definition changes. Each one gets a date, a scope, and a pre-registered list of what must and must not move, in the format from the section above. Score a model on the detours instead of on the next step. The expensive part of building an intervention benchmark is normally running the interventions, and here the regulator already did that for us.

2. Compiled rule skeletons inside procedurally generated worlds. This one is a straight continuation of the seed work. We want to make the factoring literal: compile the statutory factor into an exact, testable transition with zero learned parameters, generate the rest of the world procedurally around it, and then measure what the compiled spine actually buys. Our hypothesis is sample efficiency and robustness, and that the second matters far more. The experiment we care about: does an agent trained in a world with a compiled rule spine survive a detour that kills an agent trained in a fully learned world?

3. Practice worlds with an adversarial clock. Every simulator I have seen, including ours, gives the agent clean time. Events arrive in order, numbers are final when you see them, and the past does not change. Real regulated systems are nothing like that. Values restate. Money lands quarters late in a lump. Populations get recomputed retroactively. Definitions shift under a running series.

So we are building a simulator that does all of that on purpose, as a hazard, and training agents under it. My strong suspicion is that an agent which has only ever seen aligned clocks fails on first contact with a restatement, and that this, rather than any weakness in the model itself, is what actually kills these deployments in the field.

The only two questions that matter

As promised in the first post, every post in this series ends with the two questions you will actually be asked.

One: what does it cost me when this goes wrong? A world model that is locally accurate and incoherent under intervention does not fail on a dashboard. It fails the first time a rule changes, which in a regulated business is scheduled, published, and inevitable. And it fails in the most expensive possible way: confidently, in the same direction as every historical pattern, right at the moment the historical pattern stopped applying. The taxi model was at 99 percent until the street closed, and then it was at zero. Your regulator will change a definition next year. You already know the date.

Two: can I defend it to a regulator? This is where the reframe at the heart of this post earns its keep. People assume regulated industries demand mechanistic, hand-written models and will never accept a learned one. Look at what the rules actually require, though, and you find documentation, versioning, validation, reproducibility, and the ability to reconstruct what the model said on the day someone acted on it. Every one of those is about specification: is it written down, is it versioned, can you argue with it. Not one of them is about mechanism.

In physics-shaped domains the two look identical, because physics hands you the documentation for free. In a domain with no physics of behavior, you have to produce specification some other way, administratively, and the result is that some of the most audited and most litigated models in the country are ordinary fitted statistical models with published coefficients. Nobody minds, because you can see them and argue with them.

You are not required to be mechanistic. You are required to be specified. Immutable versioned artifacts, provenance on every input, the original bytes kept next to every transformation, a history you can reconstruct. Most good engineers already do this for correctness reasons and have no idea it is also the entire compliance story.

The short version

Regulated industries are a hard place to build world models. The clocks collide. The world reads your model and changes underneath it. The fence around your data is legal rather than technical. And if you succeed widely enough, your own output contaminates your next input.

They are also the best place to build world models, for one specific reason. Everywhere else, the detours you need to find out whether your model understands anything are expensive, rare, or impossible to run. Here they are published in advance, with dates, by somebody else.

The rules are half the physics. The other half is knowing exactly where they stop.

This is part of an ongoing series on building AI for regulated decisions you can explain, reproduce, and defend. Start at Hello World Models, then A World From a Seed and The LLM Learned to Stop Planning.

Sources and further reading

The taxi result, and testing world models

• Keyon Vafa, Justin Y. Chen, Ashesh Rambachan, Jon Kleinberg, Sendhil Mullainathan, Evaluating the World Model Implicit in a Generative Model, NeurIPS 2024 (spotlight). The Manhattan maps, the detour collapse, and the random-walk model that survived. The diagnostics are inspired by the Myhill-Nerode theorem, which is a nice reminder that the sharpest tool here is fifty years old.
• One precision note if you cite this: the 96 to 99 percent figure is route validity across all three models, not shortest-path optimality. Optimality was 97 percent and 94 percent for the two goal-directed models.

The world-model idea itself

• Yann LeCun, A Path Towards Autonomous Machine Intelligence (2022), and Meta’s I-JEPA.
• DreamerV3: learning a latent world model and training a policy inside it.
• MuZero: planning with a learned model when the rules are not given to you. Worth reading precisely because our situation is the opposite, since our rules are given to us.
• DeepMind Genie: generative interactive environments, the “world model as simulator” end of the spectrum.

Doing this properly in a regulated setting

• The ISPOR-SMDM Modeling Good Research Practices task force, a seven-report series from 2012 published jointly in Value in Health and Medical Decision Making. Report 3 is the method spec for state-transition models and Report 7 is transparency and validation. If you are building this in health, do not invent a methodology. This is it.
• CISNET, the National Cancer Institute’s comparative modeling network: multiple independent models of the same disease, run against shared inputs, specifically so the disagreement between them can be studied. Institutionalized model diversity, already sitting upstream of national policy.

Model homogeneity

• Torsten Heinrich, Juan Sabuco, J. Doyne Farmer, A simulation of the insurance industry: the problem of risk model homogeneity, Journal of Economic Interaction and Coordination 17(2), 2022, 535-576. Preprint. Too few distinct models raises default risk and lowers profits.
• Goodhart’s law, which in this setting is not an aphorism but a term in the dynamics.

The industry examples

• KFF on prior authorization determinations and denial rates by parent company, and on post-acute denial and appeal-overturn rates.
• The Senate Permanent Subcommittee on Investigations staff report on automated review in post-acute denials, and the 2026 follow-up letters asking three insurers about AI in denials.
• HHS-OIG on denial rates at the three largest Medicare Advantage organizations. Note the report does not name individual insurers; press coverage that does is going beyond the document.

← all posts