An AI can ace every test you give it and still not understand the thing it is predicting. The way to tell the difference is to change something and see whether it copes. Each experiment below runs the same setup twice, changing exactly one thing, and puts both runs side by side.
Build a model that still gets the right answer after the world changes, not one that was only ever memorising how things used to go. Each experiment below scores how close we got.
Every number here comes out of sim/run_all.py. Nothing is drawn
to illustrate a point.
Two AIs learn to drive the same small city. One studied the sensible routes people actually take. The other studied nothing but aimless wandering, strictly worse study material. Now start closing streets and see who still gets home.
Get from the dot to the square. Closer counts, so we measure how many blocks away it ended up, not just whether it made it. Turn the detour rate up and watch the closed streets pile on in orange.
On one announced day the rule for who counts as “severe” is rewritten, so more people get counted. Not one person got sicker. It is lowering the pass mark and then announcing that more students are passing. Both AIs see the new, bigger number.
Notice the count jumped, and still say correctly that nothing happened to anybody. The dark line is the truth. Stay on it and you win; jump with the count and you have mistaken the measuring stick for the world.
Same business, same rule for when to step in: act if costs look too high. On the left the numbers are final the moment you see them. On the right they are not, bills keep trickling in for months, a big one lands three quarters late, and figures you already acted on get quietly rewritten behind you. That is how regulated data actually arrives.
Make the right call each quarter, judged against the numbers once they finally settle. Nothing about the business differs between the two panels, so any calls lost on the right were lost purely to bad timing.
python3 sim/run_all.py; the figures rebuild from its output.