Beating Factorio is real progress, but games are the easy exam

Is an AI that launches a rocket in Factorio ready to run your back office? Not yet, and the gap between those two questions is the most interesting part of this week's story.

The facts are short. Developer Ryan Madden reports that GPT-5.6 Sol can beat Factorio on peaceful mode. According to summaries of the writeup, the model ran inside Codex at a low thinking level and completed the run, launching a rocket after 13.5 hours of game time. A Python library enforced human-like restrictions such as player reach and fog of war. The post also drew attention on Hacker News.

The milestone is real, and the low thinking level is the quiet headline

Factorio is not a trivia quiz. You bootstrap an economy from nothing, and every early shortcut becomes a bottleneck hours later. Keeping a coherent plan over that many steps is exactly what agents have historically been bad at. Madden's earlier projects (a Claude Code attempt, a testbed called Faketorio) suggest this was a slow climb rather than a lucky shot, and finishing is a genuine marker of long-horizon competence.

What we find most striking is the "low thinking level" detail. It suggests the endurance came from the agent loop, tooling and persistence, not from burning maximum reasoning on every move. That is good news for anyone who cares about cost, and it is a more useful signal than the rocket itself.

The documentation already hints at where it breaks

Don't skip the caveat in the coverage: production charts reportedly show scaling failures hitting after blue science. In other words, the agent finished, but it did not glide. Its factory strained as complexity grew. That is where we would look first, because real projects are mostly the part after the easy scaling stops.

And then there is the "peaceful" label. It is worth noting that, per hosting guides, peaceful mode does not completely remove enemies; they still spawn but only attack when provoked. Even so, the setup deliberately removes the main source of sudden, run-ending pressure. That is a sensible choice for an experiment. It is also a reminder that this is a friendly exam.

Games flatter agents, so grade them where errors hurt

Our opinion: a game is the best-case environment for an agent. The rules are fixed and documented. The state is observable through an API. Feedback is immediate and unambiguous. A wrong move wastes time, not money, and you can usually recover. The Python layer that enforces "human-like" limits also does a lot of work: someone had to design the interface so the model could act at all.

Offices, codebases and inboxes have none of that. Instructions are ambiguous, systems change under you, and some mistakes are irreversible: a deleted record, a wrong payment, an email you cannot unsend. An agent that is brilliant when failure is free may behave very differently when it is not.

There is also a ranking worth keeping in view. One benchmark site places GPT-5.6 Sol 12th of 132 models for agentic tool use and computer tasks, with a score of 68.5/100. We would treat that number cautiously, since it comes from an undated page, but it fits the picture: strong, not unbeatable, and certainly not settled.

Our verdict

Celebrate the Factorio run as evidence that long-horizon agents are arriving. Then change the question. Stop asking "can it finish?" and start asking "what does it cost when it is wrong, and how often?" Good agent evaluations should include messy inputs, no reset button, penalties for destructive actions, and a measure of how gracefully the agent stops and asks for help.

If you are evaluating agents for real work, run them on a reversible slice first, put a human on anything irreversible, and judge the failure rate, not the demo. The rocket launch is a great trailer. The film is the Tuesday afternoon when something goes wrong and nobody is watching.