Kimi K3.1 Pro's weights are out, and now its benchmark claims have to prove it

Is a model "open" if you can't check any of what its maker says about it? Until this week, Kimi K3.1 Pro was in that gray zone: a pile of impressive benchmark numbers with a promise of weights attached. As of November 4, the promise has been kept, and that changes what we are entitled to demand.

The weights are real, but they come with fine print

Per several of the trackers we consulted, Moonshot launched Kimi K3.1 Pro on October 23 at $2 per million input tokens and $10 per million output tokens, about a third less than Kimi K3. The weights then went up on Hugging Face, where Moonshot describes it as an open-weight, natively multimodal agentic model with a 1-million-token context window, post-trained from Kimi K3.

One write-up from OrcaRouter puts the release at 96 shards and 1.56 TB, under a custom, non-OSI license with a $20M line in it. We haven't seen the license text, so treat the details as secondhand. The headline is clear enough, though: this is open weights, not open source in the strict sense.

Leaderboards were always the weakest evidence

The numbers circulating were a mess. One site cites a score of 55 on Artificial Analysis and a #4 ranking. Codersera says the model costs $2.11 per task against $5.98 for Claude Opus 5.5. Another page says 46 on a "new index." A third repeats a 65.7% Terminal-Bench 4.0 figure. Some of these may be vendor claims, and the pages are undated, so we can't tell which snapshot is current.

That's our point. A benchmark number from the lab that trained the model is an advertisement. A number from an aggregator that may have been rounded, rescaled or copied is a rumor with a decimal point. Even the cost-per-task comparison depends on how many tokens a model burns thinking, and one source notes that K3.1 Pro's thinking can't be turned off. Cheap per token is not the same as cheap per solved problem.

We'd go further: the whole "is it as good as the best?" frame is shaky. As good at what? Terminal coding agents, long-context retrieval, multimodal reasoning and plain instruction-following can all rank models differently. A model can be fourth on one composite and still be the wrong choice for your workload.

Why the weights change the game

Weights give everyone else a hammer. Researchers can run their own evals on their own prompts, check for contamination, probe failure modes, quantize it, fine-tune it and compare outputs with the hosted API. If the self-hosted model behaves differently from what Moonshot's endpoint sells, people will notice quickly. That is real accountability, and closed frontier labs simply don't offer it.

The license matters too. A custom non-OSI license with a $20M line in it isn't necessarily a dealbreaker for most developers, but it means "open" comes with fine print. Anyone building a product on it should read the actual terms, not a blog summary of them.

There is also the practical reality of 1.56 TB. This isn't a model you run on a laptop; for most teams, "open weights" will probably mean renting hardware or using a third-party host. The benefit is optionality and auditability, not a home-lab revolution.

Our verdict

Don't take Moonshot's word for how good the model is, and don't take ours either. Over the next few weeks, watch for independent evaluations that publish their prompts, settings and failure cases. Give more weight to anything that tests agentic coding on real repositories and reports cost per solved task rather than per token.

If you are a developer, run your own bake-off on a representative slice of your work before you switch anything; the low API price makes that cheap to do. If the independent results come back close to the claims, the story won't be that China caught up, but that the evidence for it is finally public. If they don't, the weights will have done their job anyway: they let the rest of us find out.