Methodology

Today we rank models by provisional quality scores from two open sources, LMArena and Epoch AI, with what 1,000 support replies cost beside each one. Our own tests will replace them. Money never enters the formula.

What the leaderboard shows today

The home page has a tab for each category. Each tab ranks models by their provisional score in that category, highest first. Models with no open score yet come after, newest first, with no rank.

• Every score is labelled “Provisional”, with its sources and the date of the newest data it uses.
• For text models, the second column is what 1,000 support replies cost: 1,500 tokens in and 400 tokens out per reply.
• The price shown is the model’s list price: what the vendor charges on its own API, the same figure our open data and monthly report use. We show a router’s price, such as OpenRouter’s, only when the vendor lists none. Each model’s page lists every provider we track, including hosts that charge less than the vendor.
• List prices we haven’t checked by hand are marked unverified. For our October report we checked 12 against the vendors’ own pricing pages.
• Image and video models show their release date instead of a price. The token prices we get for them cover text only, so they’d understate the real cost.

Quality scores: provisional for now

Until our own tests run, quality comes from two open datasets, both published under CC BY 4.0 and credited on the board with links.

• LMArena, from its leaderboard dataset on Hugging Face. It ranks models by people’s votes in blind side-by-side comparisons. We read its text (overall, creative writing, instruction following, multi-turn and coding), WebDev, text-to-image and text-to-video leaderboards.
• Epoch AI, from its Benchmarking Hub. We use the Epoch Capabilities Index, which combines many hard benchmarks, and skip the files Epoch takes from other sources, which keep their own licences.

How a score is worked out

• One scale. Each measure becomes 0 to 100 by place among the models we track: the weakest on that measure gets 0, the strongest 100, the rest in proportion. We use only the newest run of each measure, and a measure needs at least 3 of our models before it counts.
• Best setting. Where a source lists several versions of one model, such as effort levels or dated builds, we take the highest figure.
• Missing data. A model missing one input is scored on the rest, re-weighted, and its label lists the sources used.
• Relative, not absolute. A score says where a model stands among the ones we track, so it can move when we add or drop a model. In a small field, such as image models, scores spread out to the ends.

Provisional scores will be replaced, category by category, by results from our own test rig: ten everyday tasks, each output scored 1 to 5 by two people on correctness, usefulness and tone, with time and cost recorded. We’ll publish the prompts, dates, model versions and outputs.

Data we don’t show

We don’t show Artificial Analysis data, including the figures OpenRouter relays from it, because we don’t have a licence to republish it. We also leave out benchmarks whose terms forbid commercial use or that carry no licence we can rely on.

The eight categories

We score eight categories to start. We add a category only when we can test it.

How we’ll score: formulas and weights

These are the published rules. They run on test and price data only. The weights are a first version; we’ll tune them once real scores are in and log every change in the version history below.

The recommender

First it drops every model that fails your job, budget or hard limits, such as EU hosting. Then it scores the rest on quality, cost, speed and privacy, weighted by the priority you pick:

Three picks for every job

• Best overall: the highest recommender score among models that pass every filter.
• Best budget: the cheapest model whose quality clears our “passes the test” bar for that job.
• Best premium: the highest quality regardless of cost, shown with how much more it costs.

Rank it your way

You set the weights. Score = (wQ × quality + wP × price score + wS × speed score) ÷ (wQ + wP + wS). Price score is log-scaled across the models shown: the cheapest gets 100, the dearest 25. Speed score is linear across the models shown.

Value

Value score = quality − 12 × log10(cost per 1,000 tasks). The value line sorts models by cost, cheapest first, and keeps each one that scores higher on quality than every cheaper model.

Alternatives to a model

• Cheaper, similar quality: costs at least 30% less, with quality within 5 points.
• Better, similar price: quality at least 5 points higher, with cost within 20%.
• Open or private: open weights, a zero-retention option or an EU endpoint, ranked by value score.

We’ll tune these thresholds once real data is in.

Where AI fits

A language model may explain a ranking in plain words. It never produces one. Every rank comes from the formulas above.

Money never enters the formula

No scoring function reads whether a vendor pays us. When we add affiliate links or sponsors, that information will live in separate tables, and an automated test will fail if any scoring code imports them. We’ll audit this every quarter. See how we make money.

How often the data refreshes

• Every 6 hours (00:00, 06:00, 12:00 and 18:00 UTC) we fetch prices and model details from the OpenRouter models API and keep a copy of each response. Fifteen minutes later a separate job imports models.dev the same way.
• OpenRouter price and context changes apply automatically when they look normal. A price that drops to zero or moves more than 40% waits for a person to check it. New models and retirements always wait for a person.
• We store every models.dev response and flag model ids we don’t know yet, but don’t yet compare its prices automatically for changes.
• Twice a day (05:30 and 17:30 UTC) we import the latest LMArena and Epoch AI data, keep a copy of each file, and recompute the scores.
• If our newest data is more than 24 hours old, our health check flags it.
• Price history back to October 2025 comes from LiteLLM’s public price map (MIT licence), marked unverified.
• Planned: a person will review the top 50 or so models and every recommendation page each week.

Mistakes

If a number is wrong, tell us. We fix it and add a dated correction note. See the disclosure page for our corrections policy and right of reply.