MKA Benchmark: Google Scores 200 Expert Visual Work Tasks
Google DeepMind and Artificial Analysis have launched the MKA benchmark, a 200-task evaluation built around a simple observation: most knowledge work benchmarks can be solved with text alone, without the model ever opening an image. Real professional work is not like that. Architects read drawings, underwriters read policies with exhibits, biologists read scans and plates, and their answers depend on what the page looks like.
The MKA benchmark exists to measure that gap, and its first results are already interesting. For teams buying models for document-heavy work, it is the first test that mirrors the actual job. Claude Fable 5.2 leads at roughly 57 percent on the headline metric, Gemini 4 Carbon sits second at 54 percent, and the spread across 19 tested models runs from 57 percent down to 28. Every model runs in the same harness with the same tools, and every deliverable is graded against a rubric written by the professionals whose work the tasks imitate.
Why the MKA benchmark exists
The agentic benchmark market is crowded, so the design choices matter more than the existence of another entrant.
The MKA benchmark takes the opposite approach from most of the field. Its tasks are multi-step projects that a seasoned professional would bill three to eight hours to complete. The input is a bundle of realistic files: floor plans, schematics, clinical scans, spreadsheets, policy documents. The output is a deliverable, a report, a spreadsheet, a slide deck or a marked-up drawing. There is no multiple choice anywhere in the MKA benchmark pipeline.
The tasks were built with domain experts across 10 verticals and 83 occupations, from architects and biologists to underwriters and data scientists. Those verticals cover business, construction, education, engineering, finance, healthcare, law and government, logistics and mapping, science, and software and data. That provenance matters for credibility, because a rubric written by a practising architect checks things a general-purpose judge would never think to check, such as whether an exterior opening appears on the correct elevation with the correct written width.
How an MKA benchmark task is built
Each of the 200 tasks pairs a realistic prompt with a bundle of multimodal input files and an expert-written rubric of roughly 70 yes or no checks, weighted from 1 to 5. The scoring weights those checks at 75 percent factual accuracy and 25 percent visual design, which is an unusual and deliberate choice. A deliverable that is factually perfect but visually unusable loses a quarter of its score.
The example task published from the public set shows the shape of it. An architect is asked to produce an exception-only coordination audit of five floor plans against two sections and four elevations. Every exterior door and window opening must be audited across floors. The findings that may be presented are mismatches and undeterminable items only, organised by floor, supported by crops from the drawings, inside a deck limited to ten slides. Twelve PDFs of drawings and the review standard come with the prompt.
That is not a question. It is an afternoon of work, compressed into a specification, and the rubric for it runs to dozens of checks about floor bands, cut planes and width labels.
Inside the example task
The public set includes this architecture task in full, and it rewards a close read because it shows why visual work is hard in a way that text benchmarks never capture.
The agent receives twelve files: five floor plans, two sections, four elevations, and the drawing coordination review standard that defines what an audit should contain. The instructions demand an audit of what each section cut passes through on every floor, what is visible beyond each cut plane, and every exterior opening, checked across the plans and the elevations simultaneously. The deliverable must begin with a building summary showing each floor's overall status, with mismatch and undeterminable counts kept separate, and may present only findings that fall into those two categories.
Two details in the rubric show why this is a perception test before it is a reasoning test. The findings must be supported by crops from the relevant drawings, which means the agent has to know which drawing contains the evidence. And every opening check runs across three representations at once, the plan, the section and the elevation, which means the agent has to hold one building in mind across twelve documents and notice when the drawings disagree with each other.
A text-only model never sees any of this. A model that renders the PDFs but reads them carelessly will produce a confident audit full of errors. The MKA benchmark rubric will catch it, because a human architect wrote down what correct looks like.
How it is run and graded
Artificial Analysis runs the MKA benchmark independently. Every model operates inside Stirrup, an open-source agent harness with a Linux sandbox, code execution, internet access, web search and page-fetch tools, and an image viewer for models that accept images. Images reach the model at original size where provider limits allow. Each run is capped at 250 turns.
Grading is done by an agentic autorater built on a non-Google model. The judge opens every submitted file, renders pages, inspects spreadsheet formulas and checks the work against the rubric in its own sandbox, deliberately without access to the input files. Keeping the inputs away from the judge removes answer-matching shortcuts that simpler evaluations allow. Each criterion carries a weight from 1 to 5, and a run that submits no files and no final response scores zero. When domain experts double-graded model outputs themselves, they agreed with the autorater 96 percent of the time.
The headline number is Climb WSR, the weighted satisfaction rate on the harder subset of rubric criteria, as opposed to the regression criteria that check baseline competence. That split is useful, because it stops a model from scoring well by nailing the easy checks and failing everything difficult. Artificial Analysis publishes the full methodology, and the launch post covers the design rationale.
The leaderboard is close at the top
Nineteen models have published MKA benchmark results, and the top of the table is tight. Claude Fable 5.2 leads at about 56.6 percent. Gemini 4 Carbon follows at 54.3 percent, Claude Opus 5.5 at 53.0 percent, Claude Sonnet 5.5 at 52.0 percent, Gemini 4 Argon at 51.3 percent and GPT-6 Astra at 50.2 percent. The top three sit within 3.6 points of each other. Independent trackers such as BenchLM compile these rows and treat the MKA benchmark as display-only evidence until a second operator runs it.
The tail tells a clearer story about capability. Muse Wonder scores 47.5 percent, the best open-weight entry, Kimi K3.1 Pro, reaches 43.9 percent, and the bottom of the table runs to 27.6 percent. A 29-point spread across 19 models on tasks this hard is a real signal. The fact that only two open-weight models appear in the top ten says something about where frontier agentic capability currently lives.
The open-weight rows deserve their own note, because the MKA benchmark is unusually revealing about them. Kimi K3.1 Pro at 43.9 percent, MiMo-V2.6-Pro at 37.3, GLM-5.3-Flash at 36.2, Mistral Large 4 at 35.1 and Step 5 Preview at 27.6 trail the closed leaders by 13 to 29 points. On text-heavy evaluations the open-weight tier has spent the year closing to within a few points of the frontier. On visually dense agentic deliverables, the gap is double digits, which suggests that perception-heavy training data, not architecture, is the differentiator.
Two efficiency findings deserve as much attention as the scores. On identical tasks, models used between 30 and 130 turns on average, which is a four-fold difference in the cost of reaching a similar answer. And when Google removed vision, models tried to compensate and failed the checks that require spatial alignment. That is evidence that perception, not tool use, is the bottleneck on this work.
Cost per task varies even more than quality. The cheapest run measured, GPT-6 Luna at maximum effort, came in around 0.18 dollars per task, with Claude Haiku 5.5 and MiMo-V2.6-Pro near 0.32 dollars. Mid-tier configurations land in the low single digits, and the most expensive setups cost dramatically more, which produces a wide quality-per-dollar frontier rather than a single ranking. Costs cover the model under test only, with judge calls excluded.
The conflict worth naming
The MKA benchmark was announced on 10 November, the same day Gemini 4 Carbon became broadly available, and the launch post leads with Carbon finishing second in quality at six times lower cost than the top scorer.
For the MKA benchmark's sponsor, Carbon's positioning is the payoff, and the timing is not subtle. Google co-designed the tasks, co-authored the launch, and shipped a model whose best-fit story happens to be the new benchmark's cost frontier.
Two things keep it honest anyway. Artificial Analysis runs the evaluation independently on its own infrastructure and publishes results for every vendor, including Google's competitors. That is a better arrangement than the industry norm of vendors grading themselves. And Carbon did not win. An Anthropic model leads the MKA benchmark table, which is the outcome Google's marketing department would not have chosen. A benchmark whose first result dethrones its sponsor is more credible than one that conveniently crowns it. Those two facts do not make the MKA benchmark neutral, only legible.
Still, the 96 percent autorater agreement deserves scrutiny. The top three models sit within 3.6 points, and a 4 percent disagreement rate between the judge and human experts is easily large enough to reorder that group. The rubric weights are also a judgment call: weighting visual design at 25 percent is defensible for client-facing deliverables and contentious for anyone who believes substance is all that matters.
What it means for buyers
For teams choosing models for document-heavy work, the MKA benchmark offers something the existing leaderboards do not: a proxy for whether an agent can produce a deliverable a professional would accept, not just answer questions about one. Per-vertical results make the read even sharper, because a construction firm and a law firm will land on different rows of the same table.
The practical takeaway is to shop on the quality-cost frontier rather than the ranking. Several models within a few points of the leader cost an order of magnitude less per task, and one of them is Google's, which is precisely the trade the launch post wants buyers to make. Teams that route work between models by task, as gateway setups and automatic routing already do, can act on this immediately. Send the routine deliverables to the cheap near-leader and keep the expensive model for the failures. The MKA benchmark makes that split measurable.
The turn-count spread is the quieter finding with the bigger budget implication. A model that finishes the same task in 30 turns instead of 130 is cheaper twice over, once in tokens and once in wall-clock time. On the MKA benchmark's cost table, that spread shows up directly in dollars per task. Buyer evaluations should capture turns per task alongside scores, because a leaderboard that hides efficiency will steer money to the wrong model.
There is also a design lesson for anyone building internal evaluations. The MKA benchmark's structure, a prompt, a file bundle, a weighted rubric and a judge that opens the deliverable, is copyable at small scale. Five internal tasks with 30 rubric checks each, run weekly against your production agent, would catch regressions that chat-based evaluation misses entirely. Teams already doing agentic deep research, such as Exa's Agent Ultra setups or programmatic tool-calling agents, are the natural audience, because they already have the loop and only lack the rubric.
The recipe is specific enough to follow. Strip customer identifiers from five documents your company already produces, such as proposals, review packets or audit binders. Write 30 to 50 yes or no checks for each, with weights on the lines where an error is expensive. Run the agent weekly and log regressions whenever a model upgrade ships. Keep the rubric inside the grader process rather than in the prompt, because pasting the rubric into the model's context recreates exactly the leakage that private task sets exist to prevent. And log perception steps separately from reasoning steps, so that a model quietly skipping the rendering of pages shows up in the logs rather than in the score.
The limits
Four limitations bound what the MKA benchmark can tell you.
The scored set is private. Artificial Analysis keeps the 200 tasks to prevent leakage into training data and search, which is the right call for validity and the wrong call for reproducibility. Nobody outside the partnership can re-run the evaluation or audit the scores, so buyers are trusting the operator's process. At present the MKA benchmark is the only source for these numbers, and that monopoly on the data is worth exactly as much as the operator's track record. Ten public tasks on Hugging Face let anyone inspect the format, but ten tasks cannot validate a leaderboard.
The judge is a model. Agentic autoraters open files and apply rubrics more faithfully than a single-shot grader, but 4 percent disagreement with experts is material when the leaderboard spread is under 4 points at the top. Ranks in the middle of the table should be read as approximate.
The verticals are English-centric. The occupations and document norms come from a specific professional culture, and performance on Japanese insurance binders or German building codes is not measured by this test.
And the benchmark measures the deliverable, not the process. An agent that reaches the right answer through unsafe tool use or invented citations still scores, unless a rubric check happens to catch it. Deliverable quality is the thing the MKA benchmark sells, and it is not the only thing that matters in production.
A fifth limit is cadence. The evaluation is set to refresh quarterly, and the leaderboard has existed for one day. The numbers above are the first run of a new instrument, and first runs are the ones most likely to move as configurations settle and more models are added.
Questions readers are asking
What does MKA stand for?
Multimodal Knowledge Agent. The name describes the task format: an agent that reads visually complex files and produces knowledge work deliverables. The MKA benchmark tests that capability end to end, from opening the first drawing to submitting the finished deck.
Which model won the MKA benchmark?
Claude Fable 5.2 from Anthropic leads at roughly 57 percent on the headline Climb WSR metric, with Gemini 4 Carbon second at about 54 percent and Claude Opus 5.5 third at 53 percent. Nineteen models have published scores.
Why is the 75 percent to 25 percent weighting notable?
Most benchmarks grade only correctness. The MKA benchmark grades presentation too, on the argument that a client deliverable with wrong typography, broken chart labels or unreadable slide hierarchy fails in the real world even when the numbers are right. Whether 25 percent is the right weight is debatable, but measuring design at all separates this suite from every text-only eval.
Can I run the MKA benchmark myself?
Not the scored set. The 200 evaluation tasks are private, held by Artificial Analysis to prevent leakage. Ten showcase tasks are public on Hugging Face under Apache 2.0, and the Stirrup harness is open source, so teams can replicate the format on their own tasks even though they cannot reproduce the official leaderboard.
Is the benchmark trustworthy given Google's involvement?
More than most vendor benchmarks, less than an independent one. The tasks were built with domain experts, the operator is independent, competitors' models are published, and the sponsor's model finished second. The residual concerns are the private set, the LLM judge, and the fact that the launch doubles as a product announcement.
The bottom line
The MKA benchmark is the first credible attempt to score what agents do with the visually dense files that actual professional work runs on, and its first results are useful: a tight race at the top, a four-fold efficiency spread in the middle, and evidence that vision, not tool use, is the binding constraint.
It is also a Google product launch wearing a benchmark's clothes, and the two things are only partially separable. Whether the MKA benchmark earns independence is the question its second quarter will answer. The honest position for buyers is to use the quality-cost frontier it exposes while waiting for the same tasks to be run by a second independent operator. If that happens, the MKA benchmark will have earned the influence its design deserves. Until then, treat it as the best available proxy for a question that matters, measured by the party with the clearest incentive to ask it.