2026-07-16 23:02 UTCIn-site rewrite5 min readUpdated: 2026-07-16 23:19 UTC

Kimi K3 beats GPT 5.6 Sol in agentic knowledge work

Artificial Analysis released AA-Briefcase agentic knowledge work benchmark results; Kimi K3 scores 1547 Elo, ranking first, surpassing GPT-5.6 Sol's 1495. The benchmark simulates real business workflows evaluating models on spreadsheets, presentations, memos, etc.

SourceHacker News AIAuthor: declanjackson

Artificial Analysis

All evaluations

AA-Briefcase: Agentic Knowledge Work Benchmark

A private evaluation developed by Artificial Analysis for frontier agentic capability in long-horizon knowledge work, testing agents on realistic business workflows that require deliverables such as spreadsheets, presentations, and memos.

AA-Briefcase Launch Article

AA-Briefcase Methodology

ArtificialAnalysis/AA-Briefcase-Lite

ArtificialAnalysis/Stirrup

AA-Briefcase evaluates models across four multi-week knowledge work projects, comprising thousands of input files and 91 tasks in total. Across the scenarios, models must complete realistic professional workflows in fields such as data science, product management, and corporate strategy. Each scenario is a multi-week workflow that the agent works through in sequence, each week holding several tasks. Every task is a deliverable graded against a rubric of checks. Although tasks within a scenario share files and context across weeks, models currently complete each task in an independent run, without carrying over their own prior submissions.

Each task is graded against three types of checks:

Rubric

Binary pass or fail per check

Did the model follow the task instructions, identify requirements hidden across source files, use the correct evidence, and reach the right conclusions?

Analytical Quality

Pairwise comparison

Compared against another model's submission, which deliverable is more thorough, analytically rigorous, and well-supported?

Presentation

Pairwise comparison

Compared against another model's submission, which one is more professionally presented?

A public fifth scenario has been released via Hugging Face as a representation of scenario structure, submission, and grading. This does not count toward official AA-Briefcase results, and is demonstrative only.

Results

AA-Briefcase Elo

AA-Briefcase is an agentic knowledge work benchmark developed by Artificial Analysis. AA-Briefcase Elo is a combined metric that aggregates rubric pass rate, analytical quality Elo and presentation Elo · Higher is better

Reasoning models are indicated by a lightbulb icon

AA-Briefcase Elo is a combined metric that aggregates analytical quality Elo, presentation Elo, and rubric pass rate, with rubric performance converted into Elo via synthetic head-to-head matches. Elo and 95% confidence interval bounds are clamped at 0.

Cost

AA-Briefcase Cost per Task

Mean cost (USD) per task to run AA-Briefcase, calculated from token usage and model pricing including representative cache hit rates

Reasoning models are indicated by a lightbulb icon

The total cost to run AA-Briefcase divided by the number of tasks (91 for full submission of tasks). Cost is calculated from token usage and model pricing, split across input, cache hit, cache write, reasoning, and answer token prices, including representative cache hit rates.

Example Task, Submissions, and Grading

Explore a representative AA-Briefcase week from the public Due Diligence scenario available via Hugging Face. The outputs and grading shown here illustrate what AA-Briefcase evaluates. Scores are shown for a representative model set. Submissions and verdicts in this representative scenario do not contribute to a model's AA-Briefcase Elo or other benchmark scores.

Model

market_overview.pdf

Open

market_overview.tex

Open

Score Comparisons

AA-Briefcase Elo vs. Artificial Analysis Intelligence Index

AA-Briefcase Elo · Artificial Analysis Intelligence Index

Most attractive quadrant

Artificial Analysis Intelligence Index v4.1 includes: GDPval-AA v2, 𝜏³-Banking, Terminal-Bench v2.1, SciCode, Humanity's Last Exam, GPQA Diamond, CritPt, AA-Omniscience, AA-LCR. See Intelligence Index methodology for further details, including a breakdown of each evaluation and how we run them.

File Type Results

AA-Briefcase performance broken out by the file type of the deliverable (Excel, PowerPoint, PDF, Word, Other).

AA-Briefcase Rubric Pass Rate by File Type (Normalized)

Rubric pass rate by deliverable file type · Scores are normalized per file type across all models tested, where green represents the highest score for that file type and red represents the lowest score for that file type

Reasoning models are indicated by a lightbulb icon

File types are categorized by the required submission format, with “Other” covering formats such as HTML and LaTeX.

The share of binary rubric checks the submission passed (passed checks divided by total checks), aggregated across all AA-Briefcase tasks. Rubric checks are pass/fail criteria covering whether the deliverable includes required content and cites sources correctly, and whether it resolves planted cross-source conflicts.

Token Usage

AA-Briefcase Output Tokens per Task

Mean reasoning and answer tokens consumed per AA-Briefcase task

Reasoning models are indicated by a lightbulb icon

The number of output tokens used to run the evaluation, including visible answer tokens and reasoning tokens where reported by reasoning models.

Speed

Time per Task

Wall-clock time (minutes) per task: answer and reasoning generation plus tool execution time · Lower is better

Reasoning models are indicated by a lightbulb icon

Estimated wall-clock time per task: the sum of answer and reasoning tokens per task divided by the model’s canonical answer output speed, plus mean tool execution time per task. Lower is better.

Turns

Mean Turns per Task

Average number of model turns per AA-Briefcase task · Lower is better

Reasoning models are indicated by a lightbulb icon

This chart shows the average number of turns the agent takes per task. It is a rough proxy for how many actions, tool calls, and iteration cycles an agent is using to complete benchmark tasks.

Tool Usage

Tool invocations issued by each agent during AA-Briefcase: counts by tool category, mean tool calls per turn, and source-pool exploration coverage.

AA-Briefcase Tool Calls Breakdown, Avg per Task

Average tool invocations per AA-Briefcase task, bucketed by intent

Reasoning models are indicated by a lightbulb icon

Agent tool calls are grouped into six categories: explore (navigating and searching the workspace), read (reading file contents), write (creating or editing files), compute (running code or calculations), view image (visual inspection of files), and other (anything else).

Model Size (Open Weights Models Only)

AA-Briefcase Elo vs. Total Parameters

AA-Briefcase Elo · Size in parameters (billions) · Open weights models only

Most attractive quadrant

The total number of trainable weights and biases in the model, expressed in billions. These parameters are learned during training and determine the model's ability to process and generate responses.

Score vs. Release Date

AA-Briefcase Elo vs. Release Date

AA-Briefcase Elo · Model release date

Most attractive region

Leaderboard

Creator

Name

Elo

Release Date

Anthropic

Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)1583-15 / +16Jun 2026

Kimi

Kimi K31547-13 / +14Jul 2026

OpenAI

GPT-5.6 Sol (max)1495-12 / +13Jul 2026

Anthropic

Claude Sonnet 5 (Adaptive Reasoning, Max Effort)1388-12 / +11Jun 2026

Anthropic

Claude Opus 4.8 (Adaptive Reasoning, Max Effort)1354-11 / +10May 2026

SpaceXAI

Grok 4.5 (high)1323-12 / +13Jul 2026

Anthropic

Claude Sonnet 5 (Adaptive Reasoning, Xhigh Effort)1298-11 / +12Jun 2026

Anthropic

Claude Opus 4.7 (Adaptive Reasoning, Max Effort)1285-10 / +10Apr 2026

Z AI

GLM-5.2 (max)1260-10 / +11Jun 2026

Anthropic

Claude Sonnet 5 (Adaptive Reasoning, High Effort)1199-11 / +11Jun 2026

OpenAI

GPT-5.5 (xhigh)1154-9 / +8Apr 2026

MiniMax

MiniMax-M31110-9 / +10Jun 2026

OpenAI

GPT-5.5 (high)1103-9 / +9Apr 2026

Anthropic

Claude Opus 4.7 (Non-reasoning, High Effort)1089-10 / +10Apr 2026

Anthropic

Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)1078-9 / +9Feb 2026

Anthropic

Claude Sonnet 5 (Adaptive Reasoning, Medium Effort)1059-11 / +11Jun 2026

OpenAI

GPT-5.5 (medium)1000-0 / +0Apr 2026

Z AI

GLM-5.1 (Reasoning)973-10 / +10Apr 2026

DeepSeek

DeepSeek V4 Pro (Reasoning, Max Effort)932-10 / +9Apr 2026

Anthropic

Claude Sonnet 5 (Adaptive Reasoning, Low Effort)926-10 / +10Jun 2026

Alibaba

Qwen3.7 Max908-9 / +10May 2026

Xiaomi

MiMo-V2.5-Pro873-10 / +9Apr 2026

NVIDIA

Nemotron 3 Ultra 550B A55B (Reasoning)870-10 / +10Jun 2026

Google

Gemini 3.5 Flash (medium)869-9 / +10May 2026

Google

Gemini 3.5 Flash (high)866-10 / +9May 2026

OpenAI

GPT-5.3 Codex (xhigh)864-10 / +10Feb 2026