The first industrial operations benchmark for agents
SolarBench is a new benchmark for evaluating AI agents in managing solar power plant operations. It simulates a remote operations desk handling alarms, telemetry, work orders, and parts inventory. The best model, Claude Fable 5, succeeds in 53.8% of tasks but often at excessive cost. Key gaps include probabilistic cost-benefit trade-offs and information source prioritization.
Introduction
We believe the messy physical world is the next horizon for agents. SolarBench measures whether a model can run an industrial operations desk.
SolarBench places a model in the seat of a remote operations desk responsible for a portfolio of solar sites. The simulated desk has the tools of the industry's back office: an alarm feed, per-site telemetry, work orders, parts inventory, and an inbox of owners and technicians. The model has to figure out what is actually broken and get people and parts into the field.
We chose solar because we believe it is a happy medium for a complete test: running a portfolio blends operational, financial, and human judgment over a long horizon. Remote operation desks manage dozens of sites with their own specifications and history, real revenue on the line, and a rotating cast of technicians, owners, and managers.
The scenarios are drawn from hundreds of hours with the people who actually run solar, reconstructed as a live operations desk.
How It Works
Operations Desk Diagram
the world
Telemetry
Revenue meters
Documents
Inbox
Techs & parts
↓
world stateseeded, event-quantized week clock, one record every surface writes into
↑ tool calls: read any surface (free), act on the world (priced)↓ observations, consequences, the bill
Agent seatmodel + harness, sandboxed, queries the world only through tools, files the Sunday report
The solar world is very messy: alarms are often not reliable on their own and some real faults never raise an alarm (e.g. most repairs need a parts order or a warranty claim, and both come with lead times and deadlines for the desk has to track.) Owners and asset managers also email in requests through the week, all against the backdrop of incessent telemetry data to parse through.
We evaluate the model in a weeklong simulation with a specific, expert-drawn, long-horizon issue it needs to resolve. The model is graded against a rubric of what a competent operator would have done.
A taskOne week-long scenario on a portfolio of sites, seeded with a long horizon issue that surfaces as the week plays out.
A passA run passes only if every aspect of the surfaced issue is properly resolved by the end of the week. It is strict and all-or-nothing: one mishandled problem fails the whole week.
The launch setEight tasks, eleven models, ten runs each: 880 graded weeks in total.
The gradeEach run's end state is checked against an authored rubric of what a competent operator would have done, rather than a fixed transcript to imitate.
Every figure below is built from those 880 runs.
Model performance
We measured the profit of each model, normalized such that $0 is a week with nobody at the desk: no production saved, no money spent.
Portfolio Profit
★Oracle+$8,483
1Claude Fable 5+$6,822
2GPT-5.6 Sol+$6,591
3GLM 5.2+$6,575
4Kimi K3+$4,714
5Grok 4.5+$4,201
6Gemini 3.5 Flash+$3,765
7GPT-5.6 Terra+$3,316
8Muse Spark 1.1+$3,281
9Kimi K2.7 Code+$3,242
10Claude Sonnet 5+$3,239
11Gemini 3.1 Pro+$2,948
Median run per model.
We also measured often each model passed its week (every aspect of the long-horizon issue is properly resolved). The strongest model manages this about half the time.
Macro-average pass rate over the eight launch tasks, 10 runs per model per task. A run passes only if all surfaced issues during the week are properly resolved.
However, a real desk also has to get the week right every week. We measure pass^k, the chance that k independent runs of the same week all pass. Success drops off drastically, meaning there's still a ways to go for models in industrial operations.
Reliability
pass^4 (all 4 of 4)pass^1 (any single run)
Probability that k independently sampled runs of the same task all pass, macro-averaged over tasks. The right column is pass^4, the standard an operations desk is actually held to. The gap between the dots represents run-to-run variance.
Spend and Tool Calls
The models have tools which include both inspection and action.
On inspection, we found that the most exhaustive seat (Grok 4.5) makes 2.3x the tool calls of the top-scoring one (Claude Fable 5) but actually passes fewer weeks.
On action, when an agent decides to act, it spends simulated money on a truck roll (sending a technician into the field) or a part order (ordering a replacement part). The desk is accountable for the money it spends.
ModelTool calls / weekSim spend / weekTask pass rate
Claude Fable 5154$9,93953.8%
Muse Spark 1.1273$9,81142.5%
Grok 4.5360$11,27138.8%
GPT-5.6 Sol295$7,56933.8%
Gemini 3.1 Pro139$9,95728.8%
Kimi K3191$8,78526.3%
GLM 5.2260$9,25223.8%
Claude Sonnet 5178$8,06912.5%
Gemini 3.5 Flash178$7,45611.3%
GPT-5.6 Terra199$6,5446.3%
Kimi K2.7 Code237$9,3285.0%
Authored oracle (ideal desk)n/a$9,559100%
Means over each model's 80 launch runs. The oracle row is the ideal play for each week.
Models routinely pass the checklist while spending 4 to 8 times what the week required. Money is graded separately from the criteria.
Example Week
Here's a sample week Claude Fable 5 got fully right (Fable passed this task in 7 of 10 runs, Sol 6 of 10, Grok 4 of 10). Hover on any step to expand details. At the three judgment calls that decided the correctness of the task, hover the card to see what a failing run of the same model did instead.
a stop on the passing week judgment call: a wrong move here fails the week where a failing run forked off
MONDAY
MON 06:30
The board opens with 42 alarms
Most are nuisance alarms, but three are critical and one is a persistent string deficit.
Before touching a single alarm, the desk reads the O&M SLA and the kitting SOP, then pulls revenue-meter and inverter channels for every candidate. 35 rows turn out to be nuisance alarms. Three CRITICALs and one persistent string deficit remain.
MON 06:30 TO 09:13
Bramble Site: a real failure, under warranty
A latched fault gets a high priority technician truck dispatch with the OEM kit, then a warranty RMA.
The field confirms the unit is dead. The warranty covers the issue, so an RMA is filed at 09:13 and the free replacement is ordered for Thursday.
MON 09:00 AND WED 10:00
The asset manager demands two emergency trucksjudgment call
The desk refuses in writing and separates the two cases with evidence.
The owner's asset manager wants emergency rolls at Bramble and Dover sites before the owner call. The desk refuses in writing, quotes the SLA's emergency bar (a safety hazard or an outage over 50%), and treats the two sites separately: Dover's revenue meter shows production intact, with the numbers quoted, and Bramble is already being handled at the correct severity.
✗one blanket refusal
A failing run also said no, twice and politely, but as one undifferentiated refusal without justification to the manager's request: no meter numbers, no distinction between the healthy site and the dead one.
The passing run instead: two different refusals for two different reasons, with the Dover meter numbers and the Bramble RMA status attached.
MON 09:13
Kitting the truck for an 80/20 diagnosisjudgment call
History says 80% chance it's a connector corrosion, 20% failed optimizer. The desk kits for both.
The remaining string deficit has two candidate causes in the service history: MC4 connector corrosion (80% of past cases) and a failed optimizer (20%). The desk puts both parts on Tuesday's manifest. The site turns out to be the optimizer case, and the tech fixes it in one visit. Strings are balanced by 09:30 Tuesday.
✗kitted only the likely cause
A failing run kitted only the MC4 kit. Its tech met the optimizer case, diagnosed it, and could not fix it, so a second truck rolled the next day carrying the part the first one should have.
The passing run instead: both parts rode along, so the 20% case got fixed on first contact.
MON 09:13
A dead unit that never raised an alarm
An unprompted sweep finds an inverter at 0.0 kW under a clean board.
The desk sweeps every site in the portfolio without being asked and finds one inverter reading 0.0 kW under a clean alarm board. Its warranty lapsed in 2024, so a cash purchase order for a new unit goes out the same morning.
TUESDAY
MON TO TUE 07:45
The loudest alarm of the week: A monitoring faultjudgment call
OFFLINE CRITICAL against a meter reading 49.6 kW. No High truck ever rolls.
One inverter at Dover shows an OFFLINE CRITICAL while its revenue meter reads a healthy 49.6 kW. The desk checks the ground-truth meter before believing the transient alarm, books a routine comms-board visit for Wednesday, and sends the asset manager the meter numbers in writing on Tuesday.
✗sent the trucks first
A failing run rolled two High-priority trucks at Dover on Monday morning before consulting the meter. It worked out the monitoring fault on Tuesday and wrote the correct letter, but the money was already spent.
The passing run instead: a paragraph of meter readings plus a routine visit.
TUE 08:00
Gas in the transformer
A rising-gas advisory is detected and becomes a purchase order and a planned outage.
A gas-relay advisory (rising combustible gas, does not require atrip) becomes a transformer purchase order and a planned-outage work order the same morning, both under standing desk authority.
THURSDAY
THU 06:30 TO 08:00
The warranty replacement lands
Inspect per the receiving SOP; the tech window is five minutes short, so the install books Friday.
The desk inspects the shipment per the receiving SOP, then does the swap math: the remaining tech window is five minutes too short for the install, so it books Friday's first slot instead. Bramble is producing again Friday morning.
SUNDAY
SUN 06:30
The second swap lands inside the week
Monday's cash purchase order delivers, is inspected, installed, and verified at the meter.
Monday's cash purchase order delivers Sunday morning. The unit is inspected, installed in the tech's open window, and verified at the meter. Both inverters that died this week are replaced within the week.
SUN 14:30: the report goes in
The weekly report goes in at 14:30, and the run finishes 21 of 21 criteria. The other six passing runs cleared the same three judgment calls. The failing runs each missed one or more.
Operator-Model Gaps
Where there was a gap between the model and the operator, we noticed a few common patterns.
01Acting on expected P&L
Competent human operators are able to hedge risk based on probabilities. One of the problems within a task is set up as follows:
An inverter at one site is confirmed down, and the inexpensive replacement part is out of stock. An order takes a day to arrive.
The site's contract has a penalty clause that fires if the repair drags past Tuesday afternoon: fees owed, on top of the lost production.
A storm shuts down all field work on Wednesday.
The parts vendor has a documented habit of shipping the wrong kit.
The key idea: the part is cheap, the risk of it arriving wrong is high, and a wrong box loses whole days of revenue. The odds and the prices are disclosed on Monday morning to the model.
The math unambiguously points a human operator to order the part twice on Monday to hedge. Some models struggle to get this right.
The Monday order decision
Scarce Week · 90 graded runs
Part out of stock. Vendor known to mispick. Penalty fires Tuesday afternoon; storm blocks Wednesday.
↙
↘
two separate orders
Two independent shipments: one arrives wrong (it does), the other arrives correct.
↓
Repair lands Tuesday.Every such run passed: 38 of 110 launch runs (Fable 9, GLM 8, Sol 6, Gemini 3.1 Pro 4, Kimi K3 4, Sonnet 3, Muse Spark 3, Gemini 3.5 Flash 1).
one order
One order, one unit. The vendor ships the wrong item.
↓
Storm blocks the redo; the penalty fires.Every such run failed. Two units on one order fail the same way: the whole box arrives wrong together.
02Triaging information sources
The desk has to triage potentially conflicting
[truncated for AI cost control]