Skip to content
AI News HubLIVE
More
Summary only

Are You Getting Your Money’s Worth From AI?

A CIO with a 170-person organization measured a 2.1x productivity gain from AI, but top users each spent $21,000 a month on tokens. The essay argues for rigorously quantifying AI's return instead of relying on anecdotes or gut feeling, and warns against metrics like lines of code or pull request counts.

SourceO'Reilly AI & ML RadarAuthor: James Shore
Are You Getting Your Money’s Worth From AI?
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • AI productivity gains come with steep token costs that may eventually be drawn from personnel budgets
  • A METR study found engineers believed AI made them 49% faster than it actually did
  • Lines of code and pull request counts are unreliable productivity measures, especially with AI
  • Measure specific AI approaches rather than teams or individuals to avoid Goodhart's Law

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Agents