Beyond grep: The case for a context-rich AI coding harness
In a discussion, Perneti and Wu agree on the continued exponential improvement of AI models but disagree on context assembly for coding tools. They highlight the potential shift toward smaller models as costs rise.
Ars: Thank you. I appreciate you making the time.
Perneti: Yeah, I really enjoyed the chat, Sam.
Takeaways
Both Wu and Perneti agree that they expect frontier models to continue to improve at or even above the current exponential, at least for the next year. (Neither explicitly addressed a longer time frame.) Both agreed that this is fundamentally changing the way software development is done for many organizations.
The practical dispute, such as there is one, is whether context should be assembled ahead of time or rediscovered during each task—especially with large, private codebases.
They’re also sometimes measuring and optimizing for different things. Wu specifically said that Anthropic did not find “a measurable improvement in performance” from “a few LSPs available.” It’s important to note that what Wu was describing when she said that (an LSP, or language server protocol) is not as broad as what Augment Code’s harness does.
When Perneti and Wu describe evaluations of tools or systems that go beyond grep, they are not always talking about the same thing. That is one possible reason (among others) why they come to different conclusions.
One thing Perneti was able to speak to that Wu did not is the possibility that frontier models like Anthropic’s Opus or Fable may become too expensive to remain the main or only models in developers’ and organizations’ agentic workflows, such that smaller or open-weight models might become more widely used.
As the compute crunch continues, some open-weight models—including those small enough and sufficiently quantized to run on local hardware, or at least on hardware maintained in-house by the organization—now approach relatively recent frontier performance on some coding tasks. It remains possible that they, too, will continue to improve at a rapid rate, such that they become viable for coding agents in many teams and projects. Teams may increasingly reserve expensive frontier models for the hardest problems while routing more routine, well-specified work to these cheaper or locally operated models.
Regardless of which approach is more dominant a year from now, more autonomous workflows do not eliminate the need for engineering judgment. If anything, it may be more valuable, not less, as the increases to the speed and scale of production make bad decisions carry higher costs.