What are the AI folks up to these days?

theahura

Jul 10, 2026

Note: we are hosting our second in person Agentics event (and our first bi-weekly) at The Brass Factory on July 22nd. We’ll have speakers, food, and hopefully some boardgames too. Sign up here: https://luma.com/agentics-qz0j

Do you ever look out at the beautiful Manhattan skyline, bagel in one hand, pizza slice in the other, and wonder “What are those guys who are doing the AI thing in the Bay up to these days?” They’re always saying all sorts of weird and crazy things on the bird app that no one else but the AI people use anymore.1 Probably worth checking in.

hmm…

Now, cards on the table, I generally think of myself as one of those people who is doing the AI thing, on account of being the founder of an AI company and having been in ml research for years before this2 and also writing copiously on the subject of AI things. But I mostly avoid the bird app, and I don’t live in SF, so sometimes I do get some FOMO.

I’d heard about the AI Engineering conference a few times — a few friends at Google were previous sponsors and told me it was a good place to catch up with other folks who were thinking about coding agents on the cutting edge. Applied to give a talk, got in on the online track, and the organizers very kindly sent over a few free tickets by mistake.

They also very kindly did not respond, so I assumed the extra tickets were fair game.

Just got back from the Bay, FOMO eradicated. Here are my thoughts.

At a high level, the talks were split 85-15. 85% of the people there were enterprise folks who were trying to learn how to AI good. These people were interested in attending and giving talks about how to leverage more AI in their respective organizations. The other 15% of the people there were really on the bleeding edge, people who had plumbed the depths of lights off software factories and came back to tell of Lovecraftian horrors lying beneath (hi Dex!). These guys mostly were looking for less AI. I was expecting way more of camp two than camp one, and was somewhat surprised at the skew in the other direction. Even in this highly selected environment — the ~7000 people most likely to attend an “ai engineering” conference — the vast majority of people don’t really know how to use AI effectively.

Every month the AI world is onto something new. January was takeoff, February was ‘what is a skill’, March was ‘everyone has to use Claude Code all the time’, April was get the ops team on board, May was tokenmaxxing.

Apparently, June was the month of skill sprawl. There were a dozen talks that were variants of “how do we manage the 1000 skills that people have created in our organization?” These were inevitably packed to the brim, which should give a sense of where most of the market is. Over at Nori we solved this problem back in December (see: noriskillsets.com, a team wide package manager built on our open source local skills manager client).

Many of these ‘skill management’ talks landed flat for me. I think having that many skills is a ‘code smell’. In my experience the agents are very good at figuring things out from first principles. Having a skill that 1) will drift, 2) may not actually add anything new, 3) needs to be managed is more likely to create problems rather than solve them. Bluntly, I think many of the teams who are grappling with hundreds of skill files in their organizations are dealing with an employee base that is happily having the coding agents generate skills for all sorts of things without rigorously figuring out whether they are useful. This gives the veneer of “AI enabled” work without actually being so. I wrote about some of this here:

Agentics: Your agent skills are all slop

theahura

·

Jan 18

Read full story

I was also disappointed at how many talks were about some form of self-learning that involved having agents read through transcripts and automatically create skills, so much so that I wrote about it. Anything that does not have a human in the loop for context management is going to quickly spiral out of control, which is why we end up with so many skills in the first place.

In my opinion, you cannot really rely on the agents to write good skills for you. The process needs to be intentional. There’s a particular format that we find works. We like to bundle sets of skills into specific groups instead of just having a big pile of them with a search server on top. Every prompt with an agent is like summoning an entity from the void. The particular skills that are bundled with the agent are a big part of eking out the highest quality results. Trying to slopify the agent brain will result in agents that produce slop. Don’t do it.

This was also the month of evals. Evals are something of an interesting beast; as with everything in AI, the term is wildly overloaded.

Some people (mostly researchers and model providers) use “evals” to mean “benchmarks that we can use to evaluate raw model performance, that the industry can hill-climb against to make models better overall.” SWE-Bench is an eval, and Humanity’s Last Exam is an eval, and ARC-AGI is an eval. These evals are artificial, designed to give researchers a metric to improve model training through simulated, verifiable tasks, and designed to give the industry a quick read on the relative capabilities of these models.

Other people (mostly folks in industry who are deploying these models) use “evals” to mean “regression tests that we can use to try and make production workflows as deterministic as possible.” Here, an eval is extremely task specific — things like “does the AI correctly label data” or “will it correctly respond to adversarial chat input in the customer service chat window?”

Even though the former use of the term is more relevant for folks on the cutting edge, I think the latter is what a lot of the larger enterprises are thinking about, and as a result were what much of the audience was thinking about.

It’s worth taking a step back and thinking about how AI is landing in different orgs. At tech companies and startups, folks primarily interact with AI as a developer tool. Cursor and, later, Claude Code are interfaces that a dev spends all day in. The average developer at these companies is, as a result fairly intimately aware of what AI can do, simply because they interact with it so much.

That is not how AI has landed in other industries like banks and retail. In organizations where engineering is not obviously a critical function — places where engineering is more a cost center than where the money gets made — the primary way AI gets used is as infrastructure. The most common shape is the ETL pipeline, a recurring process that Extracts data from some table, Transforms it somehow, and then Loads it into another table. These sorts of tasks are a dime a dozen at any large enterprise, and less tech-forward companies will pay a lot of money to external consultants to build these out. Traditionally, ETL pipelines are brittle and required a lot of fiddling with APIs and different fields and so on. LLMs, with their inherent flexibility, promised to massively simplify that effort.

New technology never gets adopted wholesale, especially among larger slow moving companies with a lot to lose. Instead, new technology comes in as a supplement or a replacement for something that already exists. That’s a large part of why AI landed in ETL world for so many companies — because that was the best fitting preexisting shape to slot it into. Unfortunately, if you’ve ever worked with ETLs, you’ll know that you can’t just, like, screw around with the data. These pipelines need to be consistent and auditable, and an AI agent is neither.

Which brings us back to evals.

In large tech companies, tokenmaxxing meant ‘give individual developers unlimited budgets for things like Claude Code’. Everywhere else, tokenmaxxing meant ‘let’s throw more tokens at these failing ETL pipelines to get them from 80% correct to 100% correct’. How do you do that with AI? Well, one naive way is to build another AI system that checks the output of the first. But what if that one fails? Well, you could build a third system that checks the output of the second.

Bro just one more AI bro this time it will be different this is Fable 5 bro this isnt Codex or GPT bro this is a different model this just try it I swear bro just one more AI it’ll work this time bro

You see the problem.

Now that tokenmaxxing is dead, some of these less tech enabled firms are limiting employees to ~$50-$100 token spend per month. At full unsubsidized price, that’s barely anything at all, but then again most of these teams haven’t exactly been using Claude Code or Codex anyway.

I mention all of this mostly because I find it interesting to see the gap between SF and everyone else. I’ve said in the past that SF is in a bubble, and that was very apparent during AIE. The evals track talks were all about researchers and labs trying to figure out better RL environments, but a lot of the people in the audience were like “that’s great but tell me how to make my ETLs not be bad.”

As an aside, I think there isn’t really a good way to make these things consistent / auditable, nor should you really try. The long tail is too long. Instead, have a human in the loop. AI that goes 95% of the way and has a human approve or not has, in my experience, been way more successful than any AI system that attempts to go 100% of the way.

The folks who are really on the cutting edge spent a lot of time talking about background agents / cloud agents, and the ten-million reasons why people should use them.

Now, I recognize that as the founder/ceo of a company that is selling a background agent stack, I am a bit biased here, but it seriously felt like a full half of the talks at AIE were shilling for our product. There were talks about why you need background agents for productivity, and talks about why you need background agents for security, and talks about why you need background agents for compliance. There were even talks about why you need background agents for AI enablement! I’d say a solid half of the security track, all of the sandbox track, and many many other talks (including keynotes) were about moving things from local machines to the cloud.

(This post is brought to you by Nori! When you think “background agent” or “cloud agent,” think Nori!)

If you are unfamiliar, the name basically tells you whatever you need to know: background agents / cloud agents are agents like Claude Code that run in the cloud, in the background.

Why might you want this? Well, when Claude Code is running locally on your computer, it is:

Stuck on your machine. If you close your laptop or walk away from it you cannot (easily) take the conversation with you on the go.

A massive security hole. Everything on your computer is fair game. There is a fundamental tension between usefulness and risk, and unfortunately besides a few very very stressed out CISOs people are not thinking about this trade off at all.

Unable to easily coordinate and collaborate. If you want to try and run multiple agents in the same code repository, you have to use tooling like git worktrees. If you run too many agents at once, they will happily eat all of your computer memory and kill the computer.

All of these problems are because the coding agents live on your computer, so…

Cloud agents neatly solve all three of these issues and a whole host of other things. The openclaw folks already showed how being able to access agents from anywhere was massively useful. Because they are centralized, you get massive leverage on security, observability, governance, and all those other lovely enterprise buzzwords. And cloud agents are a necessary step towards the ‘fully automated business’, because, by virtue of being not-on-your-computer, they can respond to events and timers and do things automatically.

It turns out this is an extremely powerful primit

[truncated for AI cost control]