Agentic AI overwhelmed CI, and test selection cut queueing from hours to minutes

Frederik Dudzik Aug 21, 2026 The last time I had to think seriously about CI throughput, I was working on the CI/CD team at Shopify. I didn’t expect to run into the same kind of scaling problem on a hobby project with one engineer. But coding agents helped push one of my projects to roughly 500,000 lines of code, and I hit it much sooner than I expected. Changes were arriving faster than CI could verify them. They also tended to be larger, and several could be ready during the roughly 20 minutes it took CI to verify one of them. Once one landed, the others often had to be rebased and run through CI again. Waiting created overlapping work, overlapping work created rebases, and those rebases created more CI work. One engineer with coding agents was enough to make CI a bottleneck I had previously associated with much larger engineering organizations. I addressed it by reducing how much CI each change triggered. By selecting only the tests a change could affect, many changes went from occupying the runner for around 20 minutes to taking a couple of minutes or skipping the suite entirely. Why more CI capacity wasn’t the answer At first, I paid for more CI capacity. I burned through GitHub Actions’ free minutes quickly, and the bill went from about $10 to $20 to $50 a month. It was still growing. I didn’t want CI costs to scale with the amount of work the agents produced. I moved the workload to a dedicated runner on a generic VM provider and deliberately limited myself to one machine. One VM had enough total compute. A 20-minute suite was manageable overnight, when agents could take multiple turns without me waiting for each result. The problem was the 20-minute feedback loop during active development. I could have added more runners, but I’d seen that approach at much greater scale at Shopify. More runners increase throughput, but every change still triggers the same amount of work. As the rate of changes grows, the required capacity grows with it. I decided to reduce the work each change triggered instead. Selecting only the tests a change can affect The approach was one we had also used at Shopify: instead of running the entire test suite for every change, figure out which tests the change could affect and run only those. I combine two pieces of information. Full test runs record which parts of the application each test uses. When a change enters CI, a TypeScript dependency graph shows which parts of the application the changed files can affect. CI combines those two sets of information to select the relevant tests. Most of that work does not happen on every CI run. Scheduled full-suite runs collect the runtime information, while the TypeScript dependency graph takes only a few seconds to rebuild. A localized change may select only a handful of tests. A broadly shared change can still select most or all of the suite. At Shopify, much of the codebase was Ruby, so we relied more heavily on runtime tracing to understand what each test touched. At that scale, collecting and processing those traces was expensive enough that we spent significant effort making it faster. TypeScript makes part of this much cheaper because the compiler can tell me how source files depend on one another without running the application. The selection is deliberately conservative. If CI sees a changed source file that it cannot confidently connect to existing test coverage, it records that uncertainty instead of assuming the change is safe. I still run the complete suite every night. Those runs also refresh the runtime information used for future test selections. This keeps the development loop fast while the nightly run provides broader coverage without blocking me. Test selection does not guarantee that a skipped test could never fail. It avoids spending 20 minutes rerunning unrelated tests when CI has enough information to select a much smaller set. Making test selection useful Test selection only works well if the codebase has useful boundaries. If every part of the application depends on everything else, an accurate selector will still conclude that a small change can affect most of the test suite. In this application, each page has a route file, and each group of backend operations has an API file. Larger parts of the application are also separated into packages. These boundaries give the selector useful connection points between a source change and the tests that cover it. Some changes naturally cross those boundaries. Shared test setup, global styles, build configuration, and dependency changes can affect much of the application, so those changes still select most or all of the suite. The selector also exposed weak boundaries in the codebase. If a small change consistently selected a large part of the test suite, too much code usually depended on the area being changed. Tightening those boundaries reduced the number of selected tests. It also helped with concurrent agent work because changes in separate parts of the system were less likely to overlap and require rebasing. I also found work that looked parallel in the CI configuration but was serial in practice. Two E2E jobs competed for the same runner, so they repeated setup without reducing wall time. I combined them into one job that builds the application and prepares a local database once. Within that job, state-changing tests remain serial, while isolated visual tests can run at the same time. Now the order of operations is: skip tests that cannot be relevant, make the remaining work cheaper, improve code boundaries when small changes still select too much, and only then consider adding runners. The aim is for the cost of verification to match the scope of the change. What changed in practice The changes helped even in the worst case, when the selector still chose the full pull-request suite. Metric Before After Reduction --------------------------------------------------------- Total CI wall time ~21 min ~13 min ~40% E2E lane ~12 min ~4 min ~65% Browser-test execution ~10 min <3 min ~70% Critical-path contention ~14 min ~6 min ~60% The selector itself adds about 15 seconds. The slowest runs improved much more. Across successful pull-request runs that started E2E, p95 E2E time fell from about 23 minutes to 6 minutes, a 74 percent reduction. Including queue time, p95 total CI time fell from about 7 hours 35 minutes to 35 minutes, a 92 percent reduction. Why contention grows nonlinearly. As a single runner approaches full utilization, small increases in incoming work can cause disproportionately large increases in queue time. Illustrative queueing behavior, not measured project data. With a fixed arrival rate, small reductions in test work can produce much larger reductions in queue time when the runner is close to full utilization. The 92 percent reduction mostly came from queueing, not faster tests. Some pre-change runs spent hours waiting for the single runner. Near capacity, queueing is nonlinear: a modest reduction in the work each change requires can free enough runner capacity to reduce wait times by much more than the test-time improvement itself. These are historical before-and-after measurements, not a controlled benchmark. They show two different effects: less unnecessary work improved typical runs, while less runner contention had a much larger effect on the slowest runs. What I’d do next At this point, one runner is enough again. Most pull requests only pay for the tests they can plausibly affect, while the full suite runs overnight. If the workload eventually exceeds one machine, I can add runners after removing the work that does not need to happen. Otherwise I would be spending more money to run tests that are unrelated to the change. The larger lesson is about arrival rate rather than headcount. One engineer working with coding agents produced changes fast enough to recreate a CI scaling problem I had previously encountered on a large engineering team. As agents increase the rate of change, verification has to become more selective or the infrastructure behind it has to grow at the same rate.