We scanned 1,868 AI-built apps for production readiness, and audited our scanner

This is the post behind every number we publish about AI-built code. It has two halves, and the second one is the uncomfortable one. First: what we found scanning 1,868 public apps built with AI coding tools. Second: how we audited our own scanner to make sure those numbers are honest, including the part where we discovered we'd been over-counting.

The short version: most scanners never publish their false-positive rate. We measured ours on a representative sample of our critical findings, and it was about 42%. We fixed the classes behind it, re-scanned all 1,868 apps, cut the rate to about 25%, and the corrected numbers are below.

If you arrived from a headline, this is where it has to defend itself. Definitions are pinned, caveats come before results, and the places our scanner gets things wrong are documented, not buried.

What we scanned

We targeted roughly 1,900 public GitHub repositories built with AI coding tools. A handful were unreachable (deleted, emptied, or timed out), leaving 1,868 analyzed.

For tool attribution we use hand-checked labels first (repos we identified as built with a specific tool from clear evidence, like the tool's own config files or generated artifacts), and fall back to the scanner's automatic detection only where no label exists. Every per-tool number below uses that one method.

Nine tools, each with a healthy sample: Bolt, Lovable, v0, Cursor, Replit, Tempo, Base44, Firebase Studio, and Windsurf (per-tool counts in the table below), plus 234 repos no method could attribute.

One transparency number before any results: our automatic detector now agrees with the hand-checked labels 92% of the time, up from 76% at our first report, after we fixed a detection bug (more on that below). Detecting which tool built an app is genuinely hard, because some editors leave almost no trace, which is why the table leans on hand-checked labels rather than the detector alone.

What the scanner checks

PathToShip runs 80 automated checks across 7 weighted dimensions: Security 30%, Production readiness 18%, Architecture 12%, Scalability 12%, Infrastructure 12%, Code quality 8%, Cost efficiency 8%.

Severity sets the penalty: critical −30, high −14, medium −6, low −3, and info-level findings always carry zero penalty: they're observations, not score movers. Two design choices worth knowing:

Empty repos don't win. A dimension with nothing to evaluate scores 50, not 100. The scanner does not reward having nothing to find.

The production-ready bar is 80/100, the same threshold our certification uses: one bar, applied identically to every repo, including our own.

The honest scope statement: this is pattern-based static analysis over a capped download of each repo (200 files / 500KB, which keeps a scan around 30 seconds). It cannot see runtime behavior, billing plans, deployment settings, or how data flows between files. Where that bit us, we say so.

Selection bias, stated plainly

These are public repos. People who build serious production apps tend to keep them private. Our set almost certainly over-represents hobby projects, demos, and learning exercises, and under-represents AI-assisted apps with real users and real revenue.

So every statistic here is "of the repos we scanned." We make no claims about AI-built apps in general, and if you quote one of our numbers that way, you're quoting it wrong.

The results

Score distribution: mean 68.3, median 71, range 31–95. Most of the set lands in the working-but-gapped middle band: shippable demos, not shippable products.

Headline rates, of the 1,868 repos scanned, definitions pinned:

23% pass the 80-point production-ready bar.

24% have at least one critical finding, after the accuracy work described in the next section. (The pre-audit scanner said 33% on this same set; hold that thought.)

15% ship a hardcoded secret or API key, defined as the union of our "Hardcoded Secret/Password" and "Hardcoded API Key" rules, counted per repo.

35% have an XSS-class finding, defined conservatively as the union of our two dangerouslySetInnerHTML rules (a broader definition that also counts raw innerHTML gives 43%).

By dimension, the pattern is the whole thesis: production readiness is the weakest (mean 56), then security (65) and architecture (63); scalability (80) and cost efficiency (88) are strong. AI tools generate efficient code that isn't ready to operate. What's missing is the secrets handling, the error fallbacks, the logging, and the deployment hygiene.

The per-tool table

The framing that has to come before the table: this is a snapshot of public repos per tool, not a tool ranking. Different tools attract different users building different things. Treat the rows as observations, nothing more, and read the next section before you take any single number to the bank.

ToolReposMedian scoreMean security% with ≥1 critical

Lovable259746831%

v0259726726%

Base44248737221%

Firebase Studio245726621%

Tempo243716825%

Replit165675421%

Bolt152727222%

Cursor137685722%

Windsurf134715629%

The honest headline of this table: across tools, the share of repos with at least one critical finding now sits in a 21–31% band, down from the 22–53% spread we first reported (Lovable and Windsurf at the top, most others in the low 20s). Our first report led with "some tools are far riskier than others." Most of that spread turned out to be false positives landing unevenly across tools. It wasn't real, and the next section is why we can now say so.

The findings you'll actually see

The most common findings of medium severity or higher, by share of repos:

Finding% of repos

No Content Security Policy90%

No privacy policy88%

Oversized files (300+ lines)74%

No React error boundaries60%

No linting config58%

Inline styles overuse51%

No cache invalidation48%

No hosting config detected45%

None of these is exotic. That's the point: the distance between an AI-built demo and a production app is mostly unglamorous, well-understood work.

Can you trust these numbers? We audited the scanner to find out.

Here's the part most data studies skip. A scanner's numbers are only as good as its false-positive rate, and nobody publishes theirs. We did.

A "critical" finding is the most expensive thing our scanner can say: it's what a builder drops everything to fix. So a false critical is our worst error: it inflates our stats and sends someone chasing a problem that isn't there. We'd killed some false-positive classes before, but "we fixed the ones we noticed" isn't "we measured how many are left." So we measured.

We hand-judged critical findings against the actual code, one at a time: confirmed (real), false positive (fired on something safe), or ambiguous (real pattern, arguable severity). We did it in two passes: a discovery pass to find the kinds of false positives, then a representative estimation sample of 57 findings, drawn proportionally so its false-positive share equals the fleet's.

The result: about 42% of our critical findings were false positives (95% confidence interval ±12 points). Not a comfortable number to publish. So we did the obvious thing: fixed the classes it pointed at and re-ran. Two rounds of accuracy work cut the critical false-positive rate from ~42% to ~25%. That's why the critical rate above is 24% and not 33%, and why the secret rate is 15% and not 25%.

What was actually wrong, named because "we found some bugs" isn't accountability:

Placeholder secrets in example files. The scanner read postgresql://:@host in a committed .env.example as a leaked credential. The tempting fix (skip .env.example) is wrong: we found a repo with a real Azure secret in exactly such a file. So the fix inspects the value (your-password, , ${VAR}), not the filename. The real secret still fires.

Secret-shaped names holding harmless values, the single biggest class. FORGOT_PASSWORD: '/auth/forgot-password' (a route), token: 'comment' (an editor theme color), password: 'Password is required' (an error message). The rule read the key and ignored the value. Now it reads the value.

Permissive database policies that grant nothing. A "allow everything" policy scoped to service_role (a backend key that bypasses row-level security anyway) is not a hole. One scoped to the public role is a real hole, and those still fire.

A detection bug distorting the per-tool numbers. v0 rebranded from v0.dev to v0.app; our detector still looked for the old domain and was silently missing ~85% of v0 repos. Fixing it moved tool-detection agreement from 76% to 92% and, with the other fixes, flattened that per-tool spread.

What this still doesn't fix

About a quarter of critical findings are still false positives, on purpose. We stopped at ~25%, not because we ran out of ideas but because the remaining classes (test fixtures, translated strings, HTML sanitized one function call away) can only be resolved by tracing where a value came from. Pattern-matching can't do that reliably, and a rule that guesses will eventually suppress a real vulnerability to look cleaner. For a security scanner that's the one unforgivable error, so we didn't ship it. Every fix we did ship was verified against 32 hand-confirmed real findings first, and none went silent. The residual is the price of not hiding real leaks; dataflow analysis is the next round.

The estimate has error bars. 57 hand-judgments pin the false-positive rate to within about 12 points, not to the decimal.

Static analysis is not runtime truth, and the set skews hobby. The audit corrects how often we cried wolf. It doesn't change what the dataset is.

Why we're telling you this

A scanner that never publishes its own error rate is asking you to take its findings on faith. We'd rather show the error rate, name the bugs, and fix them in the open, the same way we published the finding-by-finding receipt when the scanner scored us a 56. If you think one of our rules is wrong, tell us; two rounds of that feedback have made it measurably more accurate.

Check your own

The scan is free, takes about 30 seconds, and asks for no signup: pathtoship.com. Or read the per-tool detail: Bolt, Lovable, v0, Cursor, Replit, Tempo, Base44, Firebase Studio, and Windsurf.