Most of the engineering effort on Prbl hasn’t gone into finding more vulnerability classes. It’s gone into not crying wolf. A rule that fires on clean code trains you to ignore the tool, which defeats the purpose of having it.
The pipeline every rule has to pass
- A synthetic test suite: minimum 10 cases per rule, covering both true positives and known false-positive shapes
- A batch stress test against at least 20 real, public codebases
- A confirmed false-positive rate under 10% on human-written code before merge
- Validation against at least one codebase with 1,000+ files
Rules that skip this pipeline get reverted. One of our own rules, an SSRF detector, was removed after producing a 100% false-positive rate on the first production codebase it touched. It had been added without running any of the four steps above. That’s the failure mode this pipeline exists to catch.
Scan your own app for issues like these
Paste your live URL. We check what your app serves publicly for exposed keys and misconfigurations. No account, no install.
Where the testing has gone so far
15 rule classes, validated across 113 repositories in four separate batches: established enterprise open-source projects, Next.js/Vercel-style SaaS codebases, solo-developer tutorial projects, and the batch most relevant to Prbl’s actual thesis, output from AI app builders (Lovable, Bolt, v0). 303 regression tests currently pass with zero failures.
In the AI-builder batch specifically: 4 out of 20 generated repos contained a hardcoded fallback secret, process.env.SECRET || 'hardcoded-value', baked directly into scaffolded auth or session config. That pattern doesn’t show up in the other three batches anywhere near as often. It’s a generation-time default, not a developer mistake, and general-purpose scanners we compared against didn’t flag it.
A direct precision comparison
We ran Semgrep’s default ruleset (p/default) and Prbl against create-t3-app, a clean, widely-used Next.js scaffolding tool with no known path-traversal vulnerabilities. Semgrep’s default ruleset produced 54 findings on it. 53 of them were path-traversal false positives from pattern-matching on file-path handling that was already safe. Prbl produced zero findings on the same repo.
This isn’t a claim that Semgrep is bad. It’s a much broader tool solving a much broader problem. It’s a demonstration of what a narrower, more targeted rule set buys you: on a codebase with nothing wrong, a precision-tuned scanner should say nothing, not generate 54 things to triage.
What we don’t publish
Stress-testing against real, popular open-source repositories occasionally turns up genuine, exploitable findings in projects we don’t maintain. When that happens, we report them privately to the maintainers and give them time to fix the issue before any of it becomes a public case study. That’s the same standard disclosure window any security researcher should follow. A couple of those reports are open right now. We’ll write them up once they’re resolved, not before.
Why this matters more than rule count
Anyone can write a regex that matches password\s*=. The hard part is making it not fire on a test fixture, a Swagger example, or a TypeScript enum value that happens to contain the word “secret.” That’s where most of the actual engineering time goes, and it’s the part that doesn’t show up in a feature list.
Common questions
How does Prbl test for false positives?
Every rule ships with a synthetic test suite covering both true-positive cases (code patterns that should fire) and true-negative cases (similar-looking code that should not). Before any study publishes, we run the scanner against 113 real-world repos and manually review every high-severity finding. False-positive patterns that review surfaces get fixed in the scanner, and the repos are re-scanned before numbers go into any post.
How does Prbl's false-positive rate compare to Semgrep?
On the same 113 real repos, Semgrep's default ruleset returned 54 high-severity findings. After manual review, all 54 were false positives in that context. Prbl returned 0 false positives on the same repos. The comparison is not a claim that Semgrep is bad — it is a much broader tool. The point is that a narrower, AI-code-tuned ruleset can have near-zero noise on codebases that do nothing wrong.
What is in Prbl's regression test suite?
303 regression tests cover every rule in two directions: code that must be flagged and code that must not be. Each of the 11 false-positive patterns fixed during the study has its own regression test. A rule cannot ship unless both the positive and negative cases pass. The suite runs on every scanner build before deployment.
How many real repos does Prbl test against?
113 repos, sourced from public GitHub repositories that represent real, production-grade codebases across a range of languages and frameworks. The goal is not to find vulnerabilities in those repos — it is to verify that Prbl does not cry wolf on code that is doing the right thing.
Why does Prbl publish its testing methodology?
Anyone can claim low false positives without showing the work. Publishing the exact list of what broke, what changed, and what regression test now guards each fix makes the claim verifiable. A scanner whose testing process is opaque is asking you to take the precision on faith.