August 31, 2026 · Niklas

Seventeen comments that changed other people's code

A few weeks ago we said we weren't going to show you a benchmark yet, and that we'd rather run the reviewer on other people's repositories first and publish what actually came out. This is that.

Three of our own patches have merged upstream, in tower-http, anyio and redis-py. But the number we care about more is this: on eight repositories a maintainer read a comment we left on their pull request and changed the code because of it. Among them n8n at 203k stars, Astral's uv at 89k, and OpenHands at 86k. Seventeen comments in all, landing in commits with messages like "Address review" and "good catch on both". Six of those pull requests have since merged, carrying our changes with them.

Every pull request below is public, so you can check any of it yourself.

How we ran it

Target selection was boring on purpose: over 50 stars, three commits on the default branch in the last 30 days, one human-authored pull request over 100 lines. Alive repositories, diffs big enough that reading the patch alone wouldn't tell you the answer. That gave us 72 repositories and 174 pull requests.

We took a real pull request from each, recreated it on a fork, and reviewed it there, so the reviewer sees the change without the merge commit, without the review conversation that followed, and without whatever the maintainers caught before merge. Then a human decided whether the output was worth a maintainer's inbox, and forty findings went up under our own names, saying plainly where they came from.

The seventeen that landed

The three largest took everything we left them. n8n (203k stars) folded all four comments into two commits and merged the pull request on 25 August. Astral's uv (89k) answered all three in a single commit titled "Address review". OpenHands (86k) fixed both and merged the next day. Between them, herdtools7, opengrep and NVIDIA's cudf-spark took five more.

Two are worth showing in full, because of what you have to know to find them.

sqlfluff. A pull request changed how MySQL's IF ... THEN ... ELSE ... END IF gets parsed, ending each branch as soon as it hits an ELSE. The catch is that CASE expressions have an ELSE of their own. So an ordinary line like SELECT CASE WHEN a = 1 THEN 1 ELSE 2 END; written inside an IF closes the branch too early, and the rest of the statement stops parsing. Nothing in the diff looks wrong. You only catch it if you know what else in the language uses that keyword. The author confirmed it, our example is now a test fixture for MySQL and MariaDB, and the pull request merged on 28 August.

tarantool. A change taught the connection pool to reconnect when an instance's URI changes on config reload. Looking up an instance's URI fails if that instance is gone, so the code treated any failure as "this instance was removed": close the connection, drop it. But the same lookup also fails when a password is misconfigured. Get a credential wrong and every instance still in your cluster quietly loses its pooled connection, with nothing reported anywhere. The fix that landed checks membership explicitly and lets the other errors through, exactly as suggested.

What a maintainer told us

What we did not expect was how many maintainers came back to say the findings were worth having. On a restate pull request we flagged a lock that would reject a changed federation config after the first client was built, leaving a process restart as the only way out. The reply:

I really appreciate you stopping by, those were good findings! I hadn't yet done extensive review at the time but I had already done a single pass with both Claude Code and Codex and neither had surfaced them – whatever you're doing with Context Goblin for added context, is definitely working well.

We didn't ask for the comparison and can't reproduce his runs. But nobody with a reason to flatter us has given us a closer outside read on the actual claim, which is that building context before reading finds things reading the diff doesn't.

He was the most direct about it, not the only one. Maintainers at longhorn-manager, babashka's sci, kroxylicious and teku all confirmed the findings were real, and on sqlfluff the author simply replied "good catch on both" before fixing them.

Where this goes next

Of the forty findings, seventeen became code, eight were accepted for later, two were answered with a reason, and thirteen are still sitting in queues. Open-source maintainers have a lot of those, and a comment from a stranger waits its turn like everything else.

The change these 72 repositories argue for hardest is teaching the reviewer to recognise a decision that has already been made. On cudf-spark we described a guard accurately and were told its threshold had never been a hard limit in the first place, so being off by one against it cost nothing. That reasoning lives in the repository's own history, which we already index, so it should be reaching the agent before a finding ever gets written.

We're also going to open more pull requests and leave fewer comments. Every one of our three patches got read and merged; a finding that arrives with the fix attached is simply easier to say yes to.

If you want it pointed at your own repositories: contextgoblin.com.