How a Docs-Fixing AI Agent Fooled Itself: False Positives in a Docker Sandbox

An AI agent built to repair broken code samples in documentation produced a strange result on its first real repository. It "repaired" a bug that didn't exist, taking a perfectly good line, changing nothing and publishing the identical text as a verified fix.

- print(tqdm.__version__, sys.version, sys.platform)
+ print(tqdm.__version__, sys.version, sys.platform)

Both lines match. The agent had talked itself into fixing something that was never wrong, and that was only the first of three ways it fooled itself before the real culprit turned up.

What DocuPatch does

DocuPatch is a LangGraph-driven agent. It walks a GitHub repo's Markdown files, pulls out Python code blocks and runs them in a sandboxed Docker container. Samples that truly fail go to Gemini for a repair, the fix is checked again in the same sandbox, and only a verified result becomes a branch and pull request.

The layout of the repo:

• agent/ingest.py clones the repository and finds the Markdown files.
• agent/graph.py starts a LangGraph pipeline per file.
• agent/parser.py extracts the Python code blocks.
• agent/tester.py runs each snippet inside a Docker container.
• agent/context_collector.py gathers the relevant Python files and code for a snippet.
• agent/fixer.py produces the repaired snippet.
• agent/github_client.py forks, commits the fix and opens PRs.
• tests/ holds tests for every piece.

The premise is simple: documentation code rots. APIs shift, functions are renamed, examples stop running, and nobody notices until someone pastes a dead snippet out of a README. The agent automates the catch. It finds a suspect snippet, proves it's broken by executing it, asks an LLM for a fix, proves the fix works by executing again and only then opens a pull request.

The pipeline has a clone-and-parse stage feeding an evidence-gated loop, and a fix leaves only after two proofs: the original failed, and the replacement doesn't.

The guiding rule is evidence-gating. Nothing reaches the LLM unless a sandbox run shows it's broken, and nothing gets published unless a sandbox re-run shows the fix works. That principle survived. The sandbox itself, for a good while, did not.

First test: tqdm

The initial real-world run targeted tqdm/tqdm:

Found 9 Markdown file(s). Starting audit...

Auditing: .github\ISSUE_TEMPLATE\bug.md
  Generating repair for snippet #1
  Verifying generated repair for snippet #1
  Publishing verified repair(s)...
  VERIFIED REPAIR: Branch published

Two files were "fixed" and published, a win on paper. Then came a look at the actual diff:

- print(tqdm.__version__, sys.version, sys.platform)
+ print(tqdm.__version__, sys.version, sys.platform)

Again the two lines are identical. Gemini changed nothing and regenerated the code byte for byte. The system tested a snippet, declared it broken, "fixed" it by returning the same code, tested again, saw a pass and pushed a no-op commit to the fork.

This wasn't a fixer bug. It was a false positive at the testing layer, because the original snippet was never broken. A line importing tqdm and sys and printing version info is about as ordinary as tqdm usage gets, so why did the sandbox report failure? The question was parked as suspicious, and the next repo was tried to get a second data point before drawing conclusions.

Second test: pallets/click

On pallets/click the pattern was impossible to miss:

Auditing: docs\advanced.md
  Generating repair for snippet #2
  Verifying generated repair for snippet #2
  [Reflection Loop 2/3] Revising repair for snippet #2
  ...
  [Reflection Loop 3/3] Revising repair for snippet #3
  NOT PUBLISHED: Generated repair failed verification:
    ModuleNotFoundError: No module named 'click'

One error, every time. Six files each ran through three complete reflection loops, and every failure was the same ModuleNotFoundError: No module named 'click'.

That repetition is the tell. When six files and six separate LLM attempts all die on one import error, the shared factor isn't the code. It's the environment underneath. Rewriting a snippet can't cure a missing module if the sandbox can't see the package no matter what gets written.

Opening tester.py revealed the mechanism:

command.extend(["-v", f"{project_dir}:/workspace", "-w", "/workspace", "-e", "PYTHONPATH=/workspace"])

Inside the container, the cloned repo is mounted and PYTHONPATH is set to the repo's top level. That's fine when the package directory sits right there, and it breaks when the package is nested a level lower. Click is nested: its pyproject.toml keeps the real code under src/click/ instead of a click/ folder at the root, the src-layout so many modern projects follow. With PYTHONPATH=/workspace, the interpreter was looking in a directory that held no click folder at all.

A manual check outside the pipeline confirmed it:

docker run --rm -v "C:\temp\click_test:/workspace" -w /workspace -e PYTHONPATH=/workspace python:3.10-slim python -c "import click"
# ModuleNotFoundError: No module named 'click'

docker run --rm -v "C:\temp\click_test:/workspace" -w /workspace -e PYTHONPATH=/workspace/src python:3.10-slim python -c "import click; print(click.__version__)"
# import works... but:
# importlib.metadata.PackageNotFoundError: No package metadata was found for click

The second result taught the most. Pointing PYTHONPATH at the right folder cured the import, yet click.__version__ still crashed, because current click builds __version__ through importlib.metadata, which reads installed package metadata and not source files. That metadata exists only after a real pip install, never after a path hack.

So two bugs were stacked. PYTHONPATH aimed at the wrong folder for src-layout repos. And even aimed correctly, a plain PYTHONPATH trick doesn't produce a properly installed package, while more and more packages expect to be installed and not merely importable.

One detour deserves a mention, as it was the most instructive stretch. Plenty of time went into a hunch about a Windows-only Docker mount quirk, with experiments on quoting, path styles and argument lists against shell strings. The true explanation was dull: the test folder was empty because an earlier git clone silently failed to fill it, and nobody checked. An empty dir C:\temp\click_test listing wasn't "no answer yet" but the answer itself, and it was passed over three times before anyone stopped to read it.

The fix: install, don't guess

The real remedy isn't a smarter path guess but no guessing. pip install -e . reads a project's own pyproject.toml or setup.cfg, works out the right package location whatever the layout, and writes genuine install metadata, so importlib.metadata calls behave as they do for any real user of the library.

The new tester.py logic runs in this order:

• If a pyproject.toml, setup.py or setup.cfg exists, the sandbox builds a Docker image and runs pip install -e . inside it. This is "installed mode", where an import failure can be believed.
• With no packaging metadata but a requirements.txt, fall back to the earlier PYTHONPATH approach, "pythonpath mode".
• In pythonpath mode, if a snippet fails on an unresolved import of the project's own package, label it ENV_ERROR, an honest "can't tell", instead of FAIL.

After the change, the existing suite showed 26 of 27 tests passing untouched. The one failure was right to fail, since it asserted the old function signature that had been changed on purpose. The expectation was updated and three new tests covered the edge case, bringing the full suite to 74 passing.

A real catch in click's docs

With the sandbox installing the project and no longer guessing, click ran again. The six files that previously failed identically now got a fair chance, and one exposed a genuine, undiscovered bug in click's own documentation.

docs/commands.md walks through a chained CLI pipeline, and one example function reads:

@cli.result_callback()
def process_pipeline(processors, fin):
- iterator = (x.rstrip("\r\n") for x in input)
+ iterator = (x.rstrip("\r\n") for x in fin)

The function actually receives fin, the file handle that click supplies. The docs instead loop over input, which resolves to Python's builtin input() function and not to the file. Looping over it is invalid and fails as soon as the code runs, but a reader wouldn't spot it because the name looks plausible.

DocuPatch caught it, confirmed a real failure in the sandbox, generated the fix, verified the corrected version ran end to end and published it. It was the first result in the whole investigation that was both a true documentation bug and a true correct fix, with no sandbox noise or false positive involved. This is what the project exists to do.

Lessons from the debugging

• Verify the verdict before trusting the result. Labels like "Passed" or "Published" are only as reliable as the test beneath them. The tqdm no-op came to light solely because the diff was opened and the green checkmark wasn't taken on faith.
• Identical failures across many cases suggest one shared cause. Six files, six independent LLM attempts and one error message add up to a system-level signal.
• Observe before theorizing. Hours went to Docker and Windows path theories before anyone verified that the test folder held any files.
• A failing test can confirm and not damage. When changing the signature of run_snippet broke one test, the right question was what that test actually asserted, not what had been broken.

Now the sandbox bug is fixed, covered by tests and confirmed on a repo it used to choke on. Click's docs held a real defect, a variable name that breaks the second anyone runs the example, and the agent located it, repaired it correctly and checked the repair before publishing. That's the project's central idea proven against a real case and not a synthetic one.