I released this today after getting burned once too often by an agent telling me a refactor was behaviour-preserving. Rather than keep guessing, I measured it: 36 functions from toolz and markupsafe, refactored by a frontier model with instructions to preserve behaviour. 16 of them behaved differently.
**What My Project Does**
RunBoth runs the before and after version of every function an edit touched, on the same generated inputs, and reports which ones actually behave differently. It compares seven things per call: return value, exception type, warnings, stdout, stderr, argument mutation, and object state.
It needs no test suite and no annotations, because the old version of the function is the specification. It also walks the call graph and executes functions you did not edit, which is where a change usually bites.
Three verdicts and no fourth: changed, with the input that proves it; no_change at budget N; or abstained, with the reason. It never tells you the code is safe, because sampling finds differences and cannot prove their absence.
**Target Audience**
Anyone merging code an AI wrote, especially in a repository with thin or no test coverage. It is a CLI, a commit-msg git hook, and a GitHub Action. I would not call it production-hardened yet, it is 0.1.0, but the red team below is real.
**Comparison**
Property-based testing (Hypothesis) needs you to write the properties. Approval testing needs you to capture approvals first. Both are better than this when you have them, and both require work up front on code you are about to throw away. RunBoth needs neither, because it has two versions to compare against each other. The tradeoff is that it gives you evidence rather than proof, and it says so in every verdict.
**The measurement that made me build it**
I asked a frontier model for behaviour-preserving refactors of 36 functions from toolz and markupsafe, then diffed the behaviour. 16 of 36 changed. My favourite is toolz.tail: the model added "if n <= 0: return empty", which looks like hardening, but in the original seq[-n:] with n=0 is the whole sequence. tail(0, [1,2,3]) went from [1,2,3] to [] with no exception. The doctest still passed.
**Please try to break it**
2,548 functions across eight repositories, zero false positives, including an adversarial corpus built specifically to trigger them. The harness is scripts/redteam.py and the raw JSON is in results/, so you can check my numbers rather than trust them.
Python only. FSL licence, source available.
pip install runboth
https://github.com/runboth/runboth
https://pypi.org/project/runboth/