How Do I Know a Test Would Actually Catch the Bug?
A runnable Python lab that exposes a false-green test, proves the bug-specific test fails on broken behavior, and verifies it passes after restoration.
A green test can look like the safest line in a bug-fix review and still leave the reported behavior completely undetected. Before accepting that green result, make the same test confront one safe, known-broken version of the behavior; then restore the fix and require the same check to pass. That red-before-green round trip turns an encouraging signal into evidence about the bug the test claims to catch.
Problem: Why can a green test still miss the bug?
A green regression test can authorize a broken repair. If that test never goes red against known-broken behavior, you do not know whether it guards the reported bug or merely exercises a harmless path.
The distinction is easy to miss during review. A test may call the right function and still assert only that the result is a string. It may replace the function under test with a mock. It may calculate the expected value with the same code that produced the actual value. Every version can stay green while the customer-visible failure remains.
The direct answer is to make the test demonstrate both sides of its claim. Put the target behavior into a safe broken state, run the bug-specific test, and confirm it fails for the expected reason. Restore the fix and confirm the same test passes. That red-before-green round trip is a negative control: a deliberate condition where the test should reject the implementation.
This tutorial gives you a dependency-free fixture for that proof. It uses a small account-name normalization defect so we can see the mechanism without a framework, database, or network hiding the result. The same sequence applies to a unit test, an integration test, or a deployed probe, but the environment must match the claim you are accepting.
Value: What does the negative control buy you?
The negative control separates a working detector from a decorative test. That changes the review decision before weak evidence becomes a merge, release, or closed ticket.
It also gives you a bounded cost to compare with another investigation. Here is an illustrative workload calculation, not a measured saving: if preparing the broken state takes 8 minutes, restoring it takes 2 minutes, and the focused regression run takes 3 minutes, the proof costs 8 + 2 + 3 = 13 operator-minutes. Replace each input with your logs. For a text-only API run billed solely by input and output tokens, calculate token cost in dollars = (input tokens / 1,000,000 x current input rate in dollars per million) + (output tokens / 1,000,000 x current output rate in dollars per million). Otherwise use the provider's itemized charge, then add measured CI charges. The cited sources establish no universal billing model, rate, saving, or defect-detection percentage.
The durable benefit is an acceptance record that explains why the green result matters. A reviewer can see the broken condition, the expected failure, the restored condition, and the final pass. If the implementation, environment, or acceptance criterion changes, the record also tells the next operator which proof must be repeated.
Need help turning this pattern into a dependable AI system?
Just In Time AI can help your team design and implement the smallest useful workflow with the security, compliance, and operational controls it needs, then prove it before expanding. Explore AI Systems Setup and Coaching.
Solution: How do I prove the test detects the reported defect?
Before the steps, let us name the small system we will test. The function receives an account name such as " Alice ". The repaired behavior removes surrounding spaces and normalizes letter case, producing "alice". The known-broken implementation returns the input unchanged.
The lab contains two checks. The decorative check asks only whether the result is a string, so it passes against broken code. The bug-specific check asserts the exact normalized result. That check must fail against the broken implementation and pass against the repaired one.
Step 0 -- What do I need before I start?
Set aside about 10 minutes and use a disposable folder. You need Python 3.10 or newer and a terminal: PowerShell on Windows, or a shell on macOS or Linux. The lab uses only the Python standard library; the argparse documentation describes the command-line parser it uses.
Download verification_lab.py, expected-output.txt, raw-execution-receipt.json, and proof-record.example.json into the same folder.
Confirm the interpreter before doing anything else. On macOS or Linux, run python3 --version. In Windows PowerShell, run py -3 --version. Continue only when the command reports Python 3.10 or newer. If python3 is missing on Windows, use py -3 for every command below.
Step 1 -- What exactly will I run?
Here is the complete fixture. The broken branch returns the raw value. The fixed branch applies the two transformations the regression test claims to protect.
#!/usr/bin/env python3
"""Demonstrate a false-green test and a bug-specific negative control."""
from __future__ import annotations
import argparse
def normalize_account(raw: str, implementation: str) -> str:
"""Return either the known-broken behavior or the repaired behavior."""
if implementation == "broken":
return raw
return raw.strip().casefold()
def decorative_test(implementation: str) -> tuple[bool, str, str]:
"""A weak check that passes even when normalization is absent."""
actual = normalize_account(" Alice ", implementation)
return isinstance(actual, str), "a string", repr(actual)
def bug_specific_test(implementation: str) -> tuple[bool, str, str]:
"""The regression check for surrounding spaces and letter case."""
actual = normalize_account(" Alice ", implementation)
return actual == "alice", repr("alice"), repr(actual)
def run_one(implementation: str, test_name: str) -> int:
check = decorative_test if test_name == "decorative" else bug_specific_test
passed, expected, actual = check(implementation)
verdict = "PASS" if passed else "FAIL"
print(
f"{verdict} test={test_name} implementation={implementation} "
f"expected={expected} actual={actual}"
)
return 0 if passed else 1
def prove_round_trip() -> int:
decorative_broken = decorative_test("broken")[0]
specific_broken = bug_specific_test("broken")[0]
specific_fixed = bug_specific_test("fixed")[0]
print(f"decorative_on_broken={'PASS' if decorative_broken else 'FAIL'}")
print(f"specific_on_broken={'PASS' if specific_broken else 'FAIL_EXPECTED'}")
print(f"specific_on_fixed={'PASS' if specific_fixed else 'FAIL'}")
proof_passed = decorative_broken and not specific_broken and specific_fixed
print(f"proof={'PASS' if proof_passed else 'FAIL'}")
return 0 if proof_passed else 1
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--prove", action="store_true", help="run the complete proof")
parser.add_argument("--implementation", choices=("broken", "fixed"))
parser.add_argument("--test", choices=("decorative", "bug-specific"))
args = parser.parse_args()
if args.prove:
if args.implementation or args.test:
parser.error("--prove cannot be combined with --implementation or --test")
return prove_round_trip()
if not args.implementation or not args.test:
parser.error("use --prove or supply both --implementation and --test")
return run_one(args.implementation, args.test)
if __name__ == "__main__":
raise SystemExit(main())Save the file and run python3 verification_lab.py --help, or py -3 .\verification_lab.py --help in PowerShell. You should see --prove, --implementation, and --test. That check confirms the script starts and exposes the expected command-line interface before an expected failure complicates the picture; it does not prove the file's behavior or byte identity.
Step 2 -- How do I expose the false green?
Run python3 verification_lab.py --implementation broken --test decorative. PowerShell users run py -3 .\verification_lab.py --implementation broken --test decorative.
The command exits zero even though the output still contains spaces and an uppercase letter. That is the false green: PASS test=decorative implementation=broken expected=a string actual=' Alice '. The test executed code, but its assertion never examined the behavior named by the bug.
Step 3 -- How do I run the negative control safely?
STOP before doing this in a real repository: use a disposable worktree, scratch branch, container, or fixture switch. Do not remove a fix in a shared working tree, production deployment, or environment containing unique data. Record the original revision so restoration is mechanical.
Run python3 verification_lab.py --implementation broken --test bug-specific. In PowerShell, run py -3 .\verification_lab.py --implementation broken --test bug-specific.
This command is supposed to exit 1 and print FAIL test=bug-specific implementation=broken expected='alice' actual=' Alice '. The nonzero exit is evidence only after you inspect the message. A missing file, import failure, or broken setup is red for the wrong reason and does not prove defect detection.
On macOS or Linux, inspect the exit code immediately with echo $?. In PowerShell, use $LASTEXITCODE. The expected value is 1.
Step 4 -- How do I restore the fix and prove green?
Run python3 verification_lab.py --implementation fixed --test bug-specific, or py -3 .\verification_lab.py --implementation fixed --test bug-specific in PowerShell.
The command must exit zero and report PASS test=bug-specific implementation=fixed expected='alice' actual='alice'. If it stays red, compare the observed value with the acceptance criterion before changing the test. Weakening the assertion until it passes would erase the detection proof you just established.
Step 5 -- How do I verify the whole round trip at once?
Run python3 verification_lab.py --prove, or py -3 .\verification_lab.py --prove in PowerShell. Proof mode checks all three necessary observations and exits nonzero if any relationship changes.
decorative_on_broken=PASS
specific_on_broken=FAIL_EXPECTED
specific_on_fixed=PASS
proof=PASSThe four-line expected-output.txt file is the expected-output contract; it is not an execution log. Compare your run with it exactly. decorative_on_broken=PASS demonstrates the weak test. specific_on_broken=FAIL_EXPECTED proves the useful test rejects the defect. specific_on_fixed=PASS proves restoration. The final line is the lab's verdict on that relationship, not a claim about your production suite. The supplied raw execution receipt captures actual stdout, stderr, and exit codes from all four lab commands, together with the execution time, Python version, platform, fixture digest, input, and acceptance criterion.
The supplied receipt records a 2026-09-14 run of this fixture on CPython 3.13.13 on arm64 macOS. In that run, the decorative check passed against broken behavior, the bug-specific check failed with exit code 1, the restored check passed, and proof mode returned the four expected lines with exit code 0. That observation applies only to the fixture bytes and execution context bound in the receipt; it is not a production result.
My documented practice since 2025-12-15 is simple: I do not report a test as passed until I have read the actual result data. I want you to carry that same standard into this negative-control review: inspect the output and exit code before you repeat the word pass.
Step 6 -- How do I transfer the method to a real bug?
Write the acceptance criterion before touching the test. Name the observable symptom, the environment where it occurs, and the exact result that distinguishes broken from fixed. Capture a baseline from that environment.
Then create the narrowest safe broken condition. Revert the repair in an isolated checkout, select a known-broken fixture, or introduce one deliberate mutation at the boundary under test. Run only the bug-specific check first, inspect its failure reason, restore the exact original bytes, and rerun the same check. Finish with the affected regression set selected from the change's impact area.
Apply the source rule exactly. Whenever any one of these four triggers fires, dispatch an independent, separately tasked verifier agent before a fix is authored: the batch contains at least three diagnosed bugs; a captured log or trace may be stale; a bug is reported fixed but the symptom persists; or the diagnosis cites a file-and-line location the verifier has not independently read. The current public Adversarial Verify Before Fix pattern is the canonical name and requires that verifier to read the cited source, run a live probe, and look for a deeper or alternative cause. When logs or traces are part of the diagnosis, compare them with the current source and confirm that the referenced strings still exist. The verifier then returns exactly one outcome. CONFIRMED permits the fix author to work against the verified cause. OVERTURNED means do not fix that diagnosis; surface the contrary evidence instead. DEEPENED means the diagnosis agent records the broader cause and expands the repair scope. Fix authoring starts only after CONFIRMED or DEEPENED. Retain the narrow exception: an obvious, isolated, single-file trivial fix with an unambiguous root cause does not require a separate verifier agent.
What should I record with the test?
Keep one compact record beside the review evidence. It should name the claim, broken condition, command, exit result, failure reason, fixed condition, restored result, and regression scope. This example is deliberately narrow:
{
"claim": "The bug-specific test detects missing account normalization.",
"tested_revision": {
"path": "verification_lab.py",
"sha256": "sha256:e1c0a8b0b65d5c490d69e7921a915beb6fd43d2037a0c75ffc5e1b8c04cf52f7"
},
"execution_context": {
"recorded_at": "2026-09-14T18:29:05Z",
"python_implementation": "CPython",
"python_version": "3.13.13",
"platform": {
"system": "Darwin",
"release": "25.6.0",
"machine": "arm64",
"python_platform": "macOS-26.6.2-arm64-arm-64bit-Mach-O"
},
"working_directory": ".",
"working_directory_basis": "directory containing verification_lab.py",
"fixture_input": " Alice ",
"expected_normalized_value": "alice"
},
"broken_state": {
"implementation": "broken",
"command": "python3 verification_lab.py --implementation broken --test bug-specific",
"expected_exit_code": 1,
"observed_exit_code": 1,
"expected_test_result": "FAIL",
"observed_test_result": "FAIL",
"observed_output": "FAIL test=bug-specific implementation=broken expected='alice' actual=' Alice '",
"failure_reason": "expected 'alice'; received ' Alice '"
},
"fixed_state": {
"implementation": "fixed",
"command": "python3 verification_lab.py --implementation fixed --test bug-specific",
"expected_exit_code": 0,
"observed_exit_code": 0,
"expected_test_result": "PASS",
"observed_test_result": "PASS",
"observed_output": "PASS test=bug-specific implementation=fixed expected='alice' actual='alice'"
},
"raw_execution_receipt": {
"path": "raw-execution-receipt.json",
"sha256": "sha256:4d8033043d2756d94549f9110e24201b82445a7c5607032155531f88e6cdd622",
"proof_execution_id": "proof-round-trip",
"proof_exit_code": 0,
"observed_proof_lines": [
"decorative_on_broken=PASS",
"specific_on_broken=FAIL_EXPECTED",
"specific_on_fixed=PASS",
"proof=PASS"
]
},
"validity": {
"observed_at": "2026-09-14T18:29:05Z",
"expires_at": null,
"time_to_live_claimed": false,
"expiry_evaluation": "No universal time-to-live is claimed for this illustrative fixture. Apply the receiving review's freshness window to observed_at.",
"invalidate_immediately_if": [
"verification_lab.py bytes change",
"Python major or minor version changes",
"platform or machine architecture changes",
"fixture input or expected value changes",
"command or acceptance criterion changes"
]
},
"regression_scope": "account-name normalization fixture only",
"boundary": "Illustrative local fixture; observed values are derived from the bound raw execution receipt, and no production or customer result is claimed."
}Do not turn observed_test_result into a hand-entered success claim. This example derives the observed result, output, and exit code from the bound raw execution receipt. Its recorded_at, Python version, platform, fixture digest, input, command, and acceptance criterion let a receiver apply its freshness window and identify changes that invalidate the observation. A later source or execution-context change makes an old record historical evidence, not current acceptance.
What do I do when the result is surprising?
- The bug-specific test passes on broken code: confirm the broken condition actually reached the process, then trace the assertion to the observable behavior. The test may be hitting a mock, cached result, different binary, or different environment.
- The negative control fails before the assertion: repair the fixture or command and rerun. Setup failures do not count as detection.
- The restored implementation stays red: compare actual and expected behavior. The fix may be incomplete, or the acceptance criterion may describe the wrong result. Do not simply loosen the assertion.
- Proof mode prints
proof=FAIL: run the three individual commands to isolate which relationship changed. Treat an unexecuted or timed-out check as unresolved. - The focused test passes but the product symptom remains: reproduce on the actual routed and rendered surface. A unit result cannot settle a deployment, persistence, configuration, or integration claim it cannot observe.
Artifacts: What can I inspect or run?
- Adversarial Verify Before Fix is the current public source for the named verifier-agent rule and its four triggers.
- Python's
argparsedocumentation explains the standard-library command parser that runs the fixture. - verification_lab.py is the runnable lab.
- expected-output.txt is the expected-output contract.
- raw-execution-receipt.json records the actual commands, stdout, stderr, exits, runtime, platform, and fixture context from the supplied run.
- proof-record.example.json derives the compact review record from that receipt.
Bottom Line: What proof should I require before trusting the green test?
Require one inspectable round trip and keep its commands, exit results, failure reason, tested revision, and affected regression scope together:
- Name the observable criterion. State the exact result that distinguishes the reported bug from the repaired behavior.
- Require red for that reason. Run the bug-specific check in a safe isolated broken state and reject setup failures as proof.
- Restore and require green. Restore the exact fix, rerun the same check, and finish with the affected regression set.
Run the supplied --prove mode now. Then choose one regression test that currently has only a green history and build its safe negative control before the next review depends on it.
My shareable rule: A green test earns trust only after the same test rejects the bug it claims to catch. Share that line with the next reviewer who is handed a green-only regression result. Then record the failure reason your negative control exposed, or the fixture problem that kept the proof unresolved.
Need help turning that proof into a repeatable review control? Contact Dan Stolts about the failure signal your team needs to preserve.
For the business decision and value behind this method, read Should an AI Verifier Agent Check a Bug Diagnosis Before the Fix Author Starts?.
Frequently Asked Questions
Does a failing test always prove it catches the bug?
No. It must fail because the target behavior is wrong. A syntax error, missing dependency, unavailable database, or malformed fixture proves only that setup failed. Read the message and compare the observed value with the acceptance criterion.
Can I use a mock in the negative control?
Use mocks around external dependencies when isolation is the test's declared scope. Keep the unit whose behavior supports the claim real. If the mock replaces account normalization in this example, the test proves the configured mock output rather than the implementation.
Should every test run through a broken-state check in CI?
No universal cadence follows from this fixture. Capture the negative control when a regression test is introduced or materially changed. Automate mutation testing or repeated controls where the impact and run cost justify it, especially on authorization, persistence, billing, and other consequential paths.
What if I cannot safely remove the production fix?
Do not touch production. Reproduce the relevant behavior in a disposable checkout, isolated environment, recorded fixture, or lower-risk probe that can observe the same criterion. If no safe context can see the behavior, mark the detection claim unresolved.
How is a negative control different from code coverage?
Coverage shows that execution reached a line or branch. A negative control shows that the assertion rejects one known-broken behavior. Both can be useful, but line execution alone does not demonstrate that the test distinguishes the reported failure from the repaired result.
What if reproducing the broken state could damage production data?
Do not remove a fix or inject a fault in production. Reproduce the observable failure in an isolated fixture, disposable environment, recorded trace replay, or narrowly scoped mutation test that cannot reach live data. If no safe negative control can represent the claim, mark that criterion unproven and require a different evidence method before acceptance.
Update History
- 2026-09-14 -- Evidence, source-rule, and practitioner-close repair: At the prepublication stage, this article was revised to distinguish expected output from a real execution receipt, derive its observed results from that receipt, record the context needed for freshness review, state the complete verifier sequence including the current-source log check, and add an actor-neutral observed-run note, the source-grounded practitioner standard, a specific action, a shareable line, and a proof-record close. It also recorded the practice-origin date without claiming a prior public version, bound the paired article bytes current at that time, and declared both image compositions. These changes prevent a reviewer from mistaking an expectation template for observed proof, applying the source method too loosely, or reaching the end without a concrete next move.
- 2026-09-14 - Reader path and implementation help: Added the property-specific Just In Time AI service callout within the first third of the article and verified the paired reader path. The lifecycle-authorized article is bound to its locked canonical route; deployment and anonymous readback remain pending.
Evidence and provenance: This lifecycle-authorized tutorial is bound to https://itproguru.com/expert/2025/12/how-do-i-know-a-test-would-catch-the-bug/; deployment and anonymous public readback remain pending. Source sweep performed 2026-09-14 against the public repository and current internal method sources. Public repository: https://github.com/dstolts/jitneuro. Exact inspected pattern: https://github.com/dstolts/jitneuro/blob/421e2f26249fadfca602230db6b1973416fa1dfc/templates/_patterns/adversarial-verify-before-fix.md. The normalization lab is illustrative, dependency-free, and executed locally; it proves only the stated fixture behavior.