# Lab detail — Loop Control (who determines done in an agentic loop) Raw lab detail for labs.layer2c.com/labs/loop-control. A five-day experiment (2026-05-13 to 2026-05-17) on a single NVIDIA DGX Spark, testing whether tiered model escalation gated by a deterministic evaluator can match frontier-only accuracy at lower cost. **What ships:** the architecture, the measured results, the cost model, the hardware findings, the failure-mode catalog, the design recommendations. **What stays proprietary:** the specific task set, the extracted tests, and the tuning particulars. Returns, not the algorithm. ## Correction (July 2026) A later audit of the test harness found nine invalid validator checks: empty or self-contradictory gates that no output could pass. They had recorded escalation events for tasks that were never actually solved. Corrected, those cases are persistent failures, not escalations, and the reported escalation gain largely dissolves. The engineering conclusion is unchanged and sharper: the deterministic validator determines done, and for this class of problem the small local model cleared the solvable work without a need to escalate. The frontier tier bought throughput, not correct answers. Lab 9 (Recovered capacity is real) re-ran the same instrument at rigor and confirmed it: a 12B model at Q4 cleared the solvable set with zero confirmed false passes across 585 attempts. The escalation statistics below are left in place, marked as first reported, so the correction stays legible against them. ## The question Original: where should judgment live in a local-first agentic AI system? Refined after the data: **who determines done, and can a deterministic evaluator gate model escalation without human intervention?** The reframe: move "loop control" away from *who controls the loop* and toward *who determines done*. The support-desk analogy — an L1 analyst attempts a ticket, but a deterministic escalation rule (not the analyst's self-assessment) routes it to L2 — applied to model inference. ## Setup - **Hardware:** one NVIDIA DGX Spark (Grace Blackwell GB10, 119 GiB unified memory). - **Local models tested:** nemotron-3-super (123B dense, ~1.8 tok/s, failed repo tasks on format violations); Gemma 4 31B dense (~3–4 tok/s, passed); Qwen 2.5 Coder 32B AWQ (~1 tok/s, failed); LLaMA 3.1 70B AWQ (~0.5 tok/s, timed out — hardware ceiling); Qwen3.6-35B-A3B FP8 MoE (~48 tok/s, fastest); **Gemma 4 26B-A4B MoE (~22 tok/s, primary local worker).** - **Frontier tiers (OpenAI):** o3 (tier 2, ~$0.03/call billed), gpt-5.5 (tier 3, ~$0.40/call). - **Tasks:** bug-fix tasks mined from real repos (more-itertools, httpx, PyJWT, h2, and others) by git archaeology — roll source back to pre-fix, use the fix's own tests as an unfalsifiable gate. ## The architecture (three-tier escalation) ``` Tier 1: Local model (free, 1 attempt) -> run tests -> pass? done fail? v Tier 2: Frontier model (2 attempts, error feedback) -> run tests -> pass? done fail? v Tier 3: Most-capable model (1 attempt) -> run tests -> done ``` Each tier starts from a clean workspace (original buggy code), no prior-tier context. The evaluator (test harness) is the sole decision authority between tiers. Attempt budget matches the model: 1 for a deterministic local model (feedback can't change a temperature-0 output), 2 for a reasoning model (error output steers a genuinely different retry). ## Measured results | Metric | Value | |---|---| | Local-only pass rate | 25% (2/8) | | Local + repair loop | 25% (no gain) | | Three-tier escalation (as first reported) | **75% (6/8)**. Superseded; see Correction. Nine invalid validators booked escalations for unsolved cases. | | Small model clears the work (Lab 9) | 21 of 39 · 0 confirmed false passes across 585 attempts | | Repair-loop sweet spot | ~5% of tasks | | Task difficulty (26 calibrated) | 42% trivial · ~4% medium · 54% too hard for one pass | | Frontier (o3) second-attempt recovery (as first reported) | 67% of near-misses. Superseded by the validator audit. | | Total run cost | ~$4.60 across 156 API calls | | Escalation cost at 25% local pass | ~$0.16/task | | Local throughput, MoE FP8 vs dense | 48 vs 1 tok/s | Escalation cost scales with local failure rate: 100% local pass → $0.00; 75% → ~$0.03; 25% (observed) → ~$0.16; 0% → ~$0.14 (same as frontier-only). ## Key findings 1. **Repair loops are model-dependent.** Deterministic model (temp 0): same input, same output, zero value; near-miss tasks produced byte-identical output across 4 attempts. Reasoning model (o3): a second attempt on the evaluator's test output can recover a near-miss the first pass failed (the specific recovery rate is among the escalation figures the audit supersedes). The variable is whether the model can produce a *different* output given the same feedback. 2. **The small model cleared the solvable work; escalation was mostly mis-credited.** As first reported this read as 25% (local) to 75% (three-tier), escalation the dominant architecture. The validator audit corrected it: nine invalid checks had booked escalations for unsolved cases, and for this class of problem the local tier cleared the solvable work without a need to escalate. The frontier tier's measured advantage was throughput, not correctness. 3. **The evaluator is the authority.** The deterministic test harness provides the escalation signal, the accept/reject verdict, and the error feedback. No model judges its own or another's work. 4. **Universal review degrades accuracy.** Every all-outputs review pass made things worse (over- saving bias; governor anchoring — a frontier reviewer confirmed a local call it would have made differently on a fresh look). Gate review on a deterministic trigger, or skip it. 5. **Cost scales with difficulty; capability > architecture.** Pay for a frontier call only on rejection. A MoE model passed tasks a larger dense model and a coding-specialized model failed. 6. **Token-rate estimates overstate real o3 spend 4–5×** (~$0.14/call estimated vs ~$0.03 billed). ## Failure-mode catalog | Failure mode | Cause | Fix / workaround | |---|---|---| | Over-saving bias | Review promotes ignores to saves | Gate review with deterministic triggers | | Governor anchoring | Reviewer biased by seeing the local output | Present the task fresh, not as a review | | Format violations (nemotron) | Model puts explanation in the files dict | Switch models (Gemma 4) | | Deterministic convergence | Same input = same output at temp 0 | Drop repair loops; escalate instead | | Frontier empty-files | Model returns valid JSON, no code (~25% of calls) | No reliable fix found | | Wrong Python interpreter | System python lacks pytest | Use the project venv | | Stale bytecode across tiers | `__pycache__` from a failed tier survives | Clear it at escalation | | Bad test in extraction | Feature test passes on buggy code | Manual review of extracted tests | ## Design recommendations - **Two-tier escalation with deterministic gating.** Tier 3 didn't add value in this set. - **One attempt for deterministic local models; two for reasoning-model tiers.** - **MoE models for the local tier on edge hardware** (FP8 on Blackwell is the throughput driver). - **Invest in evaluator quality, not prompt engineering.** The test harness is the system's judgment; a bad test makes a task unsolvable regardless of model capability. Evaluator reliability is the ceiling. ## What did NOT get settled (honesty) - Small task set (8 in the culminating run). Architecture verdict, not a pass-rate benchmark. - The failure count was contaminated by nine invalid validators (empty or self-contradictory gates); corrected, those are persistent failures on broken tests, not model-capability findings. See the Correction. - Tier 3 added nothing here (proof point, not demonstrated advantage). - Temperature >0 in the local loop untested; scale economics (a 50–70% local model) not measured. - Cross-domain: the finding rests on an executable evaluator. Domains without one (triage, content, planning) have a weaker signal, not demonstrated here.