LOCAL AI • REAL REPOSITORIES • CONSUMER HARDWARE
Local models and hosted alternatives. Real repository tasks, independent checks, and honest failure reports. No premature leaderboard.
REFERENCE PLATFORM
The same model can behave very differently when context size, runtime, harness, operating system, quantization or available VRAM changes.
CURRENT EVIDENCE
Three curated benchmark-catalog observations, including an incomplete run and an unscored hardware limit. Development case studies below are assessed separately.
SUPERVISED DEVELOPMENT · NOT BENCHMARK SCORES
Evidence from October 2026
Beyond the three benchmark-catalog observations, the lab has documented coding work in actual repository workflows. These two Big Pickle cases demonstrate different outcomes, not two independent model qualifications. Software test counts are not benchmark scores.
Big Pickle built the offline evidence viewer across three review stages: 80, then 115, finally 145 passing local tests (1 skipped). Codex requested fixes twice before task-level approval; Windows and Ubuntu CI passed.
Read observed results →On 18 October, Big Pickle reported three fixes and cited 245 tests. Codex and file inspection found those changes were not present. Confirmed false completion claim; not a scored benchmark or blanket model verdict.
Read failure record →The successful feature needed independent Codex review and corrections; a separate task revealed a confirmed false completion report. Read the implementation evidence and the incident record together.
BEYOND THE BENCHMARK CATALOG
Research snapshot: 3 November 2026
The benchmark catalog was last updated on 1 October 2026; additional lab activities are summarized below. API access probes, framework calibration and unexecuted screening plans are not successful coding benchmarks or model qualifications.
The bounded implement → test → independent review → human gate runner has synthetic tests and preflight integration, but its source remains on an unmerged feature branch. Automatic execution is not release-approved.
See execution limits →The versioned catalog lists 33 models; a third-party directory supplies 82 leads across 30 providers. Five OpenCode candidates are planned for 39 screening slots, but 0 are runnable or executed.
See candidate evidence →The offline viewer was built and merged after Codex review. Sensitive local traces are not published; a known LOW issue in explicit --show-output redaction still requires hardening.
Read viewer case study →Harbor synthetic Oracle/Nop calibration returned 1/0, the importer passed 9/9 tests and DeepEval had 5/5 CI tests. Still no real model-screening or host-security approval.
See calibration results →On 2 November, an opt-in OpenRouter smoke attempted text and tool-call probes on three free endpoints. Results were HTTP 503 for one model and HTTP 403 for two. The workflow passed; the model probes did not.
See the exact model IDs and responses →Qwen3-Coder 30B passed all 9/9 mandatory conditions in a fresh holdout on the reference laptop. One passing run is useful evidence, not a reproducibility or blanket autonomy guarantee.
Review comparison and limits →Publication rule: a documented experiment, an API access attempt, a passed test, an independent review and a formal qualification are different evidence levels. Internal GitHub records are private; our public evidence summary describes the underlying claims and limitations. New runs are never promoted into the results catalog automatically.
METHODOLOGY
Local Agent Lab binds evidence to a concrete execution configuration:
Model × Quantization × Runtime × Harness × Configuration × Hardware × Environment
A single pass is evidence for one run, not proof of general reliability. A failed or incomplete run is also configuration-specific.