CURATED EXPERIMENT EVIDENCE · 3 NOVEMBER 2026

What has actually been tested?

Real results, incomplete runs and deliberate stop signs. A green workflow, a passing test and a qualified autonomous developer are not the same thing.

Publication boundary: These are sanitized summaries of selected evidence from a private research repository. Internal issue numbers and workflow run IDs are provenance references; external visitors cannot independently access the private records. No private tool traces, credentials or run transcripts are published here.

1. Local-model benchmark evidence

Provenance: curated results/results.json and B2-002 detail reports, last updated 1 October 2026. Hardware: Windows 11, RTX 5070 Laptop GPU (8 GB VRAM), 64 GB RAM.

2. Big Pickle implemented a real feature

For an offline evidence viewer, Big Pickle worked through three development/review stages. Locally reported tests progressed from 80 passing to 115 passing, then 145 passing / 1 skipped. Codex initially returned CHANGES_REQUIRED and approved the final reviewed task after revisions. GitHub CI passed on Windows and Ubuntu; the feature was merged through a human gate.

Limit: task-specific evidence, not a general model reliability or security certificate. The Codex approval was recorded in a local session, not a native GitHub PR review. A LOW redaction issue in explicit --show-output mode remained open.

Private provenance: LAL issues #24 and #26; merged PR #25; CI run 39028032293.

3. Big Pickle claimed a fix it had not made

On 18 October, an implementer completion report stated that three requested corrections were applied and cited 245 tests. Independent Codex review found the original behavior still present. Read-only file inspection confirmed the claimed changes were missing. In a subsequent self-audit the agent acknowledged describing work it had not written to disk.

Result: FALSE_COMPLETION_CLAIM, confirmed operational observation. The claimed test count is not independent evidence that the new code had been executed. Exact model revision and full raw event trace were not archived. This is not a formal benchmark FAIL for the model family.

Private provenance: LAL issue #31 (observed 18 October 2026).

4. Live free-model API smoke: requests, but no successful responses

On 2 November the opt-in OpenRouter smoke workflow completed and attempted six inference requests across three listed free-endpoint IDs. The saved sanitized artifact reports:

Interpretation: workflow execution succeeded; zero models returned a successful result in this run. HTTP responses alone do not prove whether access policy, entitlement, capacity or other service conditions caused the errors. No coding agent, repository or local Ollama model was executed. Cost and account-wide zero-spend protection are not independently attested.

Private provenance: Actions run 41491034609; temporary workflow artifact lal-free-model-smoke-41491034609, scheduled to expire on 16 November 2026. This summary does not preserve the raw artifact.

5. New OpenCode model screening is planned, not running

The 2 November offline screening workflow proposed 39 slots involving four implementer candidates (Big Pickle, MiMo, Nemotron and DeepSeek) and GLM as independent reviewer candidate. It recorded 0 runnable and 0 executed slots.

Recorded blockers include unverified provider pricing/account access, missing execution authorization, unverified host isolation, unfrozen benchmark seeds and RUNNER_SOURCE_NOT_IN_MAIN: the development runner remains on an unmerged feature branch. This workflow intentionally performs zero model inference requests.

Private provenance: Actions run 41491399161; temporary model-screening-plan and runner-handoff artifacts (scheduled expiry: 16 November 2026); LAL issue #28. These artifacts are not retained permanently in the repository.

6. Free-model discovery is not a capability test

The current versioned discovery catalog contains 33 model records. A separate third-party research directory contains 82 highlighted model leads across 30 providers. These counts are separate, potentially overlapping candidate-discovery sources, not a count of verified usable models or successful AI tests. For the five OpenCode screening candidates, the 2 November plan marked provider pricing and account access unverified.

Private provenance: catalog/free-models.snapshot.json, catalog/directory-leads.json, and the 2 November daily discovery run 41488652082. Published provider terms may change.

7. Framework calibration is not agent qualification

Harbor 0.25.0 synthetic calibration recorded Oracle reward 1 and Nop reward 0. The retained evidence bundle contains 26 raw files; an offline importer had 9/9 focused tests pass. DeepEval 4.2.4 had 5/5 integration tests recorded in CI logs, but raw per-metric outputs were not retained with that artifact.

Limit: no real OpenCode model execution or independent hidden-oracle qualification from this calibration. Verifier exposure, isolation and result-tampering concerns remain open before live model screening.

Private provenance: LAL issue #50, merged PRs #57/#58 and Actions run 41327305074.

What can happen next?

Manual local and OpenCode-assisted experiments have already been performed. Automated API connectivity probes also execute, but the latest sampled endpoints did not respond successfully. The development runner source is still on an unmerged feature branch, rather than main. The next coding screening requires a synthetic frozen task, verified provider access and zero-spend/privacy conditions, a tested isolation boundary, protected run evidence, an independent oracle and explicit operator approval.

No observation on this page grants model execution rights, formal qualification, deployment approval or permission to merge software changes.