FIELD REPORT #001

Which Local Coding Agent Can I Actually Trust With My Repository?

First results from Local Agent Lab

Public coding benchmarks answer an interesting question: how well can a model generate code or solve programming tasks?

For practical use, that is not enough for me.

I want to know:

Which local agent can I trust with which task under which technical conditions?

That is the question behind Local Agent Lab.

I am not testing the language model in isolation. The actual unit of evaluation is the complete execution configuration:

Model × Quantization × Runtime × Agent Harness × Configuration × Hardware × Environment

A model can look strong on paper and still fail inside a real repository — because of tool use, shell behavior, context management, hardware limits, or weak recovery after errors.

The test system

The current runs use ordinary consumer hardware:

• NVIDIA RTX 5070 Laptop GPU
• 8 GB VRAM
• 64 GB RAM
• Windows 11
• Ollama
• OpenCode
• a 32K context profile for the agent runs discussed here

The focus is not on isolated code snippets. An agent is expected to complete a real repository task autonomously:

Understand repository → find root cause → use tools → make a minimal change → run tests → recover from errors → complete the task

The first fresh holdout: Qwen3-Coder 30B

For B2-002 I prepared a small but real regression task. The bug involved tag canonicalization: "Agent" and "agent" had to remain distinct values, while exact duplicate collapse, whitespace handling, and sorting still had to behave correctly.

The important part was the methodology. The agent saw only the task and the frozen defective repository state. The hidden oracle stayed unavailable until the autonomous run had finished.

Qwen3-Coder 30B identified .casefold() as the root cause and corrected the production logic with a minimal change.

An independent evaluator then checked nine mandatory conditions.

Result: 9 out of 9 conditions passed.

That is strong evidence for this run, but it is deliberately not a blanket endorsement. One successful run shows that this execution configuration solved this task. It does not prove general reliability.

Qwen3.8: interesting in theory, impractical on the reference system

Next I tried Qwen3.8 27B with the same 32K profile.

The model loaded, but Ollama showed approximately:

73% CPU / 27% GPU

With only 8 GB of VRAM, a large part of the model had to run outside the GPU.

In practice, the agent did not make useful progress. I stopped the attempt manually.

That is explicitly not a benchmark FAIL. It is an Operational Observation:

This specific Qwen3.8 configuration was not practically usable for agentic work on this hardware.

That distinction matters. I am not trying to claim that the model is bad. I want to know whether a model is realistically usable on the hardware I actually have.

North Mini Code: fast enough — and still unsuccessful

North Mini Code 1.0 was more interesting.

The model responded much more quickly and OpenCode could work with it. The B2-002 comparison run finished after roughly 201 seconds.

But a completely different failure mode appeared.

The agent repeatedly tried Unix-style shell commands inside a PowerShell environment, for example:

find ... | xargs grep ...
cd ... && ...
grep ... || ...

PowerShell returned errors.

The agent saw those errors, but it did not adapt its strategy effectively enough. More variants failed. Eventually the full 32K context window was consumed.

The final state was:

• no code change;
• required test still failing;
• no final report;
• context exhausted.

That is more interesting to me than a simple red benchmark cell.

The agent did not fail because it could not run at all. It failed because of environment understanding, tool use, recovery behavior, and context management.

The benchmark exposed a weakness in the benchmark itself

The North Mini run also revealed a gap in Local Agent Lab.

The runner initially reported only:

TEST_FAILURE_UNRESOLVED

But the raw evidence also contained:

step_finish.reason = "length"

The stronger mechanical description was therefore:

CONTEXT_EXHAUSTED → TASK_INCOMPLETE

That observation led directly to a runner-hardening patch.

This is how I want the lab to evolve: not by pretending the evaluation system is perfect from day one, but by improving the measuring instrument whenever real agent behavior exposes a blind spot.

Three models, three different kinds of evidence

The current results do not fit cleanly into a simple leaderboard.

That is why there is intentionally no composite score yet, and no claim that one model is “the best local coding model.”

What I actually want to measure

1. Capability

Can the agent solve the task at all?

2. Reliability

Can it do so repeatedly, completely, and without human intervention?

3. Trust

Does it respect boundaries, permissions, and governance rules while doing the work?

An agent that produces the correct patch but reads prohibited secrets or leaves its allowed workspace would not count as a successful agent for my purposes.

That is the next stage of Local Agent Lab.

The real question

In the end, I am not interested in whether one model scores two points higher on a public leaderboard.

I am interested in this:

Can I give this agent a real development task, let it work autonomously, and trust both the result and the way it behaved?

Answering that requires more than coding scores.

That is what Local Agent Lab is designed to explore, step by step.

Methodology note: These results apply to the concrete execution configurations and reference hardware described above. B2-002 was a fresh holdout only for the first Qwen3-Coder run. Later runs on the same task are comparison runs. The Qwen3.8 attempt was manually stopped and is therefore not classified as PASS or FAIL.