Test record: I ran 23 locally installed model deployments through ten executable coding tasks on
evox3, a 128GB GMKtec EVO-X3 with a Ryzen AI MAX+ 395 and Radeon 8060S. All 230/230 scored model-task requests completed.gpt-oss-120bled accuracy at 9/10. Qwen AgentWorld reached 8/10 in 7.2 seconds per task, making it the strongest interactive result. Qwen3.8-27B Q6_K reached 8/10 in 12.5 seconds per task; its Q8_0 build tied that score but was 8.27GiB larger and slower. The published context ceilings ranged from 131,072 to 1,048,576 tokens, but every comparable coding result used the same 32,768-token context. The model with the longest context was not the best coder, and the fastest decoder was not the most accurate.
A local model can fit in memory, advertise a million-token context and produce tokens quickly, yet still be the wrong model for everyday coding.
Those are three separate product questions:
- Does the first answer actually work?
- How long does the useful answer take?
- How much context can the model and this runtime realistically hold?
I had benchmarked parts of this portfolio while bringing several new models online. I reran the original 19-model matrix in one controlled pass after fixing the special runtime paths for Nemotron Puzzle and DeepSeek V4 Flash. I then added Qwen3.8-27B Q4, Q6 and Q8 plus a Qwen3.6-27B Q4 control with the same harness and deterministic profile. The comparison now covers 23 deployments and 230 scored executable answers. These four quant rows came from separate follow-up runs using the same harness and profile. They extend the matrix; they do not make it one uninterrupted 23-deployment run.
The primary deployment decision did not change. The added controls did change how I read the Qwen3.8 Q4 result.
The short answer
| Role | Model | Result | Why I would use it |
|---|---|---|---|
| Interactive default | Qwen AgentWorld 35B-A3B Q6_K | 8/10; 7.2s/task | Best latency at the 80% quality tier; production-configured at 262K context |
| Accuracy tier | gpt-oss-120b MXFP4 | 9/10; 21.2s/task | Only model to pass nine tasks; use for difficult work and final review |
| Fast draft | Qwen3-Coder 30B-A3B Q4_K_S | 5/10; 4.0s/task | Fastest completed answers, but the quality loss is large |
| Compact dense choice | Qwen3.8-27B Q6_K + MTP | 8/10; 12.5s/task | Equal accuracy to Q8 with 8.27GiB less storage and 23.5% lower task time |
| Long-context candidate to retain | Nemotron Puzzle 75B-A9B Q4_K_M | 6/10; 14.6s/task | Stable repaired runtime and a publisher-supported 1M ceiling that still needs local long-context qualification |
| Do not use as the default | Qwopus3.6-27B preview Q6_K | 5/10; 165.4s/task | Less accurate and more than nine times slower than base Qwen3.6-27B |
gpt-oss-120b is the quality winner. AgentWorld is the product winner for the
common path. That distinction matters more than declaring one universal
champion. Qwen3.8 Q6 is the compact dense deployment I would keep from these
local results, but the exact file provenance was not retained. It does not
displace either winner.
What the benchmark measured
The suite uses ten first-response tasks covering interval merging, dependency graphs, nested configuration updates, robust JSONL processing, bounded async concurrency, HTTP retry semantics, SQL window queries, path traversal defence, transactional inventory reconciliation and lazy iteration.
A response passes only when its extracted code passes every hidden executable test for that task. Generated code runs as a non-root user in a read-only Docker container with networking disabled, capabilities dropped and CPU, memory, process and wall-time limits.
Every row used:
- 32,768 context tokens;
- a 2,048-token output ceiling;
- temperature 0;
- top-p 1 and presence penalty 0;
- thinking disabled;
- one first response, with no repair turn.
The score is therefore a small diagnostic pass@1, not a claim to reproduce SWE-bench or a complete repository agent. It is deliberately strict and easy to audit: either the code passed or it did not.
All 23 deployment results, including maximum context
The context column is the maximum the publisher advertises or supports for the base model. It is not the context used by this coding run. Native and extended limits are shown separately where the publisher requires YaRN or another override.
| Model deployment | Maximum supported context | Accuracy | Wall s/task | Decode tok/s |
|---|---|---|---|---|
| gpt-oss-120b MXFP4 | 131,072 | 9/10 | 21.2 | 50.2 |
| Qwen AgentWorld 35B-A3B Q6_K | 262,144 | 8/10 | 7.2 | 50.7 |
| Qwen3.8-27B Q6_K MTP | 262,144 native; 1,000,000 extended | 8/10 | 12.5 | 25.3 |
| Qwen3.8-27B Q8_0 MTP | 262,144 native; 1,000,000 extended | 8/10 | 16.3 | 18.8 |
| Qwen3.6-27B Q6_K MTP | 262,144 native; 1,010,000 YaRN | 8/10 | 17.9 | 20.7 |
| Qwen3.6-27B Q4_K_M MTP | 262,144 native; 1,010,000 YaRN | 8/10 | 21.6 | 19.5 |
| Qwen3.5-122B-A10B GGUF | 262,144 native; 1,010,000 YaRN | 7/10 | 14.1 | 20.8 |
| Qwen3.8-27B Q4_K_M MTP | 262,144 native; 1,000,000 extended | 7/10 | 14.2 | 22.4 |
| Qwen3.5-122B-A10B GPTQ/vLLM | 262,144 native; 1,010,000 YaRN | 7/10 | 21.2 | — |
| Gemma 4 31B QAT Q4_0 | 262,144 | 7/10 | 22.8 | 11.4 |
| Qwen3.6-35B-A3B UD-Q6_K MTP | 262,144 native; 1,010,000 YaRN | 6/10 | 7.3 | 59.5 |
| Nemotron Puzzle 75B-A9B Q4_K_M | 1,048,576; default config 262,144 | 6/10 | 14.6 | 18.0 |
| Devstral Small 2 24B Q4_K_M | 262,144 advertised | 6/10 | 15.9 | 15.0 |
| Devstral Small 2 24B Q6_K | 262,144 advertised | 6/10 | 20.0 | 11.5 |
| Mistral Medium 3.5 128B Q4_K_M | 262,144 | 6/10 | 70.4 | 3.0 |
| Qwen3-Coder 30B-A3B Q4_K_S | 262,144 native; about 1M YaRN | 5/10 | 4.0 | 67.5 |
| Qwen3-Coder 30B-A3B Q4_K_M | 262,144 native; about 1M YaRN | 5/10 | 4.1 | 65.9 |
| Nemotron 3.5 Lightning 30B-A3B Q8_0 | 1,048,576 | 5/10 | 5.6 | 50.9 |
| Qwen3-Coder-Next Q5_K_M | 262,144 | 5/10 | 10.0 | 43.7 |
| Qwopus3.6-27B preview Q6_K | 262,144 native; 1,010,000 YaRN | 5/10 | 165.4 | 9.6 |
| DeepSeek V4 Flash ROCmFP3 mixed | 1,048,576 | 4/10 | 45.5 | — |
| DeepSeek Coder V2 MLA-IQ3_M | 131,072 advertised | 3/10 | 22.5 | 13.8 |
| Qwen3.5-0.8B FP16/vLLM | 262,144 | 0/10 | 14.3 | — |
There are two source discrepancies worth keeping visible. DeepSeek advertises DeepSeek Coder V2 as 128K, while its Hugging Face configuration and the local registration declare 163,840 positions. Devstral Small 2 is advertised and launched by Mistral at 262,144, while its current text configuration declares 393,216. I use the conservative publisher-advertised ceiling in the table rather than converting a larger configuration field into an untested promise.
The machine-readable result table contains the exact unrounded measurements, installed sizes and native/maximum context fields.
What the Qwen3.8 quantisation ladder changes
Qwen3.8-27B is a dense 27B model. I first tested Q4_K_M and then added Q6_K and Q8_0 builds after the 7/10 Q4 result appeared to trail Qwen3.6 at the same parameter count. All used Lemonade 11.5.2, the qualified ROCm llama.cpp path, the same 32,768-token test context and the same MTP speculative-decoding policy.
The published record does not contain the exact conversion repositories, revisions, filenames or checksums for these four follow-up deployments. It also does not include their per-task outputs or confirmation-run logs. The linked CSV supports the aggregate scores, timings and sizes below, but it is not enough to reproduce the exact deployments. A repeat intended to isolate bit width would need to record and publish those details.
| Qwen3.8 build | Installed GiB | Accuracy | Wall s/task | Decode tok/s | Load s | Failed tasks |
|---|---|---|---|---|---|---|
| Q4_K_M + MTP | 20.62 | 7/10 | 14.15 | 22.44 | 3.53 | retry_after, safe_join, lazy_chunks |
| Q6_K + MTP | 21.31 | 8/10 | 12.46 | 25.28 | 12.87 | safe_join, lazy_chunks |
| Q8_0 + Q8_0 MTP | 29.58 | 8/10 | 16.27 | 18.83 | 13.38 | safe_join, lazy_chunks |
Of the three local Qwen3.8 deployments recorded here, Q6 is the operating choice. It matches Q8’s task-level result while using 8.27GiB less installed storage, decoding 34.3% faster and taking 23.5% less end-to-end time per task. Q8 did not recover another edge case; it produced the same two failures as Q6. More bits did not make this small deterministic suite more accurate.
Q4 contributed to the original one-task deficit: its retry_after answer
returned early for every non-digit value, making its later HTTP-date parser
unreachable. Q6 and Q8 both handled that case. This is still a deployed-build
comparison rather than a laboratory isolation of bit width: the Q4 and Q6
files came from different conversion publishers, and their MTP drafts are
packaged differently. Without the missing file-level provenance, the conclusion
applies to these local deployments rather than Qwen3.8 quantisations generally.
The more important control was matched precision. Qwen3.6 Q6 and Qwen3.8 Q6 both scored 8/10, so the original 8/10 versus 7/10 was not evidence that the newer model is generally worse. Qwen3.8 Q6 was also faster here: 12.5 rather than 17.9 seconds per task. Qwen’s broader full-precision model-card results place 3.8 ahead on several coding evaluations; those publisher results use different prompts, sampling and agent scaffolds, so they are supporting context rather than a direct comparison to this ten-task harness.
The remaining safe_join difference looks like version behaviour, not quant
damage. Both Qwen3.8 Q6 and Q8 reject every raw .. component. The hidden test
accepts an internal a/../b segment after safe normalisation, which both
Qwen3.6 quants allow. Rejecting lexical traversal literally is defensible, so
this also exposed an ambiguity in my prompt. All four Qwen3.6/3.8 Q4/Q6 control
builds missed lazy_chunks by delaying invalid-size validation until generator
iteration.
Accuracy and decode rate did not move together
The fastest decoder was Qwen3-Coder 30B-A3B Q4_K_S at 67.5 tokens per second. It passed only five tasks. Qwen3.6-35B-A3B reached 59.5 tokens per second and passed six. Nemotron Lightning reached 50.9 and also passed five.
gpt-oss-120b decoded at 50.2 tokens per second—slower on paper than all three
of those models—and passed nine tasks. Its wall time was still 21.2 seconds per
task because it emitted an average of 1,020 output tokens. AgentWorld decoded
at a similar 50.7 tokens per second but emitted 342, returning in 7.2 seconds.
That is why decode throughput alone is a poor interactive product metric. A model that writes three times as much can have the same token rate and feel three times slower. A model that emits a short wrong answer can look excellent on a throughput chart.
For the normal coding loop I would take AgentWorld’s 8/10 at 7.2 seconds. For a hard review where another correct task matters more than fourteen additional seconds, I would explicitly load gpt-oss.
The backend and quant still matter
The two Qwen3.5-122B deployments used the same base model and both scored 7/10. The GGUF/llama.cpp route finished in 14.1 seconds per task and loaded in 34.6 seconds. GPTQ/vLLM needed 21.2 seconds per task and 100.1 seconds to load. On this machine, GGUF is the better single-user 122B path.
Devstral made the quantisation decision even clearer. Q4_K_M and Q6_K passed the same six tasks with the same pass pattern. Q4 used 4.67GiB less installed storage, decoded at 15.0 rather than 11.5 tokens per second and finished about four seconds sooner. The larger quant bought nothing measurable in this suite.
The two original Qwen3-Coder quants also tied at 5/10. Q4_K_S was slightly faster and smaller. Once correctness is tied, the operational choice becomes easy.
The Qwen3.8 ladder shows why quant choice should be measured instead of inferred from bit width. Q6 was both more accurate and faster than Q4 in this run, while Q8 was larger and slower without improving accuracy. Different output lengths, conversion sources and MTP packaging mean these are operational comparisons of complete deployments, not pure quantisation microbenchmarks.
Maximum context is a capability claim, not a free feature
The table contains several million-token models. That does not mean I tested a million-token coding prompt on every one of them, or that a 128GB workstation can allocate each advertised window with the chosen cache precision and still remain useful.
There are at least four different numbers that often get collapsed into “context length”:
- the model’s native trained or configured window;
- a publisher-supported extension using YaRN or another RoPE override;
- the context the serving runtime is configured to allocate;
- the number of input and output tokens a specific test actually consumes.
This benchmark used 32,768 for number three and much less for number four. AgentWorld separately reproduced its 8/10 result with the server allocated at 262,144, which supports keeping that production configuration. It was not a 262K-filled prompt test.
Qwen3.5 and Qwen3.6’s 1,010,000-token modes require explicit YaRN configuration, while Qwen3.8 publishes a 1,000,000-token extended ceiling. Nemotron Puzzle supports 1M even though its default Hugging Face configuration is 256K. NVIDIA’s Nemotron Lightning card makes the hardware qualification explicit: the model supports 1M, while a single 80GB H100 BF16 recipe is limited to 256K by memory. The model ceiling and the deployable ceiling are not always the same thing.
I would therefore treat context as a workload-specific qualification. Load the
intended cache precision, fill the actual prompt range, measure prompt
processing and memory, verify retrieval quality near the end of the window,
then run a soak. A successful server start at --ctx-size 1048576 is only the
first check.
The two repaired models now belong in the comparison
Nemotron Puzzle and DeepSeek V4 Flash initially failed for different reasons.
Puzzle’s GGUF contains per-layer expert widths and top-k values plus an embedded two-layer MTP head. The general llama.cpp build expected scalar expert metadata. I built the exact Puzzle/MTP revision pinned by the quantisation manifest and routed only that model to it. Puzzle then completed the full suite twice with the same 6/10 pass pattern.
DeepSeek V4 Flash uses a custom ROCmFP3 mixed tensor format that general llama.cpp does not understand. The existing qualified Lucebox backend could load it, so Lemonade now dispatches that filename to the specialised runtime. It completed all ten requests, scoring 5/10 in the first repaired run and 4/10 in the full repeat. The runtime problem is fixed; the task-level variance and 45.5-second latency still make it a poor default for this workload.
This is the right separation of concerns. “It loads” is a runtime result. “It passed four tasks” is a model result. Both belong in the record.
The repeat and extension were reassuringly boring
Seventeen of the eighteen models with a previous complete ten-task result
reproduced their accuracy exactly. DeepSeek V4 Flash was the only movement,
losing the retry_after task. Qwopus had previously been stopped after a
partial result; this time it completed at 5/10.
Qwopus also demonstrated why completing the row mattered. It averaged 1,573 output tokens and 165.4 seconds per task. Base Qwen3.6-27B passed eight tasks in 17.9 seconds. The preview’s advertised long context is real metadata, but it does not rescue the observed coding product.
The retained run summaries record the same task-level result for Qwen3.8 Q4 in warm and cold runs. They also record matching 8/10 task matrices and matching completion-token counts when Qwen3.8 Q6 and Qwen3.6 Q4 were rerun from cold. Those summaries are not part of the published dataset, so the public table uses one aggregate result per deployment and does not count the confirmation runs in the 230 scored requests.
Across the controlled matrix and its four quant extensions, all 23 deployments loaded, all 230 scored task requests completed and the evaluator reference implementation passed. The original matrix automatically restored AgentWorld; after the extensions I explicitly unloaded the tested Qwen deployment to return Lemonade to an idle state.
What I would operate
The portfolio I would keep is smaller than the benchmark matrix:
- Qwen AgentWorld 35B-A3B Q6_K as the interactive default;
- gpt-oss-120b MXFP4 as the explicit accuracy and review tier;
- Qwen3-Coder 30B-A3B Q4_K_S only when rapid drafting is worth a large measured quality trade-off;
- Qwen3.8-27B Q6_K MTP as the compact dense choice; Q8 bought no measured accuracy and Q4 lost a fragile protocol edge case;
- Qwen3.5-122B GGUF for Qwen-specific 122B work and long-context qualification;
- Nemotron Puzzle as a retained specialist while its isolated runtime is required;
- every other row as evaluation inventory rather than an automatically routed production choice.
I would not make the million-token number the routing policy. I would route on the actual task: interactive edit, difficult verification, long document or model-specific evaluation. Context, correctness and response time remain separate controls.
The useful outcome of testing 23 deployments is not that the workstation should serve all 23. It is knowing which few deserve to be easy to reach—and why.
Sources checked 14 August 2026: the linked publisher model cards and current Hugging Face configurations for Qwen, OpenAI gpt-oss, NVIDIA Nemotron, Google Gemma, Mistral, DeepSeek and Qwopus. Context ceilings are publisher/configuration claims unless explicitly described as an EVO-X3 test. Benchmark figures come from the deterministic 19-model rerun and same-profile Qwen Q4/Q6/Q8 extensions completed on 14 August 2026. The published CSV contains aggregate results; the follow-up deployments’ exact file provenance and task-level logs were not included and remain a limit on independent reproduction.