Qwen3.8 Flash Next: Vulkan 0.7.1 and Q8 in production

Test record: I restarted the preferred 103.7 GiB Qwen3.8 Flash Next service on my 128 GB Ryzen AI MAX+ 395 workstation, kept its two 262K slots and explicit prompt cache, then qualified a Vulkan 0.7.1 userspace bundle with Q8 K/V storage. The production result improved cold prompt processing by 8.5% and cached two-request throughput by 5.1%. It completed a 30-minute mixed soak without a wrong answer, request leak, failed recovery or AMD GPU fault. The former fast profile was not used as a fallback.

I did not want this test rescued by another model. The question was narrower: could the preferred Qwen3.8 Flash Next route restart with two slots, restore cached prefixes correctly and remain the service behind Lemonade?

It can. The profile now running on evox3 uses Q8 rather than F16 K/V storage and a v0.7.1 Vulkan bundle. The gain is not spectacular, but it is broad: cold prefill, repeated decode and simultaneous requests all moved in the right direction. More importantly, the memory shape survived the soak that the F16 version could not start cleanly.

What changed in the production profile

This follows my earlier ROCm-versus-Vulkan Qwen3.8 work. The model is still Unsloth’s four-shard Qwen3.8-Flash-Next-UD-Q4_K_XL, and the unusual 26.82 GiB per-layer token-embedding table still remains CPU-backed. I did not change the model quantisation or increase the 40-layer GPU offload.

The served profile is now:

total context          524,288 tokens
parallel slots         2
context per slot       262,144 tokens
GPU-offloaded layers   40
batch / micro-batch    512 / 128
K/V cache              Q8_0 / Q8_0
host prompt cache      4 GiB
prompt caching         enabled
idle-slot caching      disabled
model loading          mmap
speculative decoding   disabled

The executable reports llama.cpp build 10637 at commit 39817c47; the previous production path reported build 10707 at 250b61446. Those build numbers come from different source lines, so I am not treating the smaller number as an upgrade by itself. The item under test was the complete compatible runtime: the experimental Qwen4Exp implementation, the v0.7.1 Vulkan userspace bundle and the Q8 cache configuration.

The cache representation still mattered to qualification. The first v0.7.1 run retained the F16 K/V cache and failed with Not enough memory for command submission, followed by a Vulkan device loss. The v0.7.1/Q8 candidate then completed the same two-slot test shape and peaked at 70.89 GiB of GTT.

A separate two-slot F16/lazy control on build 10707 peaked at 77.31 GiB, while minimum available system memory was 38.53 GiB rather than the Q8 candidate’s 46.12 GiB. That is useful directional evidence, but it is not a matched v0.7.1 F16 comparison. The runtime builds differ, and the failed v0.7.1/F16 run did not produce a peak-GTT result. I therefore cannot attribute the 6.42 GiB difference to Q8 alone or prove that K/V precision alone caused the first run to fail.

The operational conclusion is narrower: v0.7.1/F16 failed under the tested profile, while v0.7.1/Q8 completed the qualification. I deployed the complete configuration that passed.

The new baseline

I compared the previous preferred service with the promoted profile on the same machine. Both used two slots, 40 offloaded layers, the same model and the same 4 GiB host prompt cache. The material difference was the runtime package and K/V representation.

The improvement also appeared in the small developer workload set. The three repeats for each case all passed on the candidate:

The old Python record failed its strict formatter because the answer included Markdown fences. I corrected that validation before the new run, so the table is a performance comparison, not evidence that the runtime alone repaired code formatting.

The production comparison data contains the unrounded measurements and test boundaries.

Cache hits survived slot changes and divergent prompts

A quick identical repeat is not enough to qualify prompt caching. It can hide a same-slot shortcut and says nothing about what happens after the other slot has been used.

The production gate began with an uncached 3,149-token prompt. It took 28.609 seconds end to end. The immediate repeat restored 3,145 tokens and completed in 0.697 seconds. After a different 3,388-token history had occupied the other path, the first prefix restored from the bounded RAM cache and completed in 0.683 seconds.

I then changed only the suffix, returned to the original request and alternated the two histories for another 20 cycles. Every answer was exact. Every cycle restored at least 3,145 prompt tokens. The measured prompt-processing speed-up was 148.9 times for the immediate hit and 159.3 times after RAM restoration; those figures refer to prompt work, not total generation speed.

The simultaneous test sent two different exact-output requests through Lemonade at the same time. Both began together, both returned their own marker sequence and both received a cache hit on the repeated pair. That matters more than a higher single-request number: the service is intended to accept two independent clients without allowing one response to appear in the other.

Two slots do not create one 524K request. /slots reports two independent 262,144-token allocations.

The half-hour run found one problem in the test, not the server

The extended run lasted 1,820.47 seconds. It completed ten structured-JSON iterations, ten isolation iterations, ten cache pairs, ten cancellation and recovery iterations, and three long prose requests. The long requests each used 31,012 prompt tokens and generated 768 tokens.

The raw soak runner initially labelled those three requests as failures because it still contained a 17 tokens/s threshold written for a smaller speculative ROCmFPX model. That was the wrong comparator for a 103.7 GiB non-speculative model whose matched production baseline was 11.86 tokens/s. I did not edit the raw result to make it green. I retained the three threshold failures, added a separate adjudication and required every functional check plus a faster same-model result. Median long-output generation was 12.29 tokens/s, 3.6% above the matched baseline.

After the soak, the two-request check reached 19.00 aggregate tokens/s uncached and 25.24 tokens/s cached, with both answers correct. Across 1,832 resource samples, available memory did not fall below 45.64 GiB and GTT did not exceed 70.91 GiB. The kernel capture contained no matching AMD GPU reset, page fault, device loss or SVM mapping failure.

The soak summary records the raw threshold outcome alongside the production-specific adjudication. That distinction is deliberate.

The tempting alternatives did not make the default

I tested four nearby options before promoting this one.

First, --tensor-read-lazy on on the old build did not release useful memory. Cold prefill fell from 103.76 to 95.03 tokens/s, while uncached and cached two-request throughput also slipped. I rejected it.

Second, the newer EngramHalo Vulkan source at commit 930918c worked with Q8. Its ordinary decode was marginally faster, but prompt processing on the three developer workloads was 8–10% behind the v0.7.1 package. It was not a broad replacement.

Third, adding the Q8 MTP sidecar to that branch produced 26.78 tokens/s on structured JSON and 45.36 tokens/s on Python code. Prose went the other way: draft acceptance fell to 61.7%, and generation dropped to 14.44 tokens/s from 17.92 tokens/s without MTP. That is a useful specialist result, not a safe general default.

Finally, the EngramHalo ROCm 7.14 build failed even as a one-slot Q8 profile. The kernel repeatedly reported SVM mapping failed, exceeds resident system memory limit, including with offload limited to 40 layers. The branch’s suggested workaround was to disable IOMMU. I had already tested that system change: it delivered little performance and removed functionality I need, so I did not weaken the host configuration to make this model start.

The candidate decision record keeps the positive and negative results together.

Moving it behind Lemonade

The production change is release 20260828-v13-qwen4exp-v071-q8. Lemonade 11.8 remains the public service on port 13305; llama.cpp listens only on 127.0.0.1:8001. There is no separate port 18802 service.

The guarded launcher now refuses to start if another llama-server is resident or if model memory has not cleared. While the model is running, it records available memory and GTT once per second. It stops the backend if available memory falls below 18 GiB, GTT exceeds 90 GiB or a new AMD GPU memory fault appears.

The first deployment attempt proved the rollback rather than the model. My production cache script omitted the model field accepted by direct llama.cpp but required by Lemonade, so the router returned HTTP 400. The deployment guard restored the previous preferred profile automatically. I corrected the request, repeated the cutover and only marked the release passed after the cache, concurrency, executable-identity and kernel checks completed.

llm, llm-exact and Jarvis now resolve to the new preferred profile. The older fast profile remains registered under its explicit llm-fast name, but it was not loaded during qualification or deployment and normal traffic does not fall back to it.

Where I would stop claiming

This test proves the two-slot allocation, cache behaviour, concurrent request isolation and a 31K-plus-output workload for 30 minutes. It does not prove two simultaneous near-262K prompts, a 24-hour memory trend or identical behaviour under unrelated desktop memory pressure. Q8 K/V storage also changes cache precision, so the exact-output and developer checks are evidence for these workloads rather than a universal quality result.

For the general Qwen route, I would keep the v0.7.1/Q8 profile now in production. It completed the preferred dual-slot qualification, used less GTT than the separate F16 control and added roughly 4–8% where it matters, without changing how clients reach Lemonade. I would revisit the MTP branch only as an explicit high-acceptance coding profile, after a longer mixed-output qualification.

Benchmark and deployment record: 28 August 2026. GMKtec EVO-X3; AMD Ryzen AI MAX+ 395 / Radeon 8060S; 128 GB unified memory; Lemonade 11.8.0; Qwen3.8 Flash Next UD-Q4_K_XL; llama.cpp build 10637 at 39817c47; Vulkan v0.7.1 userspace bundle. See the qualification manifest for hashes, limits and retained-result locations.