Strata vulkan native expert streaming vs llama.cpp Vulkan on an Intel Arc B390 iGPU

Measured 19 November 2026 on one machine, one model, one day. Short version: on an Intel Arc B390 iGPU, llama.cpp Vulkan is about 14× faster at decode and batches a realistic prompt about 870× faster at prefill, but Strata's native streaming engine keeps the footprint at about 3.5 GiB of shared RAM instead of 20 GiB. That is the trade: speed versus fitting a 35B MoE in a 30 GiB machine with room left over.

5-run benchmark: throughput, startup latency, memory and the per-token trajectories

The "Strata native" engine here is not in a released Strata package. It is an opt-in research-preview path on the vulkan_arc branch of the grigio/Strata fork, built from a source checkout. The branch also carries the llama.cpp Vulkan "bridge" used for the comparison. See why and how for how to get it and when to pick each path.

Why this test is different

An earlier version of this comparison used a 7-token prompt, 64 generated tokens and three runs. That test was too thin, and this one fixes three things it got wrong:

  • A real prompt and a real answer. Both engines now get the same fixed 418-token prompt (a paragraph about MoE routing) and generate 256 greedy tokens into a 2048-token context, with 8 threads and temperature=0. The old 7-token prompt measured almost no attention context; the new one makes each decode step pay for a growing KV cache.
  • 5 independent runs per engine, each a fresh process, reported as mean ± SD with medians where it helps. Each engine turned out to be deterministic across its own five runs, so the spread is between engines and across conditions, not within one engine's sampling.
  • Per-token telemetry and memory sampled throughout. The engines print one T line per token and flush it, so I timestamped the lines to get a token-by-token trace. Memory was sampled at 0.2 s intervals from /proc/meminfo, /proc/<pid>/status, smaps_rollup and the Vulkan /dev/dri mappings. The short test took a session-ready snapshot, which missed what happens during decode.

The result is that several headline numbers from the short test change — some by a lot. They are reported below with the conditions that produced them.

Machine and software

CPU Intel Core Ultra X7 358H (Panther Lake), 16c/16t
GPU Integrated Intel Arc B390 (PTL), Mesa ANV Vulkan (driver 26.2.6, Vulkan 1.4.354, UMA = 1, fp16/bf16, coopmat)
Memory 30 GiB system RAM + 15 GiB zram; ~26 GiB available at the start
Storage YMTC PC411-1TB-D NVMe (LUKS/btrfs, zstd:1)
OS CachyOS (Arch), kernel as installed 2026-11-19
Model /home/g/Code/llama.cpp/models/Qwen3.8-35B-A3B-Q4_K_M.gguf — 20.21 GiB / 35.51 B params, 41 layers, 256 experts per layer, top-8 routing
Strata grigio/Strata branch vulkan_arc, HEAD 678c9dbf1fc66feb3f7fcb2138e525da8436a7e7, clean tree; native engine terminal/engine/vulkan/engine/strata-vulkan-engine built 2026-11-19 18:40
llama.cpp b1386-a956a852c (build 1386), Clang 23.1.4; libggml-vulkan.so in /home/g/Code/llama.cpp/build/bin

On this iGPU the DEVICE_LOCAL Vulkan heap is system RAM. There is no dedicated VRAM; "GPU memory" below is shared system memory mapped from /dev/dri/renderD128.

What I compared

  • Strata native streaming Vulkan — the default strata-vulkan-engine. Dense weights (1.56 GiB, 630 tensors) stay resident; the 256 experts per layer are read from the GGUF on the NVMe only when the router asks, into a bounded GPU slot pool (2048 slots here). Router, combine and sampler are Vulkan kernels. Forward passes are one token at a time.
  • llama.cpp Vulkan through Strata's bridge — strata-llama-vulkan-engine driving the same libggml-vulkan.so from /home/g/Code/llama.cpp/build/bin over Strata's line protocol. All weights are resident; the prompt is batched with n_batch=512. This keeps the prompt, decoder, timing lines and memory measurement identical between the two engines.

Both got the same 418 token ids, produced by Strata's reference tokenizer from the prompt text with the trailing newline removed. Both were asked for 256 tokens with temperature=0, 2048 context, 8 threads, no warmup. Timings come from the engines' own PP and DONE lines, so HTTP/server overhead is excluded.

How memory was measured

ANV maps Vulkan device memory as rw-s /dev/dri/renderD128 pages. They are system RAM but are not in process VmRSS. I measured:

  • Non-reclaimable RAM — the drop in system MemAvailable while the model was loaded, sampled throughout the run. This is the number that reflects "how close am I to OOM".
  • Vulkan / shared GPU RAM — the sum of the process's rw-s /dev/dri/renderD128 map sizes from /proc/<pid>/maps, sampled every second.
  • Peak RSS and smaps_rollup — from /proc/<pid>/status and /proc/<pid>/smaps_rollup, recorded for completeness. Both are misleading here: Vulkan's device maps are invisible to RSS, and the native engine's GGUF pages are file-backed and reclaimable.

Results (5 runs each)

All numbers are mean ± SD over five fresh processes; medians in parentheses.

Metric Strata native (streaming) llama.cpp Vulkan (bridge) Difference
Decode, 256 tokens 2.12 ± 0.02 tok/s (2.13) 30.47 ± 4.83 tok/s (30.30) llama.cpp ~14×
Prefill, 418 tokens (PP line) 3.4 ± 0.2 tok/s (3.4) 2,928 ± 669 tok/s (2,842) llama.cpp ~870×
Time to first token 125.3 ± 6.3 s 11.6 ± 1.0 s ~11× faster
Model load (READY) 0.6 ± 0.2 s 10.9 ± 1.0 s —
Non-reclaimable RAM 3.54 ± 0.06 GiB 20.28 ± 0.27 GiB Strata ~5.7× less
Vulkan / shared GPU RAM 3.37 GiB (all 5) 20.33 GiB (all 5) Strata ~6× less
Peak process RSS 0.04 GiB 0.12 GiB —
Peak system RAM used 5.77 ± 0.03 GiB 22.27 ± 0.06 GiB Strata ~3.9× less
Wall time per 256-token run 246.2 ± 7.4 s 19.9 ± 2.1 s —

Per-run decode tok/s: native 2.08 / 2.11 / 2.13 / 2.14 / 2.13 — remarkably tight, SD 0.02. llama.cpp 27.91 / 30.30 / 24.52 / 37.43 / 32.18 — SD 4.83, over 200× the spread of the native engine. Neither engine hit EOS; all ten runs finished all 256 tokens on length.

What changed from the short test, and why

  • Decode dropped for both. The old 7-token/64-token test reported 4.76 tok/s native and 38.24 tok/s bridge. Now it is 2.12 and 30.47. The 418-token prompt leaves a much larger KV cache, so every decoded token does more attention work. The native engine also had a warmed 64-token window in the old test; here there is no warmup.
  • Prefill says much more about llama.cpp. The old test timed 7 tokens one way and got 294 tok/s. With 418 tokens the bridge batches them into a single llama_decode call (n_batch=512) and reaches ~2,928 tok/s. The old "~51× faster at prefill" is not a number that describes a realistic prompt.
  • The bridge's PP line is optimistic. After PP is printed there is still a ~0.5 s gap before the first T appears — a GPU sync the engine's own prefill timer does not include. The native engine has no such gap (first T ~0.7 ms after PP). For wall-clock comparisons use time to first token: native 125 s, bridge 11.6 s (of which 10.9 s is model load, so only ~0.64 s from READY to first token on llama.cpp).

What the per-token trace adds

The 256-token average hides the shape of each run. The trajectory panel in the chart plots all five runs of both engines on a log scale:

  • Native slows down as it goes. Its per-token time grows from ~424 ms over the first 64 tokens to ~527 ms over the last 64 — about 2.36 tok/s falling to 1.90 tok/s. This is a warm-down, not a warm-up: the KV cache grows from 418 to 674 positions and dense grouped-query attention costs more each step. The expert cache does not rescue it. VSTAT was identical in all five runs: hits=581103 misses=362497 prefetches=0 mb=231181.8 — a 61.6 % hit rate and roughly 231 GB of GGUF reads across 674 forwards, sustained at about 1 GB/s from the NVMe. I did not independently verify what the mb counter includes.
  • llama.cpp is fast but hiccups. Its steady token is 24.7 ms (~40 tok/s). Four of five runs had 2–7 stalls of roughly 0.4 s each (one run had none), which is where the SD of 4.83 tok/s comes from. I did not determine the cause of the stalls; they look like queue or sync pauses, not memory reclaim — MemAvailable stayed stable in every run.

The greedy streams are not identical

Both engines are deterministic: each run of the same engine matched its own first run on all 256 tokens. But the two engines agree for only the first 48 tokens. At token 49 the native stream emits an extra 10885 where llama.cpp emits 310; from there the summaries differ ("Routing tokens to the wrong…" on native vs "Routing to the wrong expert…" on llama.cpp). This is almost certainly a near-tie argmax flipping under tiny numerical differences, the same cascade the branch's docs warn about. The branch ships compare_greedy.py and an EVAL teacher-forcing gate for checking this; I did not rerun them for this report. So the throughput comparison is fair — same prompt, same decoding method — but the two engines do not give the same answer over a 256-token answer.

The vulkan_arc branch: why and how

The statements in this section come from the branch's docs/VULKAN.md, its README and the local git log — read, not measured for this report. The numbers in Results are mine.

Git-wise the branch is upstream v0.2.16 (64524fb9) plus one commit, 678c9db ("[Engine] opt-in Vulkan expert-streaming path for the Arc B390 iGPU"), which adds terminal/engine/vulkan/, docs/VULKAN.md and the --backend vulkan launch flags. There is no release package for it: it is a research preview on a fork, and it is Linux-only (no Windows or macOS per the branch docs).

Why use it instead of llama.cpp Vulkan

  • It fills a gap in Strata itself. Strata's maintained Intel route is SYCL/oneAPI (docs/INTEL_ARC.md). This branch adds a Vulkan 1.3 + C++17-only path for Arc iGPUs where oneAPI or Level Zero is unavailable or unwanted. llama.cpp Vulkan is also Vulkan, but it does not give you Strata's server.
  • Memory, not speed, is the reason to prefer its native engine. Only the dense weights stay resident; the 256 experts per layer are read from the GGUF on demand into a bounded slot pool. In my runs that was 3.54 GiB of non-reclaimable RAM against llama.cpp's 20.28 GiB. The pool is bounded, so memory does not grow with the number of experts in the file — this is the route to an MoE larger than RAM on the same hardware.
  • You keep the Strata tooling on a Vulkan-only box: the normal OpenAI-/Anthropic-compatible API, strata launch, --status, --show-key, presets and strata connect <app> --check. Stock llama.cpp Vulkan gives you llama.cpp's CLI/server/bench tools instead.
  • It is opt-in. The default CUDA, HIP, SYCL and Metal launch flow is unchanged. The branch also includes the llama.cpp Vulkan bridge, so you can switch to --vulkan-impl bridge for speed without leaving Strata.

None of these is a reason to choose it for speed — see Results.

How to get and run it

It is a source checkout, not an installer. You need a Vulkan 1.3 driver, a C++17 toolchain and one Qwen3.8-35B-A3B GGUF:

git clone https://github.com/grigio/Strata.git
cd Strata && git checkout vulkan_arc
strata doctor --vulkan                 # device, heaps, MemAvailable-sized cache
strata launch --backend vulkan --model-path /path/to/model-Q4_K_M.gguf --plan
strata launch --backend vulkan --model-path /path/to/model-Q4_K_M.gguf

The --plan call only prints the plan. The next one builds the native engine, writes a config and starts the server on 127.0.0.1; the address prints once the ~21 GB GGUF has loaded. The native engine is the default. To get llama.cpp Vulkan speed through the same Strata server, use the bridge on the same branch:

strata launch --backend vulkan --vulkan-impl bridge --model-path /path/to/model.gguf
strata launch --status
strata launch --show-key --preset vulkan
strata launch --stop --preset vulkan
strata connect opencode --check

--vulkan-slots sizes the expert slot pool (the branch docs say 64 slots is about 55 MiB) and --save-preset NAME lets strata launch --preset NAME repeat the setup. strata doctor --vulkan matters here because on an iGPU the DEVICE_LOCAL heap is system RAM and vram_free_mib is a MemAvailable estimate, not dedicated VRAM (issue #515).

For this report I did not use strata launch: the timing runs invoked the two binaries directly, as shown in Reproduce. The launch commands above are the branch's documented workflow, not steps I re-ran.

Pros and cons

The native engine below is the default on vulkan_arc; the llama.cpp results come from that branch's opt-in bridge. The branch-level trade-off is: you gain a small-memory, server-backed Vulkan path, and you pay in tokens/s and in being on a preview fork.

Strata native Vulkan expert streaming

Pros

  • Fits a 35B model in a small footprint: 3.54 GiB of non-reclaimable RAM and ~5.8 GiB of system use, against ~20.3 and ~22.3 GiB for llama.cpp. On this 30 GiB box that leaves roughly 25 GiB for the OS, context and other apps.
  • Experts are read on demand from the NVMe into a bounded slot pool. Memory does not scale with the number of experts in the file — this is the path to models bigger than RAM on the same hardware.
  • Uses the normal Strata server and API; it needs only Vulkan 1.3 and C++17, so it is an option where oneAPI / Level Zero is unavailable (the maintained Intel route is SYCL).
  • Extremely predictable: decode SD 0.02 tok/s across five runs, and no stalls or pressure-degraded runs.

Cons

  • Slow under a real prompt: ~2.1 tok/s decode and about two minutes to first token. The old short test's 4.76 tok/s was a warmed, near-empty-context window; do not plan around it.
  • One token at a time, fp32 activations, host-side sampling, no prompt batching. A 418-token prompt takes ~125 s to prefill, and decode slows from ~2.36 to ~1.90 tok/s as the answer grows.
  • Pays for streaming with NVMe traffic: the engine's own counter reported ~231 GB of GGUF reads in one 674-forward session at a 61.6 % cache hit rate. Speed depends on the SSD, and the latency is per token.
  • Not the same answer as llama.cpp over a long generation: the greedy streams agree for 48 tokens, then diverge at a near-tie. Teacher-forcing agreement was not rechecked.
  • Research preview on a fork branch, not a released engine package: build from source, Linux only, no Windows or macOS. QSA sparse attention, KV quantization, gated residual and n-gram PLE are not ported. vram_free_mib is a shared-RAM estimate, so it is easy to misread.

llama.cpp Vulkan (bridge)

Pros

  • Far faster here: ~30.5 tok/s decode (steady ~40) and ~2,900 tok/s batched prefill through the bridge.
  • All weights are resident, so there are no per-token expert reads from disk.
  • Standard CLI/server/bench tooling, and no streaming-specific knobs to tune.

Cons

  • Needs ~20.3 GiB of shared GPU memory and ~22.3 GiB of system RAM just for this model and a small context. On a 30 GiB machine that is most of the memory; longer contexts or other apps will start swapping.
  • No expert streaming. The approach cannot run a model whose weights do not fit in RAM on this hardware.
  • On an iGPU the "GPU memory" line is just system RAM; llama.cpp does not call out the shared-memory caveat the way Strata's doctor does.
  • The run-to-run spread is large (SD 4.83 tok/s) because of periodic ~0.4 s stalls, and the PP line hides a ~0.5 s GPU sync — trust time to first token instead.
  • Through Strata it is the opt-in --vulkan-impl bridge path, not the default; the project notes a KV-cache-clear bug in the bridge that was fixed recently.

Choosing between them

Three options on this hardware. The two Strata rows are this test; the stock row comes from the earlier short test and was not rerun with the 418-token prompt:

Option Decode Prefill Non-reclaimable RAM Pick it when
Branch native engine (default) 2.12 tok/s 3.4 tok/s 3.54 GiB RAM is the limit; MoE bigger than RAM; you need Strata's API on a Vulkan-only box and can accept a couple of tok/s
Branch bridge (--vulkan-impl bridge) 30.47 tok/s 2,928 tok/s (PP) 20.28 GiB You want llama.cpp speed but through Strata's server/API
Stock llama.cpp Vulkan (-ngl 99) ~40.8 tok/s in llama-bench (short test) 132.9 tok/s bench pp7, not comparable ~20 GiB (inferred from the bridge) You want raw inference and llama.cpp's own tools; no fork or server needed

Use stock llama.cpp Vulkan when the 21 GB model and your context fit comfortably, you want interactive speed, and you do not need Strata. Use the branch's bridge for the same speed with Strata's server/API. Use the branch's native engine when RAM is the constraint, when you want to try a larger MoE than fits in memory, or when you need Strata's server/API on a Vulkan-only machine and can accept a couple of tokens/s and a two-minute wait for the first token.

Limits of this comparison

  • One machine, one day, one model and one quant (Q4_K_M); a 418-token prompt and 256 greedy tokens, 2048 context, 8 threads. No longer context, no multi-turn soak, no other quant (e.g. IQ2_XS/IQ3_XXS), no vision.
  • No HTTP round trip: timings come from the engines' own PP/DONE lines and timestamped T lines, so server overhead is excluded.
  • The greedy streams diverge at token 49, so the two engines are not computing the same answer over the full 256 tokens. The branch's teacher-forcing gate was not rerun; no KL/perplexity quality check.
  • No CUDA, HIP or SYCL comparison; no other runtime version of either project. The stock llama-bench cross-check is from the earlier short test, not this one.
  • The native mb counter (~231 GB) was read from VSTAT, not independently verified. VSTAT was byte-identical across all five runs because the runs were deterministic.
  • System "used" includes the OS and other processes; use the MemAvailable drop for a per-run comparison. VmRSS is misleading on this iGPU.
  • The vulkan_arc section — fork layout, the single commit on top of v0.2.16, launch commands, --vulkan-slots, doctor behaviour, platform support — comes from the branch's docs and the local git log, not from runs made for this report.

Reproduce

# 1. Tokenize the fixed prompt with Strata's reference tokenizer
cd /home/g/Code/strata-research/Strata/terminal/tools
python3 -c "
import strata_tokenizer as st
tk = st.Tokenizer.from_gguf('/home/g/Code/llama.cpp/models/Qwen3.8-35B-A3B-Q4_K_M.gguf')
text = open('/tmp/opencode/bench-prompt.txt').read().rstrip(chr(10))
open('/tmp/opencode/prompt_ids418.txt','w').write(','.join(map(str, tk.encode(text))))
"

# 2. Five fresh runs per engine, 418-token prompt, 256 tokens, memory sampled at 0.2 s
python3 /tmp/opencode/bench_strata5.py --tokens 256 --repeat 5 \
  --engines native,bridge --tag s5 --out /tmp/opencode/bench_strata5.json

# 3. Rebuild the chart
python3 /tmp/opencode/chart_strata5.py

# Direct engine invocation (the harness uses these commands)
printf 'GEN 256 temperature=0 <418 ids>\nVRAM\nVSTAT\nQUIT\n' | \
  /home/g/Code/strata-research/Strata/terminal/engine/vulkan/engine/strata-vulkan-engine \
    --model /home/g/Code/llama.cpp/models/Qwen3.8-35B-A3B-Q4_K_M.gguf \
    --ctx 2048 --threads 8 --lp 0 \
    --native-shaders /home/g/Code/strata-research/Strata/terminal/engine/vulkan/native/shaders \
    --expert-shaders /home/g/Code/strata-research/Strata/terminal/engine/vulkan/shaders \
    --q4-shader /home/g/Code/strata-research/Strata/terminal/engine/vulkan/shaders/expert_q4_k_gemv.comp.spv \
    --down-shader /home/g/Code/strata-research/Strata/terminal/engine/vulkan/shaders/expert_q6_k_gemv.comp.spv \
    --q8-shader /home/g/Code/strata-research/Strata/terminal/engine/vulkan/shaders/expert_q8_0_gemv.comp.spv \
    --silu-shader /home/g/Code/strata-research/Strata/terminal/engine/vulkan/shaders/expert_silu_mul.comp.spv \
    --router-gemv-shader /home/g/Code/strata-research/Strata/terminal/engine/vulkan/shaders/router_gemv.comp.spv \
    --router-topk-shader /home/g/Code/strata-research/Strata/terminal/engine/vulkan/shaders/router_topk.comp.spv

Raw per-run JSON is in /tmp/opencode/bench_strata5.json; the engine stderr logs are /tmp/opencode/s5-<engine>-r<N>.err. The memory sampler reads Linux /proc only. Chart: strata-vs-llamacpp.png.