llama.cpp Vulkan vs BigMoeOnEdge benchmark of a 21 GB MoE on an iGPU Intel B390

Measured 19 November 2026 on a Xiaomi Book Pro 14 — Intel Core Ultra X7 358H (16 cores), Intel Arc B390 iGPU, ~31 GB RAM, NVMe SSD. Model: Qwen3.8-35B-A3B-Q4_K_M.gguf (21 GB, 35.5 B params, 256 experts, top-8, 40 layers).

5-run benchmark: throughput, latency, memory and BigMoe's per-token trajectory

Why this test is different

An earlier version of this comparison used a single 64-token run, and the numbers were too thin to trust: BigMoe's expert cache had not settled, llama.cpp's prefill varied by 3× between runs, and a session-ready memory snapshot badly understated what the engine actually consumes during decode. So I redid it properly:

  • Same fixed 418-token prompt and 256 greedy generated tokens for both engines, 2048-token context.
  • 5 independent runs per engine, reported as mean ± SD, with medians where the distribution is skewed.
  • Per-token telemetry from BigMoe's --progress stream (wall_ms, cache hit, evictions) to see what happens during the run, not just the end-of-run average.
  • Memory sampled throughout via /proc/meminfo, /proc/<pid>/status and smaps_rollup — because per-process RSS is actively misleading here.

How "Vulkan mode" differs between the two

They do not mean the same thing, which is the story of the comparison.

llama.cpp builds libggml-vulkan.so and offloads every layer with -ngl 99. All weights go into Vulkan device memory; on an iGPU that is system RAM presented to Vulkan as "VRAM" through DRM device mappings. Decode, prefill and the expert GEMMs all run on the GPU.

BigMoeOnEdge builds the same kind of module (scripts/build-host.sh -DGGML_VULKAN=ON), loads backends as plugins, and uses --auto to decide what to do with them. The planner measured this machine:

  • Vulkan is 1.4× faster at 128-token-wide matmul (324 vs 231 GiB/s), so prefill runs there.
  • Vulkan is slower one token at a time (82 vs 143 GiB/s), so decode stays on the CPU.
  • Because the iGPU reads host RAM, putting layers on it frees no memory. The capacity fitter's answer is deliberately discarded ("placement-declined"). There is no capacity win to buy with -ngl.

So BigMoe's Vulkan mode is prefill on Vulkan, decode on the cores, experts streamed from flash — a different design point, not a different spelling of -ngl.

Results (5 runs each)

Metric llama.cpp Vulkan BigMoeOnEdge (--auto, Vulkan prefill + stream)
Decode 40.26 ± 0.32 tok/s (median 40.4) 11.96 ± 2.25 tok/s (median 13.05) — 3.4× slower
Prompt prefill (418 tokens) mean 408 ± 154, median 500 tok/s 80.4 ± 1.9 tok/s
Model load 8.8 ± 0.7 s 19.0 ± 0.6 s
Time to first token 10.0 ± 0.6 s 24.2 ± 0.6 s
Non-reclaimable RAM 20.2 ± 0.3 GiB 10.7 ± 0.8 GiB
Peak process RSS 0.25 GiB¹ 26.9 ± 0.7 GiB²
Expert cache n/a 18.6 GiB, 90.8 % mean hit rate
Pressure-affected runs 0 / 5 2 / 5

¹ Vulkan device memory is mapped through /dev/dri as device pages, excluded from RSS and PSS. ² Mostly file-backed mmap of the GGUF that the kernel can evict.

llama.cpp b1386 (a956a852c) via the llama-cli-vulkan wrapper; BigMoeOnEdge 0.46.0, explicit config taken from the planner minus the first-answer trials. Identical prompt and seed conditions.

What the per-token trace adds

The 256-token average hides the most interesting part. BigMoe starts each run at ~7 tok/s and climbs to ~16–18 tok/s by token 100–120 as the expert cache fills — the trajectory panel in the chart shows five runs doing this. That warm-up is why a 64-token test understates the engine's steady state by ~20–30 %.

It also shows the failures clearly. Two of five runs (red) never reached the fast regime. The engine reported 3,545–4,290 experts dropped by the kernel under memory pressure and 54–501 cache evictions; decode collapsed to 8.1 and 11.7 tok/s. This is a genuine property of the design, not noise: a cache that relies on reclaimable pages is at the mercy of the kernel's reclaim decisions, and on a 31 GB machine running repeated loads, those decisions sometimes hurt.

Memory: the correction that mattered

The short test reported ~2.6 GiB of non-reclaimable RAM for BigMoe, taken at session-ready with an 11-token prompt. The full-run number is 10.7 GiB. As tokens stream in, the reserved expert slots are actually committed and become anonymous, non-reclaimable pages; by the end of a 256-token run roughly 10–12 GiB has left the available pool. The honest comparison is therefore:

  • llama.cpp Vulkan: 20.2 GiB non-reclaimable, very consistent (SD 0.3).
  • BigMoe streaming: 10.7 GiB non-reclaimable, roughly half — but growing with generation length, and with two runs showing the kernel clawing back cached experts mid-run.

Per-process RSS still tells the opposite story (0.25 GiB vs 26.9 GiB), which is why time -v or top cannot answer this question. Vulkan's device maps are invisible to RSS; BigMoe's file-backed pages are visible but reclaimable. The drop in MemAvailable is the number that reflects "how close am I to OOM".

Prefill also deserves a note: llama.cpp's median is ~500 tok/s but two of five runs dipped to 151–377 tok/s for no obvious reason (probably iGPU clock/thermal behaviour), while BigMoe's Vulkan prefill was rock steady at 78–83 tok/s. Decode, by contrast, was extremely stable for llama.cpp (SD 0.32) and variable for BigMoe (SD 2.25) — exactly the inverse of what you would expect from the two architectures.

Pros and cons

llama.cpp Vulkan — all layers on the GPU

Pros

  • 3.4× faster decode and ~6× faster prefill at steady state, with the tightest numbers in the whole test (±0.32 tok/s on decode).
  • Fastest startup — 8.8 s to load, 10.0 s to first token.
  • One flag (-ngl 99), no planner, no cache warm-up, no pressure sensitivity. All five runs behaved identically.
  • No flash traffic during decode; the experts are already in device memory.

Cons

  • ~20 GiB of RAM becomes non-reclaimable, leaving ~8 GB for the OS and everything else. A model meaningfully larger than RAM simply will not load — there is no streaming fallback.
  • On an iGPU the "VRAM" is the same pool the CPU uses; there is no discrete memory being offloaded to. The win is wide arithmetic, not freed host RAM.
  • Prefill is unpredictably variable on this chip (151–507 tok/s across runs).
  • No mechanism to survive memory pressure: the system starts swapping device-backed pages.

BigMoeOnEdge — Vulkan prefill + expert streaming

Pros

  • Roughly half the non-reclaimable footprint (10.7 vs 20.2 GiB) over a full run, and most of its RSS is file-backed pages the kernel can drop. This is the property that lets it run models far larger than RAM — the case llama.cpp cannot handle at all.
  • Uses Vulkan where it helps. The planner measured the 1.4× wide-matmul advantage and put prefill there, while leaving one-token decode on the faster cores. It declines layer placement that would double-count shared iGPU memory.
  • Steady prefill (78–83 tok/s, SD 1.9) and a clean warm-up trajectory: decode nearly doubles over the first ~120 tokens as the cache settles.
  • Correctness is checked before trust: Vulkan prefill agreed with host output at 97 % top token, logit NMSE 1.5e-3.

Cons

  • 3.4× slower decode when the model fits in RAM. Streaming reads ~50 MiB/token from flash even at a 91 % cache hit rate.
  • Slower cold start — 19 s load, 24 s to first token, plus ~100 tokens before the cache reaches full speed.
  • Fragile under pressure. 2/5 runs were degraded by the kernel dropping cached experts; this is a design consequence, not a bug that has been fixed.
  • More moving parts (cache sizing, O_DIRECT lanes, prefill arena, trial knobs). --auto hides most of it, but it is still a more complex machine than -ngl 99.
  • Turning streaming off does not rescue speed here: a separate short check gave 5.0 tok/s and a 45 s load, and the engine itself labels that mode a baseline rather than its engine.

Verdict

For this model on this machine, llama.cpp Vulkan is the faster and far more predictable tool: 40.3 vs 12.0 tok/s, 10 vs 24 s to first token, and 0.3 tok/s of variation across five runs instead of 2.3. If your models fit in RAM, that is the simple answer.

BigMoeOnEdge buys a different property: about half the non-reclaimable memory and the ability to keep going when the model does not fit, by holding weights as reclaimable pages and re-reading routed experts from flash. On a 21 GB model that advantage costs 3.4× decode speed and comes with a 40 % chance of a pressure-degraded run. On a 100+ GB MoE on the same laptop, llama.cpp Vulkan cannot start at all, while this is the mode that works — and the warm-up trajectory shows it will get faster the longer you keep it alive.

Put bluntly: fits in RAM → llama.cpp Vulkan. Does not fit → BigMoeOnEdge. Both use the Vulkan backend; they just spend it on different things. The 5-run test sharpens the earlier single-run conclusion without overturning it, but it does correct the memory number: 10.7 GiB, not 2.6 GiB.

Reproduce

# llama.cpp Vulkan (all layers to the iGPU)
cd /home/g/Code/llama.cpp
build/bin/llama-cli-vulkan -m models/Qwen3.8-35B-A3B-Q4_K_M.gguf \
  -ngl 99 -c 2048 -n 256 -p "$(cat prompt.txt)"

# BigMoeOnEdge, Vulkan prefill + expert streaming (built with -DGGML_VULKAN=ON)
cd /home/g/Code/BigMoeOnEdge
build/bin/bmoe-cli -m /home/g/Code/llama.cpp/models/Qwen3.8-35B-A3B-Q4_K_M.gguf \
  --threads 16 --trial-threads "" --moe-stream --cache-mb 18600 \
  --dense-mapped token_embd.weight --prefill-device Vulkan0 \
  --prefill-best-effort --prefill-slots-on-demand --no-think --progress \
  -n 256 -p "$(cat prompt.txt)"

Chart: lla-v-BigM-chart.png (also SVG). Raw per-run data and telemetry were kept under /tmp/opencode/ for this session. Memory figures are the drop in system MemAvailable sampled at 0.2 s intervals; Vulkan/DRM device memory does not appear in per-process RSS.