Running Inkling 2 Mini in llama.cpp: How Vulkan Support Cut the CPU Tax in Half

Thinking Machines Lab released Inkling 2 Mini on 20 November 2026, and Unsloth shipped a 16.6 GB UD-Q4_K_M GGUF the same day. It is an unusual model: a 26.2B-parameter mixture-of-experts where the middle 12 of 24 layers run three times with shared weights (48 layer applications per token), giving roughly 5.4B effective active parameters. It is also natively multimodal, with a 1M-token context window. tested in Ship Harness Bench

IInkling 2 Mini 26B A5B with Vulkan in ship harness bench

Getting it to run on a local Intel Arc GPU turned into a small journey through llama.cpp internals. This post is about what the Vulkan support actually fixed (https://github.com/danielhanchen/llama.cpp/pull/10), and the server/Pi configuration that made it usable in practice.

The problem: the attention kernel quietly ran on the CPU

Inkling's hybrid attention uses a llama.cpp feature called KV row lists (ggml_flash_attn_ext_rows). Instead of scanning the whole KV cache, each attention stream carries a small I32 list of the physical KV rows it should attend to — an elegant way to handle the looped, banded attention this architecture uses.

There was just one catch: the Vulkan backend explicitly refused those nodes.

// K/V row lists (ggml_flash_attn_ext_rows) are not supported
if (op->src[5] != nullptr) {
    return false;
}

supports_op returning false means llama.cpp quietly schedules that node on the CPU. The model still ran — but in llama-cli it managed only 7.5–8.3 tokens/s, while comparable Qwen MoE models on the same Arc B390 hit 30–40 t/s. The GPU was sitting idle for a large part of every forward pass.

The upstream PRs did not help: a stock build reported unknown model architecture: 'inkling2', and the merged PR #32694 expected 48 physical blocks while this GGUF stores 24 and loops them. The working architecture implementation turned out to be draft PR #32695.

A stopgap existed — LLAMA_KV_ROWS_DISABLE=1, which makes the graph use ordinary ggml_flash_attn_ext — and it got ~12 t/s. But that sidesteps the feature rather than supporting it.

The fix: per-stream row gathering in the Vulkan shader

I implemented KV row-list support directly in the Vulkan backend (draft PR danielhanchen/llama.cpp#10, stacked on #32695):

  • Binding 7 now carries the kv_rows I32 tensor (shape [n_kv_cols, n_streams]).
  • A new Flags bit (256) selects row-list mode in the shader, so every other architecture is unaffected.
  • fa_kv_index() maps each linear mask column to the physical K/V row from the list, while the attention mask keeps its linear index — a subtle detail that took a second pass to get right.
  • Sparse gather, split-K and the Intel Xe fast path are disabled in this mode; the scalar shader path handles it.
  • I also removed some unconditional std::cerr debug spam that was left in ggml_vk_flash_attn.

The companion fix in the same PR wires the model's per-expert NVFP4 scales into the MoE graph. The GGUF stores 20 ffn_gate_up_exps.scale tensors that the draft PR ignored; without them the model either refused to load (expected 734, got 714 tensors) or produced multilingual gibberish.

Measured result

Setup Decode speed
Old CPU fallback for attention 7.5–8.3 t/s
LLAMA_KV_ROWS_DISABLE=1 workaround ~12.0 t/s
Native Vulkan KV row-list kernel 13.4 t/s (CLI), ~12.0 t/s (API)

Same prompt, same output (The capital of France is Paris.). The improvement is modest in absolute terms because this model does a lot of work per token — 48 layer applications and 8-of-128 expert routing with NVFP4 scales — but the CPU tax is gone, and the Vulkan device is now doing the attention work it was built for.

Configuring the server

The patched build lives in /home/g/Code/llama-inkling/build-vulkan (Clang for both C and C++; GCC as the C compiler fails on Clang-specific warning flags). The router start command:

cd /home/g/Code/llama.cpp
BUILD_DIR=/home/g/Code/llama-inkling/build-vulkan \
CTX=24576 BATCH=256 UBATCH=256 CACHE_K=q4_0 CACHE_V=q4_0 \
./llama-server-start-multi.sh --start

Why those numbers:

  • CTX=24576 — the sweet spot. 32768 OOMs on this ~20 GiB free UMA heap; 16384 loads but is too small for Pi (see below).
  • BATCH=UBATCH=256 — the batch size, not the context, was the original OOM trigger. 512 or 2048 fails immediately.
  • CACHE_K/V=q4_0 — halves KV memory versus q8_0.

The server runs in router mode with on-demand loading, so the first request pays a ~10 s load cost and later requests reuse the instance. Verify with:

curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/models | jq

The models.ini entry is simply:

[Inkling-2-Mini-UD-Q4_K_M]
model = ./models/Inkling-2-Mini-UD-Q4_K_M.gguf
alias = Inkling-2-Mini-UD-Q4_K_M

No spec-type (no MTP head) and no mmproj — the matching projector exists in the HF repo but has not been downloaded, so this setup is text-only for now.

Configuring Pi (the part that actually caused "Response was truncated")

After the kernel fix, Pi still reported "Response was truncated before completion." Twice. Both times the server was healthy; the problem was on the client side.

First cause: the GGUF chat template defaults to reasoning_effort: high, and Pi's usual enable_thinking kwarg means nothing to this template. Fix — a model-level compat block in ~/.pi/agent/models.json:

"compat": {
  "chatTemplateKwargs": {
    "reasoning_effort": "minimal"
  }
}

Second, bigger cause: Pi hardcodes a 4096-token output-safety margin and computes maxTokens = min(model.maxTokens, contextWindow - input - 4096). With a 12k-token prompt and contextWindow: 16384, that left roughly 170 output tokens. The model always reasons — even reasoning_effort: "none" produced a long thinking trace — so a simple tool call got cut off mid-arguments. Fix — grow both the server context and Pi's declared window together:

{
  "id": "Inkling-2-Mini-UD-Q4_K_M",
  "reasoning": true,
  "input": ["text"],
  "contextWindow": 24576,
  "maxTokens": 4096,
  "samplingParams": { "temperature": 1.0, "top_p": 1.0, "top_k": 0 }
}

And in ~/.pi/agent/settings.json, make sure the model is enabled with a sensible compaction reserve:

"llamacpp/Inkling-2-Mini-UD-Q4_K_M": {
  "reserveTokens": 4096,
  "keepRecentTokens": 8192
}

Restart Pi after editing — it reads these files at startup. With 4096 tokens available, a request that previously truncated completed cleanly in 898 tokens (about 500 thinking tokens plus a full 150-word answer), finish_reason: stop.

Takeaways

  1. A "working" model can still be leaving performance on the table. Nothing in the logs said the attention was on CPU; the symptom was just a model that felt twice as slow as it should.
  2. supports_op is a powerful debugging entry point in llama.cpp backends — one boolean decides GPU versus CPU for a whole class of nodes.
  3. Match client and server context windows. Pi's safety margin is invisible until you do the arithmetic; a larger maxTokens in the model config alone will not help if the window is too small.
  4. Looped models reason a lot. Inkling 2 Mini ignores attempts to fully disable thinking, so budget output tokens generously.
  5. The stack is still new. These fixes are in draft PRs as of 21 November 2026; expect upstream to catch up quickly.