Intel Arc Pro B70 Inference Cookbook

Repeatable vLLM XPU and llama.cpp SYCL recipes for Intel Arc Pro B60/B70 GPUs.

This is one cookbook with one page per model family. Do not start a second repo when a new architecture lands: add docs/<family>/ + benchmarks/<family>/ and pin that family's image digest. Do not mix patch lists or numbers across families. The image that serves Qwen3.6 Pi is not the image that served Nemotron DFlash.

Model family hub

Family	Engine	What is proven	Headline	Page
Qwen3.6-35B-A3B	vLLM XPU (Pi digest)	Native MTP 1/2/4, 128K	MTP4 p512/g128 170.91 client post-first n=5	this README §MoE
Qwen3.8-27B	vLLM XPU (nightly digest)	Dense GPTQ-INT4 + MTP4; optional draft-INT4; concurrent serving via mixed-split v5	C1 106.7 n=5 current stack (LMX `cmt03mj040eh8ms01trjvhm75`); cache-off 112.65 (`cmszpqy000e8fms014ty6i5x3`), BF16-draft 83.7 (`cmsur82fz06svms01ga1f0z83`). Concurrent (v5 + draft-INT4, prefix on): C5 realistic 127.4 Σ-streams / 25.5 per-user (`cmt03mjo60ehbms0117c5i745`), short-prompt C5 203.8 / C32 224.2 (lmx harness), C32 Σ-streams 903. Prefix reuse largely fails at C5 on this build (0–38% hits vs 91% at C1) — warm-session TTFT at Cn is an open issue	QWEN38-VLLM-XPU · Windows 11
Qwen3.6-27B	vLLM XPU (same Pi digest)	Dense GPTQ-INT4 + MTP, fp8 KV	MTP4 p512/g128 69.30 n=5	this README §Dense
Nemotron-3.5-Lightning-30B-A3B	vLLM XPU (newer digest)	DFlash n=7; native MTP 0%	186.61 C1 client post-first at p2048/g128 n=5; cold input 7160 (prompt/TTFT) at p8192/g1	NEMOTRON-DFLASH-B70
Muse-Glimmer-30B	llama.cpp SYCL	Vision + DFlash n2; vLLM still experimental	26.8 engine t/s at p512/g128 128K n=5	MUSE-GLIMMER-B70
Ornith-1.5-35B-A3B	vLLM XPU (Qwen3.8 nightly digest)	Local GPTQ-INT4 MixedCal-v2, MTP1 + DraftINT4 default; 262K C1; 150↔230 W prefill A/B	Self-reported E2: combined 230 W LMX `tokSOut` 108.4 / `tokSPrefill` 9073 (`cmt2tdx5q0hy0mv01koh4xwpw`); host p512/g128 106.64. BF16-draft MTP1 150 W 96.43. No-spec 230 W prefill 9780 (`cmt2sr6gq0himmv01ogieh0c8`)	ORNITH-VLLM-XPU

Image + patch pin: IMAGE-AND-PATCH-MATRIX.md. Every speed cell is C1 unless a table says otherwise. LocalMaxxing APPROVED means the payload was accepted into the public leaderboard.

Qwen3.6 family — current tested stack

Use this table only for Qwen3.6 Pi / dense. Nemotron uses a different digest.

Component	Exact tested value
Public image	`vllm/vllm-openai-xpu@sha256:2c427ef477da092eb6f2cdbbbd24950b5fa171565b916db69d4c7bb10e68ca97`
vLLM observed in image	`0.26.1rc1.dev457+gc810e5ee9.xpu`
`vllm-xpu-kernels` observed in image	`0.1.12`
MoE model	`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-GPTQ-Int4`
Dense model	`llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-GPTQ-Int4`
Muse-Glimmer-30B (llama.cpp only)	`unsloth/Muse-Glimmer-30B-GGUF` UD-Q4_K_XL + mmproj-kquant + dflash-kquant → docs/muse-glimmer/MUSE-GLIMMER-B70.md
Target / draft weights	GPTQ INT4 target / preserved BF16 MTP layer (both Qwen models)
Patches, in order	`patch_mtp_nightly.py`, then `patch_mtp_boundary.py`
MoE context / scheduler / memory	131,072 / 8,192 / `gpu-memory-utilization=0.85`
Dense context / scheduler / memory	131,072 / 8,192 / `gpu-memory-utilization=0.88` (MTP4) or `0.90` (no-spec/MTP1/MTP2)
Dense KV cache	`fp8` required — dense 27B needs 9.5 GiB fp16 KV at 128K, which does not fit; fp8 halves it

PyPI vllm-xpu-kernels 0.1.12.2 is newer, but it was not installed or tested in this campaign. The historical intel/vllm:0.21.0-xpu-int4moe image was local and was never published.

Short setup

Both launchers below already include the tool-calling flags (--enable-auto-tool-choice --tool-call-parser qwen3_coder) — required for Pi / omp / agent clients that send tool_choice: "auto". Without them those clients get 400: "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set.

git clone https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook.git
cd intel-arc-pro-b70-inference-cookbook
export MODEL_DIR="$HOME/models/Qwen3.6-35B-A3B-MTP-Preserved-GPTQ-Int4"
export MODEL_ID='llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-GPTQ-Int4'
docker pull 'vllm/vllm-openai-xpu@sha256:2c427ef477da092eb6f2cdbbbd24950b5fa171565b916db69d4c7bb10e68ca97'
bash benchmarks/qwen36-35a3/launch-vllm-128k-mode.sh "$MODEL_DIR" mtp2 on 8000
curl -f http://127.0.0.1:8000/health

Dense 27B (same image and patches, fp8 KV):

export DENSE_DIR="$HOME/models/Qwen3.6-27B-MTP-Preserved-GPTQ-Int4"
bash benchmarks/qwen36-27/launch-dense27-128k-mode.sh "$DENSE_DIR" mtp4 on 8000
curl -f http://127.0.0.1:8000/health

Serving with Pi / omp / agents: point the client at http://127.0.0.1:8000/v1 and use the served model name (Qwen3.6-35B-A3B-MTP-Preserved-GPTQ-Int4 or Qwen3.6-27B-MTP-Preserved-GPTQ-Int4). Tool calling is enabled by the launchers, so tool_choice: "auto" works out of the box. For a persistent server, wrap either launcher in your own systemd unit or process supervisor — the scripts are self-contained and portable (no host-specific paths).

Windows 11 hosts (WSLC / Docker Desktop)

Qwen3.8-27B also runs on Windows 11 with the same image digest. Two standalone PowerShell kits (Docker Desktop — proven, ~70 tok/s class on the 2026.08.18 BF16-draft measure; Microsoft WSLC — experimental, 2.4–2.8× slower) devised and tested by Ian Hudson (aitesthive.com). They reserve GPU memory for the Windows desktop (gpu-memory-utilization 0.75 + explicit 4.25 GiB fp8 KV) because a single-B70 Windows machine drives its display from the same 32 GB card. Image 2026.08.19 adds draft-INT4 S+M1 and turns prefix cache on for real sessions. If you already have the 18 August kit: .\Upgrade-Qwen38-Docker.ps1 — do not re-download the model. Guide: docs/qwen38-27/WINDOWS-STANDALONE.md (kits in windows/).

Connecting Pi / omp / Hermes

See CONNECTING-CLIENTS.md for the full client quick start: Hermes config.yaml provider block, omp base URL, Pi client settings, the port table (8000 launcher / 8765 bridge), the active model alias, API key setup, and a copy-paste tool-call smoke test.

Exact software versions (do not substitute):

Component	Exact tested value
Image	`vllm/vllm-openai-xpu@sha256:2c427ef477da092eb6f2cdbbbd24950b5fa171565b916db69d4c7bb10e68ca97`
vLLM	`0.26.1rc1.dev457+gc810e5ee9.xpu`
`vllm-xpu-kernels`	`0.1.12`
Tool-call parser	`qwen3_coder` (`Qwen3EngineToolParser`)

Use Full setup commands for the render-device check, model download and verification, package check, patch hashes, endpoint checks, and full matrix.

Benchmark graphics are rendered from the canonical summary.json with the public renderer benchmarks/render-prefill-decode-svg.py (dashboard + method diagram).

MoE: Qwen3.6-35B-A3B — whole analysis

Scope for every table below: one Intel Arc Pro B70, C1, median of n=5 after one same-output same-shape warmup, prefix cache enabled, unique entropy-first cold prefixes, zero cache-hit delta, scheduler 8,192, context 131,072, configured cap 165 W, client monotonic SSE timing.

Two model checkpoints were verified on this stack: the llmfan46/...MTP-Preserved GPTQ-INT4 (all matrix tables) and the byte-exact palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 incl. mtp.safetensors (claim-reproduction tests, below).

Cold prefill proxy: actual input tokens / TTFT, tok/s

Mode	p512	p2048	p4096	p6144	p8192	Full p131071
No spec	5,156	6,674	7,197	7,451	7,576	3,144
MTP1	4,840	7,377	6,999	7,189	7,264	2,679
MTP2	4,843	7,341	7,002	7,140	7,229	2,683
MTP4	4,532	7,401	6,868	7,057	7,197	2,678

This rate includes scheduling, uncached prompt processing, and first-token work. It is not isolated engine prefill and is not llama-bench pp.

Decode at p512: client post-first tok/s

Mode	g32	g128	g256	g512
No spec	97.43	96.79	96.60	96.13
MTP1	122.21	124.57	123.82	120.58
MTP2	162.90	153.17	148.31	141.80
MTP4	178.34	170.91	167.85	148.35

Decode at p8192: client post-first tok/s

Mode	g32	g128	g256	g512
No spec	85.92	90.34	90.91	91.26
MTP1	108.41	118.41	118.49	117.45
MTP2	143.95	145.43	143.82	135.61
MTP4	156.28	164.36	163.89	138.03

Historical control: p9445/g128

Mode	Client post-first median (tok/s)
No spec	89.68
MTP1	116.85
MTP2	142.02
MTP4	160.42

The MTP4 result of 160.42 tok/s reproduces the prior 158.83 tok/s scheduler-control result within 1.0%. It does not make the exact-128K cells equivalent.

Scheduler and context findings (2026-08-09, focused probes)

--max-num-batched-tokens is a cap, not a target: at p4096 both 8,192 and 16,384 prefill in one chunk, yet the larger budget is measurably faster at the same 128K recipe, same seqs 64, same prompts (exact palmfuture checkpoint, 230 W, MTP4):

Budget	p4096 prefill	g128 decode
8,192	6,525 t/s	133.1 t/s
16,384	7,672 t/s	149.1 t/s
Δ	+17.6%	+12.0%

The gain is scheduler/memory-layout, not chunk count. Do not blanket-adopt 16,384 without testing mixed long-prefill + short-chat loads: one 16K prefill step starves short requests (head-of-line), and activation spikes eat VRAM headroom (128K recipe already loads with ~1 GB free).

Prefill is essentially flat across context (p4096 input rate, batch 16,384, seqs 16): 8K ctx 7,727 t/s · 16K 6,622 · 32K 7,740 · 128K 7,672. Context length is not a prefill lever.

The 12,400 tok/s LocalMaxxing claim — reproduction verdict

We reproduced the claimed config exactly (byte-identical palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 incl. mtp.safetensors, hash-verified; context 32,768; p4096/g1; batch 16,384; seqs 16; MTP4; fp8 KV; cache off; 230 W). Measured: 7,740 t/s median, not 12,400. The claim's cited build hash 568afb3a1 is an upstream macOS-CI commit (#49901), not an XPU kernel change — not a meaningful reproduction target; their entry ran vLLM 0.26.1.dev0 on Windows 11, our stack is 0.26.1rc1.dev457 on Linux.

Verdict: not reproduced. The 1.6× gap is build/OS or their prefill definition (their implied TTFT 0.330 s vs our 0.529 s).

Full-context decode

Mode	p130944/g128 (tok/s)	MTP accept	p130560/g512 (tok/s)	MTP accept
No spec	57.35	n/a	57.14	n/a
MTP1	84.88	89.22%	82.74	85.32%
MTP2	101.64	85.81%	94.01	76.45%
MTP4	93.53	66.91%	93.83	59.81%

The original no-spec p130560/g512 cell stopped at EOS in three of five requests. It is excluded and retained in the evidence. The 57.14 tok/s row is the forced exact-output replacement.

Client post-first is (completion tokens - 1) / (request end - first generated token). It is request-side timing, not engine-native vLLM decode.

Choose a mode by workload

Short C1 responses: MTP4 was fastest in the p512 and p8192 g32/g128 cells.
Exact 128K, g128: MTP2 was fastest at 101.64 client post-first tok/s.
Exact 128K, g512: MTP2 and MTP4 were effectively tied at 94.01 and 93.83 tok/s in this campaign.
Resident long sessions: test cache reuse separately. The earlier matched cache campaign found MTP2 + cache on had the best resident end-to-end median.
Mixed long prefill plus short requests: use no-spec on this stack. The MTP mixed-token XPU path remains unsupported.

Dense: Qwen3.6-27B — whole analysis

Scope for every table below: one Intel Arc Pro B70, C1, median of n=5 after one same-output same-shape warmup, prefix cache enabled, unique entropy-first cold prefixes, zero cache-hit delta, scheduler 8,192, context 131,072, --kv-cache-dtype fp8 (required for dense 128K), configured cap 230 W, client monotonic SSE timing. Dense 27B GPTQ-INT4 runs on the pinned image (vLLM 0.26.1rc1.dev457+gc810e5ee9.xpu) via XPUwNa16LinearKernel; both MTP patches apply unchanged to the dense Qwen3_5ForConditionalGeneration architecture (same shared qwen3_5_mtp.py / gdn_attn.py).

Cold input rate: actual input tokens / TTFT, tok/s

Mode	p2048	p4096	p6144	p8192
No spec	1,781	1,813	1,782	1,742
MTP1	1,816	1,776	1,747	1,713
MTP2	1,812	1,767	1,744	1,711
MTP4	1,755	1,693	1,683	1,654

This rate includes scheduling, uncached prompt processing, and first-token work. It is not isolated engine prefill and is not llama-bench pp. Dense prefill is compute-bound (~10% of XMX peak at p4096) and collapses at long context (p130944 ≈ 547 t/s) — the full-attention O(N²) term.

Decode at p512: client post-first tok/s

Mode	g32	g128	g256	g512
No spec	32.90	32.85	32.78	31.54
MTP1	50.00	50.47	50.19	48.88
MTP2	62.15	63.59	61.45	59.95
MTP4	72.78	69.30	64.06	64.13

Decode at p8192: client post-first tok/s

Mode	g32	g128	g256	g512
No spec	31.48	31.46	31.45	31.42
MTP1	48.08	46.90	47.97	47.33
MTP2	63.98	60.73	59.62	57.10
MTP4	67.44	64.11	65.87	57.79

Historical control: p9445/g128

Mode	Client post-first median (tok/s)
No spec	31.35
MTP1	48.41
MTP2	60.12
MTP4	67.25

Full-context decode (exact 131,072 total tokens)

Mode	p130944/g128 (tok/s)	MTP accept	p130560/g512 (tok/s)	MTP accept
No spec	23.14	n/a	23.05	n/a
MTP1	36.77	90.9%	37.21	93.6%
MTP2	42.67	91.1%	36.18	87.8%
MTP4	47.61	89.2%	42.56	75.9%

MTP acceptance is the totals-diff value per cell (accepted/proposed draft tokens). Higher acceptance does NOT mean higher throughput: MTP4 leads every full-context cell despite the lowest per-cell acceptance.

Measured power draw — matched A/B (2026-08-10, authoritative)

Same mixed workload (1× p2048/g1 prefill + 2× p2048/g128 decode), fresh entropy prompts, true per-mode server, 230 W cap, monitor windowing only the active requests, cooldown ≤55°C between modes. Live energy1_input deltas, 0.5 s interval average:

Mode	Mean (W)	Max 0.5s (W)	pkg max (°C)	vram max (°C)
No spec	149.9	238.2	70	72
MTP4	151.0	251.5	73	72
MTP1	156.1	249.6	74	74
MTP2	153.3	242.9	72	72

All four modes within a 6 W band — MTP depth is not a power lever on dense. Max 0.5 s samples above the 230 W cap are short-burst overshoot before the cap controller engages (card TDP ~300 W).

Earlier campaign-window monitor means (195/197/146/146 W) were coverage artifacts — the no-spec/MTP1 windows included the heavy full-context 130K prefill cells, and the MTP2 monitor only caught a 223 s decode-only window. They must not be cited as a mode-vs-mode power comparison.

Real Pi workload — document session with follow-ups (MTP4, g128 outputs, 2026-08-10)

The test that matters: you give Pi a 32K-token document and ask it eight questions in a row. First question reads the whole document cold; every follow-up reuses the cached document. Fresh-server run:

Step	Prompt tokens	Cache hits	Hit %	TTFT (s)	Post-first (tok/s)
1. First read (cold)	32,640	0	0%	38.191	41.2
2. Follow-up 1	32,789	29,952	91.3%	4.069	41.0
3. Follow-up 2	32,884	29,952	91.1%	4.162	48.0
4. Follow-up 3	32,961	29,952	90.9%	4.241	46.7
5. Follow-up 4	33,054	29,952	90.6%	4.491	49.3
6. Follow-up 5	33,151	29,952	90.4%	4.553	43.9
7. Follow-up 6	33,208	29,952	90.2%	4.586	44.1
8. Follow-up 7	33,313	29,952	89.9%	4.932	49.4
9. Follow-up 8	33,377	31,616	94.7%	2.591	55.8

The result: cold document read = 38.2 s TTFT (fresh server; 25.2 s once the server is warm). Every follow-up = 2.6–4.9 s TTFT with 89.9–94.7% token reuse — 8–15× faster — even though the session grows 32.8K → 33.4K tokens as each Q&A is appended. The document is read once and stays resident; only the new question and reply are processed. Reuse wobbles because the cache matches in 64-token blocks at the document/conversation boundary. Cache eliminates input tokens — decode stays flat at 41–56 t/s.

Short-turn context (same server, single requests): cold conversation 54.2 tok/s (TTFT 0.424 s) · warm shared system 62.9 (0.423) · short multi-turn 46.9 (0.477) · RAG append 55.4 (0.574). These short scenarios show 0 cache hits by design: the shared prefix is the 557-token Pi system prompt, shorter than one 1,088-token cache page — a hit requires a full page, so only page-spanning content (documents, long sessions) reuses cache.

Realistic short-turn serving decode is 44–56 t/s, not the 73 t/s synthetic peak. Cache reuse eliminates tokens; it does not speed up per-token prefill.

Client post-first is (completion tokens - 1) / (request end - first generated token). It is request-side timing, not engine-native vLLM decode.

Dense 27B key constraints

fp8 KV is required for 128K: dense attention needs 9.5 GiB fp16 KV, which does not fit at gpu-memory-utilization 0.85–0.90; fp8 halves it to ~4.75 GiB (156,745–160,799-token capacity).
128K is the safe ceiling. 200K loads but leaves 3 MiB free after load (the §6 abort zone); 256K is VRAM-infeasible.
MTP4 needs gpu-memory-utilization=0.88 — at 0.90 the MTP4 spec buffers fill the card (0–2 MiB free). no-spec/MTP1/MTP2 run at 0.90.
Power lever: dense prefill scales with power (+52% at 230 W vs 165 W, Run 30); the MoE's power-flatness does not apply to dense.
Open blocker: no FP8 linear kernel on XPU (KeyError: PlatformEnum.XPU in choose_scaled_mm_linear_kernel); FP8 checkpoints can't load, INT4 works. See docs/qwen36-27/DENSE-FP8-GAP.md.

Dense vs llama.cpp GGUF baseline

The vLLM dense INT4 path is ~2.4–3× faster than the mature llama.cpp GGUF path: 73.2 t/s synthetic C1 decode (MTP4, p512) vs ~24–29 t/s GGUF Q4/Q6-MTP, and 1,754–1,816 t/s cold input vs ~936 t/s llama.cpp prefill at pp4096. Both paths share the 128K ceiling with MTP; vLLM additionally needs fp8 KV.

Choose a mode by workload (dense 27B)

Short C1 responses: MTP4 was fastest in every p512/p8192 g32–g512 cell (69.3 tok/s at p512/g128, vs 63.6 MTP2 / 50.5 MTP1 / 32.9 no-spec).
Exact 128K: MTP4 wins g128 (47.61 tok/s); at g512 MTP4 still leads (42.56 vs MTP2 36.18).
Power-sensitive serving: matched A/B shows no power difference across MTP depth (149.9–156.1 W mean at 230 W cap) — choose the mode on speed or latency, not draw.
Realistic Pi sessions: 44–56 t/s decode; MTP4 with cache on gives the fastest resident follow-ups (91.3% cache reuse).
Mixed long prefill + short requests: no-spec is the safe path on this stack (MTP4 mixed-token causal_conv1d crash remains open).

Nemotron: 3.5-Lightning-30B-A3B — vLLM XPU + DFlash (2026-08-13)

Two separate recipes. Do not mix their tables.

LocalMaxxing cmsr9po4w000ams01e4fc5qhj is APPROVED.

DFlash (current headline, isolated n=5): official method=dflash n_spec=7 on a local GPTQ-INT4 G64 target + local NVFP4→BF16 draft. Representative C1 client post-first decode 186.61 t/s at p2048/g128 (174.60–201.83). p8192/g128 157.92 t/s (1.81× vs matched no-spec 87.25 on that cell). p8192/g1 cold input 7160 t/s (prompt/TTFT — not isolated engine prefill). Window acceptance 52.0%. Do not headline p512 194.6 (family range 140–220). An earlier ~10.3k figure is a no-spec n=3 TTFT-derived rate on a decode cell — not this campaign.

No-spec graphs (still useful): 21.8 → 93.00 / 87.25 t/s (p512/p8192 g128, n=5) after XPU graph capture. Native MTP remains 0% acceptance. N-gram is not a production path.

Artifacts (canonical two-i account): SergiioB/Nemotron-3.5-Lightning-30B-A3B-GPTQ-INT4-G64-sym + SergiioB/Nemotron-3.5-Lightning-30B-A3B-DFlash-BF16.

Family index + claim lock: docs/nemotron35-30a3/README.md, CLAIMS.md
DFlash recipe: docs/nemotron35-30a3/NEMOTRON-DFLASH-B70.md
Dashboard: docs/assets/b70-nemotron-dflash-dashboard.svg
No-spec recipe: docs/nemotron35-30a3/NEMOTRON-B70.md
Launchers: launch-nemotron-dflash.sh TARGET DRAFT 8001 / launch-nemotron-graph.sh TARGET 8001
Runtime patches: patches/patch_xpu_grouped_topk_native_v2.py, patches/ssu-b70-b8w4/
Open PRs (2026-08-13): vllm#52159, vllm-xpu-kernels#524. Source copies: patches/vllm-xpu-kernels/

Ornith-1.5-35B-A3B — MixedCal-v2

Local experts-only GPTQ INT4 G128, MTP BF16, same qwen3_5_moe topology as Qwen3.6-35B-A3B, same nightly digest as Qwen3.8 (f01e24f6). Default serve is MTP1 + DraftINT4 (runtime overlay; shards stay BF16 MTP).

Self-reported E2, isolated C1, cache off, greedy diagnostic. LocalMaxxing APPROVED means accepted self-report, not independent reproduction.

Default decode is MTP1 + DraftINT4: host p512/g128 106.64, LMX tokSOut 108.4 / tokSPrefill 9073 at configured 230 W (cmt2tdx5q0hy0mv01koh4xwpw). BF16-draft MTP1 at 150 W is 96.43. No-spec 230 W long-prompt prefill is 9780 / decode 69.9 (cmt2sr6gq0himmv01ogieh0c8). MTP1 BF16 150 W p32 is tokSOut 94.1 (cmt2sl6eg0hdcmv01gre5o3ub). Those three LMX rows differ in speculation, prompt length, and cap. Combined tokSPrefill includes MTP first-token work, so 9073 ≠ no-spec 9780.

Recipe: docs/ornith15-35a3/ORNITH-VLLM-XPU.md
Measured tables: CLAIMS.md
Conversion (experts-only GPTQ, mixed-domain cal, RTN 24.76% → 10.37%; speed at 150 W is parity): MIXEDCAL-V2.md
Launcher: benchmarks/ornith15-35a3/launch-ornith-mtp1.sh MODEL_DIR 16384 8000
Weights: SergiioB/Ornith-1.5-35B-A3B-GPTQ-Int4-sym-G128-MTP-BF16-MixedCal-v2

DFlash is not a serving path — there is no Ornith hidden-2048 draft.

Muse: Glimmer-30B — llama.cpp analysis (2026-08-10)

First B70 run of Meta Muse-Glimmer-30B (dense 27.85B text + ViT-G/14 vision, 128K ctx, reasoning model). Public recipe: llama.cpp SYCL (DFlash n_max=2). vLLM Muse is an experimental PR-#51655 overlay plus a compressed-tensors INT4 n=3 screen — slower than this GGUF path and not a pullable image. FP8-block still does not fit 32 GB. Decode = engine timings.predicted_per_second (C1 cold, 128K ctx, 230 W cap, -ub 8192); prefill = llama-bench pp.

Metric	DFlash n2	no-spec	Δ
p512/g128 decode (t/s)	26.8	22.5	+19%
p8192/g128 decode (t/s)	22.9	17.2	+33%
p32768/g128 decode (t/s)	21.1	13.9	+52%
prefill pp4096 (t/s)	—	1,301	llama-bench
prefill pp32768 (t/s)	—	865	llama-bench

DFlash n_max screen (n=3, p512/p8192): n1 24.8/20.0 · n2 27.6/21.4 · n3 26.3/21.3 · n4 27.5/18.2 · n5-7 collapse (acceptance 0.30-0.56) · n8 aborts the server. Acceptance decays 0.91→0.30 with depth; the DFlash gain grows with context (+19% → +52%). Vision + reasoning + DFlash all verified.

Full commands, provenance, failures, and quality caveats: docs/muse-glimmer/MUSE-GLIMMER-B70.md.

LocalMaxxing submission (2026-08-10): leaderboard — engine cmsnly2su00goo001wn6c98ly, benchmark run cmsnly2sy00gqo001ui1k5l67 (record: submissions/llamacpp-muse-glimmer-30b.json).

Reproduce the matrix

The runner does not change host power. CONFIGURED_CAP_W records the cap selected by the operator.

CONFIGURED_CAP_W=165 \
  bash benchmarks/b70-pi-prefill-decode-matrix.sh "$MODEL_DIR"

Dense 27B (same matrix contract, fp8 KV, 230 W, GPU util 0.88 for MTP4):

CONFIGURED_CAP_W=230 \
  bash benchmarks/qwen36-27/launch-dense27-128k-mode.sh "$DENSE_DIR" mtp4 on 8000

Evidence and format:

vLLM runtime decisions — what this stack uses (both MoE and dense)

The pinned image runs vLLM V1 (0.26.1rc1.dev457+gc810e5ee9.xpu) on a single-socket single-GPU host. Of the five runtime decisions commonly discussed, here is exactly where this stack stands (verified from the running server's own config log, 2026-08-10):

Decision	This stack	Evidence
NUMA binding	N/A — single socket. `Socket(s): 1`, `NUMA node(s): 1`. There is no inter-socket link to cross; the "wrong socket" problem cannot occur on one NUMA node. vLLM's `--numa-memory-tracking` / node pinning is irrelevant here and would change nothing.	`lscpu`
Chunked prefill	Already ON (V1 default). Server config: `enable_chunked_prefill=True`. `--max-num-batched-tokens 8192` is the chunk cap; large prompts are sliced and decode interleaves between chunks. Scheduler-budget probes on the MoE (+17.6% at 16,384) and dense (flat) show the cap also shapes throughput — see the scheduler findings above.	server config log
Recompute instead of swap	Already the V1 behavior. vLLM V1 has no KV swap path — evicted/recomputed requests rebuild from the prompt (recompute) rather than moving KV to CPU. `swap_space` is a V0 concept; on this V1 build there is nothing to set to 0. The `vllm:prefix_cache_*` counters confirm hits are served from GPU KV, not CPU.	V1 source + metrics
Skip memory profiling	Not used — and not worth it here. We pass `--gpu-memory-utilization 0.88` (dense) / `0.85` (MoE); the memory-profile/warmup phase costs 0.40 s + 0.03 s of a 139.77 s engine init (compilation 106.37 s). `--kv-cache-memory` would skip ~0.4 s of a 140 s boot — 0.3%. Startup is dominated by Triton JIT + CUDA graph capture, not profiling.	server log
Eager mode	Not used — correct for serving. `enforce_eager=False`, `cudagraph_mode: FULL_AND_PIECEWISE` with capture sizes 1-256. Graph capture is the 106 s of the 140 s boot, and it is what makes steady-state decode fast (MTP4 69.3 t/s dense, 170.9 MoE). `--enforce-eager` would cut boot but trade away most decode throughput — only sensible for throwaway dev loops, not the production profile.	server config log

Tool calling (Pi / omp / OpenAI clients): both model paths must run with --enable-auto-tool-choice --tool-call-parser qwen3_coder (the Qwen3EngineToolParser in this build). Without them, clients that send tool_choice: "auto" (Pi, omp, most agents) get 400: "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set. The launcher profiles for both models include these flags; the raw launcher scripts in benchmarks/ include them for the serve command.

Bottom line: of the five levers, this stack already uses chunked prefill and V1 recompute (both defaults), does not need NUMA (single socket), and correctly skips eager mode and --kv-cache-memory — the profiling saving is 0.3% of boot while the eager trade would cost most decode throughput. The actionable runtime lever measured here was the scheduler budget (see MoE scheduler findings) and prefix caching (see the resident-session section).

Correctness limitation

Prompt hashes match across no-spec, MTP1, MTP2, and MTP4. Output parity does not. Depending on the longer-decode cell, only 0 to 4 of 5 repetitions matched exact output text across all four modes. The campaign shows speed and completed exact token shapes, not token, logit, KL, or task-quality parity. Do not use speed as correctness proof.

Repository map

benchmarks/
  qwen36-35a3/       MoE Qwen3.6-35B-A3B launchers and model-specific campaigns
  qwen36-27/         Dense Qwen3.6-27B launchers (launch-dense27-128k-mode.sh)
  nemotron35-30a3/   Nemotron DFlash + no-spec graph launchers
  ornith15-35a3/     Ornith-1.5 MixedCal-v2 MTP1 launcher
  <root>             shared: matrix runner, harness, monitor, prompt generation, compiler, renderers
windows/             Windows 11 standalone kits (WSLC + Docker Desktop) — see docs/qwen38-27/WINDOWS-STANDALONE.md
patches/             family-tagged patches — see IMAGE-AND-PATCH-MATRIX.md
docs/
  qwen36-35a3/       MoE-specific reference (QUANTIZATION-QUALITY.md)
  qwen36-27/         Dense-specific reference (DENSE-FP8-GAP.md)
  nemotron35-30a3/   Nemotron DFlash + no-spec recipes
  ornith15-35a3/     Ornith MixedCal-v2 recipe + measured tables
  muse-glimmer/      Muse llama.cpp recipe
  <root>             shared: setup, benchmark contract, methodology, compatibility, history
results/
  qwen36-35a3/       MoE machine-readable summaries and engine grids
  qwen36-27/         Dense summaries (dense27 model card, llama.cpp grids)
  <root>             shared cross-model summaries
research/            kernel and quantization investigations
submissions/         historical LocalMaxxing payloads

Model-specific files live under the family directory; cross-model contracts (benchmark format, setup, image/patch matrix) stay at the shared root. A new architecture gets a new family folder, not a new cookbook repo.

Code is MIT licensed. Measurement reports and prose are CC BY 4.0. See LICENSE.

Navigation Menu

Repository files navigation

Intel Arc Pro B70 Inference Cookbook

Model family hub

Qwen3.6 family — current tested stack

Short setup

Windows 11 hosts (WSLC / Docker Desktop)

Connecting Pi / omp / Hermes

MoE: Qwen3.6-35B-A3B — whole analysis

Cold prefill proxy: actual input tokens / TTFT, tok/s

Decode at p512: client post-first tok/s

Decode at p8192: client post-first tok/s

Historical control: p9445/g128

Scheduler and context findings (2026-08-09, focused probes)

The 12,400 tok/s LocalMaxxing claim — reproduction verdict

Full-context decode

Choose a mode by workload

Dense: Qwen3.6-27B — whole analysis

Cold input rate: actual input tokens / TTFT, tok/s

Decode at p512: client post-first tok/s

Decode at p8192: client post-first tok/s

Historical control: p9445/g128

Full-context decode (exact 131,072 total tokens)

Measured power draw — matched A/B (2026-08-10, authoritative)

Real Pi workload — document session with follow-ups (MTP4, g128 outputs, 2026-08-10)

Dense 27B key constraints

Dense vs llama.cpp GGUF baseline

Choose a mode by workload (dense 27B)

Nemotron: 3.5-Lightning-30B-A3B — vLLM XPU + DFlash (2026-08-13)

Ornith-1.5-35B-A3B — MixedCal-v2

Muse: Glimmer-30B — llama.cpp analysis (2026-08-10)

Reproduce the matrix

vLLM runtime decisions — what this stack uses (both MoE and dense)

Correctness limitation

Repository map

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.agentic/skills/b70-benchmark-visuals		.agentic/skills/b70-benchmark-visuals
benchmarks		benchmarks
docs		docs
patches		patches
research		research
results		results
submissions		submissions
windows		windows
.editorconfig		.editorconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

Intel Arc Pro B70 Inference Cookbook

Model family hub

Qwen3.6 family — current tested stack

Short setup

Windows 11 hosts (WSLC / Docker Desktop)

Connecting Pi / omp / Hermes

MoE: Qwen3.6-35B-A3B — whole analysis

Cold prefill proxy: actual input tokens / TTFT, tok/s

Decode at p512: client post-first tok/s

Decode at p8192: client post-first tok/s

Historical control: p9445/g128

Scheduler and context findings (2026-08-09, focused probes)

The 12,400 tok/s LocalMaxxing claim — reproduction verdict

Full-context decode

Choose a mode by workload

Dense: Qwen3.6-27B — whole analysis

Cold input rate: actual input tokens / TTFT, tok/s

Decode at p512: client post-first tok/s

Decode at p8192: client post-first tok/s

Historical control: p9445/g128

Full-context decode (exact 131,072 total tokens)

Measured power draw — matched A/B (2026-08-10, authoritative)

Real Pi workload — document session with follow-ups (MTP4, g128 outputs, 2026-08-10)

Dense 27B key constraints

Dense vs llama.cpp GGUF baseline

Choose a mode by workload (dense 27B)

Nemotron: 3.5-Lightning-30B-A3B — vLLM XPU + DFlash (2026-08-13)

Ornith-1.5-35B-A3B — MixedCal-v2

Muse: Glimmer-30B — llama.cpp analysis (2026-08-10)

Reproduce the matrix

vLLM runtime decisions — what this stack uses (both MoE and dense)

Correctness limitation

Repository map

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages