Qwen3.8-27B on a 24 GB RTX 3090: measured settings, speeds and energy

01

What this page is, and what it measured

This page tells you how to run one model — Qwen3.8-27B — on one graphics card, a 24 GB RTX 3090, and what to expect when you do. Every setting it recommends was measured on that machine, and the measurement that decides each recommendation is printed beside it. It also carries settings for other cards, marked clearly as calculated rather than measured. It is not a review, not a comparison between models, and not a set of numbers you should expect to match exactly on different hardware. If all you want is a working configuration, go to §03 and copy the block for your card.

The limits, stated plainly: one machine, one model family, one operating system. Every measured row on this page is the same RTX 3090 on Windows 11 with driver 596.36. Nothing here has been checked on Linux, on AMD, on Apple silicon, or on a second copy of the same card. Where a number was calculated instead of measured, it says so in the cell.

02

Plain words: every term used here

These are the technical words this page uses without stopping to explain them again. Each is defined once, here, with one of this campaign's own numbers attached wherever a number makes it concrete. Every later section links back to this box instead of defining anything again.

03

The recipes — copy one for your card

One recipe per class of card. Each is the single best configuration this page can defend from its own measurements, not a menu of options. The 24–32 GB class is the one exception: two files won there, split by what you value, so both are printed and the rule for choosing between them is stated first. The reasoning behind every flag is in §04 to §09; here you copy a block.

The choosing rule for a 24 GB card, in one sentence: take the default recipe (UD-IQ4_XS) unless you want the highest measured quality at a 122k window, in which case take the Q4_K_M recipe (Q4_K_M) and accept that it leaves only about 0.3 GiB of the card for your desktop — less than a browser typically holds. The two files are statistically tied on quality — perplexity 6.535 against 6.596, a 0.9% gap against a combined error bar of ±0.063, so the difference is smaller than the uncertainty on it — while B is 2.1 GiB smaller and measured faster in every comparison run: 42.97 against 39.99 t/s with no drafter, 93.9 against 81.7 at the best drafter settings, both pairs from the same matched sweep on the same prompt. The smaller file is what buys you both vision and a big window at the same time.

NVIDIA 24–32 GB · 3090 / 4090 / 5090 — THE DEFAULT · UD-IQ4_XS (vision, speed, big window)

The file: unsloth/Qwen3.8-27B-GGUF, UD-IQ4_XS — 14.25 GB on the download page, 13.3 GiB resident (download pages count in decimal GB, memory budgets in binary GiB; the ~7% gap is real and matters whenever memory is tight). Plus the BF16 mmproj projector from either repository if you want vision. The window: -c 122880. The flags: all layers on the GPU, one slot, an 8-bit KV cache, and the conservative drafter. The effort levels it offers: all three — low, medium and xhigh. Nothing is excluded here: xhigh's measured thinking appetite is 61,500–75,800 tokens and a 122,880-token window holds its upper tail with room for a prompt and an answer. Expected speed, measured with these exact flags: 83–86 t/s of answer tokens on short-context code — 83.5 on a novel rate-limiter prompt at -c 32768 and 86.3 at a 1,458-token fill under the cooled protocol, the two endpoints differing by content rather than by configuration. Then 64.8 at 91k of depth, and 37–39 t/s on the xhigh reasoning stream at 91k — which is where a long agent run actually spends its wall clock. VRAM at the top of this window: 20,658 MiB of server process by the measured budget model, leaving 2.7–3.7 GiB of board VRAM depending on what your desktop is holding. Room left for your desktop: 2.7–3.7 GiB — this is the row chosen so that a browser and an agent's web interface do not push it over.

:: TAKE THIS ONE for everyday use: quality statistically TIED with Q4_K_M (6.596 vs
:: 6.535, +0.9%, smaller than the +/-0.063 combined error bar), measured FASTER
:: everywhere (decode 42.97 vs 39.99 with no drafter and 93.9 vs 81.7 at n10/p0.5,
:: both from one matched sweep; prefill 1365 vs 1303), 2.1 GiB smaller — and
:: this block ships VISION on. The Q4_K_M block below is kept only because you have
:: 122k, and only if you can free the VRAM your desktop is using - A leaves it about
:: 0.3 GiB. Caveat: IQ-format decode (files whose name contains IQ) was measured on
:: CUDA only - verify on Vulkan or Metal before trusting these speeds there.
:: file: unsloth/Qwen3.8-27B-GGUF (UD-IQ4_XS, 14.25 GB = 13.3 GiB) + BF16 mmproj
llama-server.exe -m Qwen3.8-27B-UD-IQ4_XS.gguf  --alias qwen/qwen3.8-27b
  --mmproj mmproj-Qwen3.8-27B-BF16.gguf  --image-min-tokens 1024  --image-max-tokens 10580
                                :: min 1024 is a FLOOR only — it lifts small images and never
                                :: shrinks a big one (measured: 720p 922 -> 1,077 tokens, 1440p
                                :: unchanged at 3,602). max 10580 = 4K-class detail. Per-image
                                :: token costs: section 12. The projector costs 1,138 MiB of
                                :: VRAM and 0% of decode (0.04-0.09% at a 91k fill, section 06)
  -c 122880                     :: 20,658 MiB of server VRAM by the measured budget model
                                :: (section 05), leaving 2.7-3.7 GiB for a desktop that itself
                                :: measured 1,179-1,669 MiB idle, no server. The largest window
                                :: whose every byte still lives on the card in THIS configuration
                                :: is 163,840; 122,880 is the daily value, chosen to leave a
                                :: desktop room
  -ngl 99  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0
                                :: -ngl 99 is not a guess: llama.cpp counts the output layer as
                                :: layer 65, so -ngl 64 leaves the output layer on the CPU and
                                :: costs 29.5% of decode with NO memory signature (section 11)
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75  :: the CONSERVATIVE
                                :: drafter, kept here on purpose. n-max 10 / p-min 0.5 is the
                                :: measured PEAK on code — 93.9 vs 83.5 t/s on this file, 81.7
                                :: vs 69.8 on Q4_K_M, same ranking on both (section 06) — but it
                                :: costs 898 MiB, and THIS row already spends 1,138 MiB on the
                                :: projector. It also buys least where this default actually
                                :: runs: on xhigh REASONING tokens at 91k it is worth 5.6%
                                :: (38.7 vs 36.6 t/s). Put n10/p0.5 on the text-only rows.
                                :: On PROSE, drop speculation entirely: 1.16x for ~1.8 GiB
  --jinja --host 127.0.0.1 --port 1234
  --chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}"  :: all three levels fit this window ONLY IF you leave
                                :: room for them. The window is one POT: prompt + thinking +
                                :: answer share it. xhigh wants 61,500-75,800 tokens of
                                :: THINKING, so your prompt and history must stay under about
                                :: 47,000 of these 122,880 - 38%% of the window. Past that,
                                :: xhigh truncates and returns NOTHING (section 09). medium
                                :: when you are waiting at the keyboard, and whenever the
                                :: context is deep
:: FLAGS DELIBERATELY OMITTED HERE:
::   --parallel 2 — SETTLED 2026-08-25 by the matched drafter-ON pair: +22.0% aggregate
::     throughput but -35.4% per slot (82.98 -> 101.25 t/s aggregate, 85.79 -> 55.41
::     per slot). The old drafter-OFF +60% was a measurement error, not a real gain -
::     with the drafter on for one user, because the drafter is already saving most of
::     the repeated reading of the model that a second slot would otherwise save.
::     Serving ONE person, one slot is faster; raise it only to serve two users at
::     once (section 03's axis table).
::   -ctk q4_0 — measured at +0.693% perplexity against fp16 (q8_0 costs +0.309%).
::     A knowing trade for a bigger window, not a default (section 08).
::   nvidia-smi -pl — MEASURED 2026-08-25. Capping the board is a real
::     efficiency win and the only setting here that lowers wattage: 300 W costs
::     5.0% of throughput for 11.2% less power, 250 W costs 12.8% for 24.8%.
::     Left out of the recipe because it is a persistent hardware setting and
::     needs an elevated shell, not because it does not work (section 11).
:: text-only variants — drop the --mmproj/--image-* line and the window grows:
::   -c 180224 + n-max 10 / p-min 0.5: the measured text-only ceiling WITH that
::     drafter - 23,729 MiB of board VRAM, 847 MiB of slack left for a desktop.
::     (196,608 leaves 415 MiB and 212,992 leaves 280, with decode already
::     sagging 56.6 -> 51.9 -> 49.2 on a
::     SHORT probe. Earlier editions shipped 196,608 and claimed ~3 GiB of slack;
::     that was wrong by about 2.6 GiB.)
::   -c 262144 THE FULL NATIVE CONTEXT - but ONLY with --spec-type none AND with no
::     graphical session using this card ("headless", section 02: closing everything
::     that draws, a second card driving the screen, or a remote login with nobody
::     logged in locally - NOT just unplugging the monitor).
::     Drafter off it derives to 23,216 MiB (22.7 GiB), leaving 1,360 MiB of board
::     VRAM - which is 179 MiB once a desktop takes its measured worst case. WITH the
::     drafter on it does not fit at all: 2,364 MiB measured living in system RAM,
::     and a 91k-token document then decodes at 8.0 t/s (section 05).
::   The rule: a ceiling belongs to the whole configuration — file + drafter flags +
::   projector + the desktop you run. Quote all four or the ceiling is not portable.

NVIDIA 24–32 GB — THE BIG-WINDOW PICK · UD-Q2_K_XL (vision and the drafter and 196,608 tokens)

The file: unsloth UD-Q2_K_XL — 9.83 GB, 9.154 GiB resident. The window: -c 196608. The flags: the universal set, the conservative drafter, and the full image budget. VRAM at depth: 22,014 MiB meas — peak, with a 1440p screenshot in flight on 163,124 tokens, sampled every 0.5 s. Room left for your desktop: 2,562 MiB, clearing this page’s 1,796 MiB reserve by 766. Effort: medium is what this pick ships, and the reason is the window rather than the clock. The window is one pot — prompt, thinking and answer share it. An ordinary xhigh answer only wants about 2,217 tokens of thinking, which fits here easily; but this pick exists to fill 196,608, and at the 163,124 tokens it was measured at only 33,484 remain. A complete-build task — the one thing measured wanting 61,500–75,800, four samples of one task (§09) — would truncate there and return nothing. So: xhigh for questions of a full window, medium when you are asking for a whole program. Earlier editions blamed the 278-second prefill SUPERSEDED: effort changes how much the model writes, not how long it takes to read your prompt.

Why this pick exists, added 2026-08-25. A reader wanted the largest window they could get while keeping both vision and the drafter, and this page had no answer for them — the 4-bit file cannot do it, and the full 262,144 fails in every drafter state once an image is in flight. The 2-bit file can, and three separate measurements say it costs less than its name suggests: it ties the 4-bit reference on accuracy (one of 75 paired items different, p=1.00), it reads images identically (7/7 against 7/7 on a generated detail target), and its window genuinely retrieves — 5 of 5 needles found at every depth out to 241,655 tokens, with a clean control (§08). What you actually trade is decode speed at short windows, where the 4-bit file drafts better.

196,608 is the ceiling, measured. -c 229376 peaks at 23,529 MiB and keeps only 1,047 — it is under the reserve at load, before an image is even sent. So this window is not a round number chosen for looks; it is the largest that survived the test. One honest note on that failing run: its answer to the image question came back as prose rather than the bare number, which the scoring here cannot grade either way. It is reported as a memory failure only, because that is what was measured.

The cost that matters more than VRAM: prefill. Filling this window took 278 seconds meas, against 210 s at -c 163840 and roughly 8 minutes at the full 262,144. Prefill is compute-bound and does not get cheaper. If your agent appends to its context you pay this once and the prefix cache carries you; if it rebuilds context every turn you pay it every turn, and -c 163840 saves you a minute for 32,768 fewer tokens. Choose on that, not on the bigger number.

:: THE BIG-WINDOW PICK: vision + drafter + 196,608 tokens, which no 4-bit
:: configuration on this page can hold. MEASURED 2026-08-25 with a 1440p image
:: in flight at 163,124 tokens: peak 22,014 MiB, 2,562 MiB left for a desktop.
:: 196,608 IS THE CEILING, and it is measured rather than assumed:
::   -c 196608   peak 22,014 MiB   2,562 MiB slack   CLEARS by 766
::   -c 229376   peak 23,529 MiB   1,047 MiB slack   FAILS by 749
::   -c 262144   peak 22,877 MiB   1,699 MiB slack   FAILS by 97 (drafter OFF!)
:: 229,376 is already under the reserve AT LOAD (1,516) before an image lands.
:: file: unsloth/Qwen3.8-27B-GGUF (UD-Q2_K_XL, 9.83 GB = 9.154 GiB)
llama-server.exe -m Qwen3.8-27B-UD-Q2_K_XL.gguf  --alias qwen/qwen3.8-27b
  --mmproj mmproj-Qwen3.8-27B-BF16.gguf  --image-min-tokens 1024  --image-max-tokens 10580
                                :: full image budget - the 1024 cap makes the model
                                :: confidently MISREAD fine detail (section 05)
  -c 196608  -ngl 99  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0
                                :: 22,014 MiB peak MEASURED at depth with an image, not
                                :: at load. Drop to -c 163840 for 3,852 MiB of slack and
                                :: a minute less prefill
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75  :: the wider
                                :: n10/p0.5 is the CODE peak but costs ~900 MiB and is
                                :: not measured at this window - do not assume it fits
  --jinja --host 127.0.0.1 --port 1234
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"  :: the window is one POT
                                :: an ordinary xhigh answer averages 2,217 tokens (175 benchmark
                                :: prompts), which the 33,484 left at the 163,124-token depth above
                                :: holds 15 times over. medium is shipped because a COMPLETE BUILD
                                :: is the one thing measured wanting 61,500-75,800 (four samples of
                                :: one task, section 09) and that would truncate here. Ask questions
                                :: of a full window at xhigh; ask for a whole program at medium.
                                :: This has nothing to do with the 278 s prefill: effort changes how
                                :: much the model WRITES, not how long it takes to READ your prompt.

NVIDIA 24–32 GB — THE ONE YOU HAVE HEARD OF · Q4_K_M (no measurable quality advantage, 2.1 GiB larger, ~0.3 GiB left for your desktop)

The file: lmstudio-community/Qwen3.8-27B-GGUF, Q4_K_M — 16.5 GB on the download page, 15.4 GiB resident. The window: -c 122880. The flags: the same universal set as the default recipe, with the conservative drafter — there is no room here for the wider one. The effort levels it offers: all three, for the same reason as the default recipe. An ordinary xhigh answer averages 2,217 tokens of thinking across 175 benchmark prompts; the 61,500–75,800 figure quoted elsewhere on this page is a ceiling from four samples of one task — a complete build — and a 122,880-token window holds even that, leaving about 47,000 for your prompt. Expected speed, measured with these exact flags: 69.8 t/s of answer tokens on short-context code and 57.9 t/s on the reasoning stream; the 81.7 t/s figure this file is famous for on this page belongs to the wider drafter, which this row cannot afford — it is a cross-reference, not this recipe's speed. VRAM at the top of this window: about 23.7 GiB of board VRAM with a desktop up, measured, leaving roughly 0.3 GiB. Room left for your desktop: about 0.3 GiB — less than a browser typically holds. §11 measured this very configuration spilling about 1.9 GB with two browsers open and falling to 20–35 t/s.

:: TAKE THIS ONE only if you believe an unresolvable 0.9% perplexity edge is
:: worth 2.1 GiB AND you can free the VRAM your desktop uses - this row leaves
:: about 0.3 GiB, less than a browser typically holds. The edge is 0.061 of
:: perplexity against this page's own +/-0.063 comparison error: INSIDE THE BAR.
:: The GSM8K half of the old argument is unaudited. THE DEFAULT ABOVE is better
:: on context, speed, vision and desktop headroom, and no worse on any
:: measurement this page can resolve.
:: file: lmstudio-community/Qwen3.8-27B-GGUF (Q4_K_M, 16.5 GB = 15.4 GiB)
:: before first run: set "Prefer No Sysmem Fallback" (NVIDIA Control Panel) and
:: never write -ngl 64 — both explained in section 11
llama-server.exe -m Qwen3.8-27B-Q4_K_M.gguf  --alias qwen/qwen3.8-27b
  -c 122880                     :: 24 GB: the largest window whose every byte still lives on the
                                :: card is ~131k, so this leaves slack (section 05's two
                                :: ceilings). 5090 32 GB: -c 262144
                                :: fits — budget ~11 GiB of KV at the MEASURED slope, not the
                                :: 8.5 GiB the q8 arithmetic suggests
  -ngl 99  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75  :: NOT a per-file
                                :: setting — the 2026-08-23 matched sweep ranks n10/p0.5 first on
                                :: BOTH files (81.7 vs 69.8 here). It is kept at n4 only because
                                :: n10 costs 898 MiB and this ~0.3 GiB-slack row cannot spare it.
                                :: BUDGET the drafter: 1,008 MiB fixed + 11.4% of your window
                                :: (section 05). --spec-type none frees all of it; on prose it is
                                :: worth only ~1.16x, so that is the easiest 1.8 GiB to reclaim
:: vision: add  --mmproj mmproj-Qwen3.8-27B-BF16.gguf --image-min-tokens 1024
:: --image-max-tokens 10580  when you need it. The projector costs 1,138 MiB
:: once loaded - not the 0.867 GiB its file weighs - and with this file at -c 122880
:: that is the measured ~23.7 GiB reference configuration: it fits, but leaves only
:: ~0.3 GiB of the card for your desktop. With a desktop running, drop to -c 65536,
:: or -c 49152 if you keep browsers open.
:: Budget from the MEASURED slope: each 32k of window costs ~1.375 GiB
:: with the drafter on and ~1.22 GiB with it off; the 1.06 GiB q8 arithmetic is a
:: FLOOR. Vision + a desktop + a BIG window = the default recipe above.
:: On the 1 GiB-larger UD-Q4_K_XL, projector + full context needs more than 24 GB -
:: use -c 98304
  --jinja --host 127.0.0.1 --port 1234
  --chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}"  :: quality-first default;
                                :: medium when the wait matters (section 09's price list)
:: FLAGS DELIBERATELY OMITTED: the same three as the default recipe — --parallel 2 (+22.0%
:: aggregate but -35.4% per slot with the drafter on, measured 2026-08-25 — slower
:: for one user), -ctk q4_0 (+0.693% perplexity, a window trade), and nvidia-smi -pl
:: (unmeasured, needs an elevated shell).

The full menu for a 24 GB card, losers included

The reference machine's launcher offers eight configurations, and this is all of them — including the ones that lost, each with the number that beat them, so that an option you have already heard of is answered rather than absent. Hover or tap a row to see the flags that produce it.

Every "Expect" band carries its token regime, because answer tokens run about 1.7× the speed of reasoning tokens on code and a band without its regime is not a number you can plan with. Speed also falls with depth, not only with spill: on the shipped flags, answer tokens measure 86.3 → 80.2 → 64.8 t/s at 1.5k / 28k / 91k of fill under the cooled protocol, while the reasoning stream on the same file and flags runs 51.2 → 47.1 → 36.6. An agent session holding 10–30k of context therefore lives near 80 t/s of deliverable; the same session left on the xhigh default over a 91k document spends most of its wall clock at 37–39 t/s, whichever drafter flags it carries. Slack figures are the board VRAM measured left over with little on screen; the desktop's own share swung 1,179–1,669 MiB in direct no-server readings on one machine (§05), so a row keeping under about 1 GiB has room for a near-idle desktop and nothing beyond it.

@echo off
rem ============================================================================
rem  serve-qwen.bat - llama.cpp server for Qwen3.8-27B on RTX 3090 24GB
rem ============================================================================
rem  Measured 2026-08-22, corrected 2026-08-23. Every pick carries the
rem  measurement that justifies it, because this file travels to other
rem  machines without the guide.
rem  - -ngl 99 always (llama.cpp counts the output layer as layer 65; -ngl 64
rem    leaves the output layer on CPU: 25.7 vs 39.7 t/s on Q4_K_M,
rem    29.8 vs 42.3 on UD-IQ4_XS, with NO memory signature)
rem  - MTP flags are PER PICK, and the split is NOT per file. A matched sweep
rem    (2026-08-23, both quants, novel code, thinking OFF) peaks at n-max 10 /
rem    p-min 0.5 on BOTH files: IQ4_XS 93.9 t/s (2.18x over its 43.0 no-drafter
rem    floor), Q4_K_M 81.7 (2.04x over 40.0). The shipped n-max 4 / p-min 0.75
rem    reads 83.5 and 69.8 on the same probe - 11%% / 15%% off the peak, and
rem    898 MiB cheaper. So: n10/p0.5 on the text-only picks where that VRAM is
rem    spare, n4/p0.75 wherever the projector is loaded or the slack is thin.
rem    Acceptance is a property of the MTP head, not of the quant - the two
rem    files land within 1.6 points at six of seven configs (3.7 at the seventh).
rem  - the drafter is also the cheapest ENERGY setting here: 3.21 J per decode
rem    token at n10/p0.5 against 8.10 with --spec-type none (2.52x), and
rem    the board draws the SAME power while decoding either way (344.6 / 341.7 /
rem    341.0 W): the entire win is throughput, not wattage.
rem  - TOKEN REGIME decides the number. Answer tokens (thinking off) run ~1.7x
rem    reasoning tokens, and mean DRAFT LENGTH is what predicts speed, not
rem    acceptance: at 91k depth the reasoning stream drafts 2.99 tokens per
rem    pass (36.6 t/s) and the answer stream 4.31 (62.0) - at the same ~0.90
rem    acceptance. Highest-acceptance config (n2/p0.75, 96.5%%) is the slowest.
rem  - Depth, answer tokens, n4/p0.75, cooled probes: 86.3 t/s at 1.5k, 80.2 at
rem    28k, 64.8 at 91k. On the xhigh REASONING stream at 91k expect 37-39 t/s
rem    (n10/p0.5 buys only 5.6%% there - not worth 898 MiB).
rem  - UD-IQ4_XS (13.3 GiB): quality tied with Q4_K_M (perplexity 6.596 vs 6.535;
rem    the GSM8K half of that comparison is unaudited - see the guide),
rem    FASTER everywhere measured, and 8%% cheaper per token in energy.
rem    Measured ceilings for a window whose every byte still lives on the card:
rem    -c 180224 text-only WITH n10/p0.5; the full native -c 262144 ONLY with
rem    --spec-type none (1,360 MiB of board VRAM left, and that window needs no
rem    graphical session on the card at all); -c 163840 with vision at n4/p0.75.
rem    122880 + vision is the daily default, chosen to leave a desktop room.
rem    Filling a window is the only test that counts - a short
rem    probe at 262144 with the drafter on looks fine and delivers 8 t/s on a
rem    91k document.
rem  - Q4_K_M (15.4 GiB): the quality leader, by one metric and narrowly; +mmproj
rem    at -c 122880 (~0.3 GiB of slack - room for a near-idle desktop, no more).
rem  - the BF16 mmproj costs 1,138 MiB of VRAM and 0%% of decode: measured at
rem    90,862 tokens of fill, projector on vs off, 0.04-0.09%% apart.
rem  - KV cache: q8_0 costs +0.309%% perplexity, q4_0 +0.693%% (measured
rem    2026-08-23, super-linear in bits). q8_0 is what these recipes ship.
rem  - reasoning_effort is THE wall-clock knob: ~4x medium on one long
rem    authoring task, 1.8x across a 175-prompt benchmark suite where the
rem    three levels scored within 1.6 points of each other. llama-server
rem    ignores per-request effort, so it must be set here.
rem ----------------------------------------------------------------------------
rem  USAGE:  serve-qwen.bat [low^|medium^|xhigh] [context] [1-8]
rem          arg1 = reasoning effort, default xhigh (quality-first)
rem          arg2 = context override (otherwise each choice's safe default)
rem          arg3 = model choice 1-8, skips the menu (for scripts)
rem          No args: menu below, auto-picks [1] after 8 seconds, so an
rem          unattended restart never blocks on a human.
rem ----------------------------------------------------------------------------
set EFFORT=%1
if "%EFFORT%"=="" set EFFORT=xhigh
if /i "%EFFORT%"=="low" goto effort_ok
if /i "%EFFORT%"=="medium" goto effort_ok
if /i "%EFFORT%"=="xhigh" goto effort_ok
echo Unknown reasoning effort "%EFFORT%". Usage: serve-qwen.bat [low^|medium^|xhigh] [context] [1-8]
pause
exit /b 1
:effort_ok
set CTXARG=%2
set PICK=%3

set MODELS=C:\path\to\your\models
set MMPROJ=%MODELS%\mmproj-Qwen3.8-27B-BF16.gguf

rem  the drafter is PER PICK (it is a load-time flag): n10/p0.5 is the code peak
rem  on both files but costs 898 MiB, so it rides the text-only picks; every pick
rem  carrying the projector keeps n4/p0.75.
set SPEC_FAST=--spec-type draft-mtp --spec-draft-n-max 10 --spec-draft-p-min 0.5
set SPEC_SAFE=--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75

if not "%PICK%"=="" goto pick_%PICK%

echo.
echo  Qwen3.8-27B - pick a model (effort: %EFFORT%):
echo    [1] UD-IQ4_XS   + HI-RES vision  -c 122880  n4/p0.75  ~3 GiB slack (DEFAULT)
echo                                          ~83-86 t/s answer tokens on code,
echo                                          ~65 at a 91k fill, 37-39 on xhigh
echo                                          reasoning at 91k;
echo                                          screenshots to ~4K detail (1080p=2.0k /
echo                                          1440p=3.6k / 4K=8.2k ctx tokens each);
echo                                          full xhigh cycle + ~10 shots fit
echo    [2] UD-IQ4_XS   text-only  -c 180224  n10/p0.5  code-regime peak 93.9 t/s
echo                                          the measured text-only ceiling WITH this
echo                                          drafter: 847 MiB slack, little on screen
echo    [3] UD-IQ4_XS   text-only  -c 262144  DRAFTER OFF - required, not optional
echo                                          full native - needs the card free of any
echo                                          graphical session;
echo                                          no drafter = ~43 t/s short-context floor
echo    [4] Q4_K_M      + vision   -c 122880  n4/p0.75  highest measured quality,
echo                                          reference config
echo                                          ~0.3 GiB slack - a near-idle desktop only
echo    [5] UD-Q4_K_XL  text-only  -c 122880  n4/p0.75  quality LOSES to Q4_K_M
echo                                          (perplexity +2.3%%) and is 1 GB bigger:
echo                                          gains nothing
echo    [6] UD-Q4_K_M   + vision   -c 122880  n4/p0.75  measured worse than plain Q4_K_M
echo                                          (perplexity +1.8%%) - skip
echo    [7] DFlash2 build                     ~46 t/s on real code (loses to built-in MTP)
echo    [8] NVFP4 HIGH                        ~48-54 t/s  dequant fallback, lower quality,
echo                                          +13%% J/token vs IQ4_XS
echo.
echo    t/s above are ANSWER tokens (thinking off) at short context unless said
echo    otherwise. Reasoning tokens run ~1.7x slower on the same server, and
echo    decode falls with DEPTH: 86 shallow / 80 at 28k / 65 at 91k (answer),
echo    51 / 47 / 37 (reasoning). Acceptance actually RISES with depth
echo    (0.80 -^> 0.92): the cost is KV reads, not the drafter failing.
echo.
choice /C 12345678 /T 8 /D 1 /M "Choice"
goto pick_%ERRORLEVEL%

:pick_1
set MODEL=%MODELS%\Qwen3.8-27B-UD-IQ4_XS.gguf
set CTX=122880
set MM=--mmproj "%MMPROJ%" --image-min-tokens 1024 --image-max-tokens 10580
rem  n4/p0.75 on purpose: this pick already spends 1,138 MiB on the projector,
rem  and the n10 flags would take another 898 MiB out of the slack that
rem  absorbs the desktop (measured 1,179-1,669 MiB). It also buys least here -
rem  5.6%% on the xhigh reasoning stream this default actually decodes at depth.
set SPEC=%SPEC_SAFE%
goto launch
:pick_2
set MODEL=%MODELS%\Qwen3.8-27B-UD-IQ4_XS.gguf
set CTX=180224
set MM=
rem  text-only: the 898 MiB is spare here, so take the code peak (+12%%).
set SPEC=%SPEC_FAST%
goto launch
:pick_3
set MODEL=%MODELS%\Qwen3.8-27B-UD-IQ4_XS.gguf
set CTX=262144
set MM=
rem  drafter OFF is required, not a preference: with it on this window needs
rem  ~25.5 GiB and 2,364 MiB was measured living in system RAM.
set SPEC=--spec-type none
echo.
echo  WARNING: -c 262144 fits ONLY with the drafter off, and then only just:
echo  23,216 MiB derived, 1,360 MiB of board VRAM left - 179 MiB once a worst-case
echo  desktop takes its measured share. WITH the drafter on it does not fit at
echo  all, and a 91k-token document then decodes at 8.0 t/s. A browser or agent
echo  web UI will spill it either way. Run this with NO graphical session using
echo  the card: close everything that draws, drive the screen from a second card,
echo  or log in from another machine. Unplugging the monitor does not do it.
echo.
goto launch
:pick_4
set MODEL=%MODELS%\Qwen3.8-27B-Q4_K_M.gguf
set CTX=122880
set MM=--mmproj "%MMPROJ%" --image-min-tokens 1024
rem  ~0.3 GiB slack with the projector loaded: no room for the n10 flags here.
set SPEC=%SPEC_SAFE%
goto launch
:pick_5
set MODEL=%MODELS%\Qwen3.8-27B-UD-Q4_K_XL.gguf
set CTX=122880
set MM=
rem  kept in the menu as an answered loser: perplexity 6.682 vs Q4_K_M's 6.535.
set SPEC=%SPEC_SAFE%
goto launch
:pick_6
set MODEL=%MODELS%\Qwen3.8-27B-UD-Q4_K_M.gguf
set CTX=122880
set MM=--mmproj "%MMPROJ%" --image-min-tokens 1024
rem  answered loser: perplexity 6.654, +1.8%% over plain Q4_K_M at the same size.
set SPEC=%SPEC_SAFE%
goto launch
:pick_7
rem  DFlash2 needs its own build (llama.cpp PR 27342) and a separate 1.14 GB
rem  drafter; best on real code measured 46.5 t/s at n-max 2, below MTP's 57.9.
call serve-qwen-dflash2.bat %EFFORT%
exit /b
:pick_8
rem  NVFP4 on Ampere runs through a software fallback: slower, +4.4%% perplexity,
rem  +13%% J/token. Kept for Blackwell users and for the record.
call serve-qwen-nvfp4.bat HIGH %EFFORT%
exit /b

:launch
if not "%CTXARG%"=="" set CTX=%CTXARG%
if not exist "%MODEL%" (
    echo Model not found: %MODEL%
    pause
    exit /b 1
)
echo Serving %MODEL%
echo   effort=%EFFORT%  ctx=%CTX%
echo   spec=%SPEC%
llama-server.exe ^
    -m "%MODEL%" ^
    %MM% ^
    --alias qwen/qwen3.8-27b ^
    -c %CTX% ^
    -ngl 99 ^
    --parallel 1 ^
    --load-mode none ^
    --api-key dummy ^
    -ctk q8_0 -ctv q8_0 ^
    %SPEC% ^
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 ^
    --reasoning-preserve ^
    --chat-template-kwargs "{\"reasoning_effort\":\"%EFFORT%\"}" ^
    --jinja ^
    --host 127.0.0.1 ^
    --port 1234

NVIDIA 16 GB · 5080 / 4080 / 4070 Ti S / 5060 Ti

The file: unsloth UD-Q2_K_XL — 9.83 GB, 9.154 GiB resident. The window: -c 65536. The flags: the universal set plus the conservative drafter. VRAM at the top of this window: 13,982 MiB meas. Room left for your desktop: 606 MiB against the 14,588 MiB this page budgets for a 16 GB card — enough for a light desktop, not for a browser full of tabs; drop to -c 49152 if you need more. The effort levels it offers: low and medium. xhigh is not offered: its measured thinking appetite is 61,500–75,800 tokens against a 65,536 window, so it would fit on its shortest runs and truncate on its longest — which is worse than a clean no, because you would not know which you got. Expected speed: 25–50 t/s drafter-off, derived from the bandwidth formula in §04 der; the drafter adds to that. No speed on a 16 GB card is measured on this page and none can be, from a machine that owns one 24 GB card. The memory figure is measured and transfers to any card.

This recipe changed on 2026-08-25, and the reason is worth more than the change. It used to ship UD-Q3_K_XL at -c 49152, priced at “about 15.3 GiB” SUPERSEDED. That figure was derived from a bandwidth estimate, and when the requirement was finally measured file by file (§08) it came out wrong in the direction that breaks a machine: UD-Q3_K_XL with this drafter needs 15,530 MiB at -c 32768 and 16,906 MiB at -c 65536, so -c 49152 lands near 16,218 MiB — 1,142 MiB over the budget on this very page. It does not fit. The smaller file gives you twice the window inside less memory, and on the paired accuracy test it ties the 4-bit reference (§08), so almost nothing is being traded away. The old recipe is left below rather than deleted, because a derived number that was never checked against a measurement is exactly the failure this page exists to document.

:: best daily experience on 16 GB, MEASURED 2026-08-25: the 2-bit file, not the
:: 3-bit one. UD-Q2_K_XL at -c 65536 with this drafter needs 13,982 MiB; UD-Q3_K_XL
:: at the same window needs 16,906 and does not fit. See section 08's requirement table.
:: Q4-with-CPU-offload preserves more fidelity but crawls at ~8-12 t/s (section 07).
:: file: unsloth/Qwen3.8-27B-GGUF (UD-Q2_K_XL, 9.83 GB = 9.154 GiB)
llama-server.exe -m Qwen3.8-27B-UD-Q2_K_XL.gguf  --alias qwen/qwen3.8-27b
  -c 65536  -ngl 99  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0
                                :: 13,982 MiB MEASURED on the reference 3090 at this exact
                                :: window with this exact drafter - not arithmetic. Against
                                :: the 14,588 MiB this page budgets for a 16 GB card that
                                :: leaves 606 MiB for your desktop. Drop to -c 49152 if
                                :: you run a browser, or --spec-type none to buy back ~1.3 GiB.
                                :: A >131k window cannot fit resident on 16 GB - that needs
                                :: the Q4 offload path (section 07)
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75  :: start at 4 and
                                :: sweep on your own card (section 06): the optimum moved with
                                :: content and hardware in every run measured here
  --jinja --host 127.0.0.1 --port 1234
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"  :: xhigh NOT OFFERED:
                                :: measured appetite 61.5-75.8k tokens against a 65,536 window,
                                :: so it would fit on short runs and truncate on long ones. You
                                :: would not know which you got - see section 09's ceiling table
:: WHAT THIS RECIPE USED TO BE, kept because the error is instructive: UD-Q3_K_XL
:: at -c 49152, priced at "~15.3 GiB" from a DERIVED bandwidth estimate. Measured,
:: it needs ~16,218 MiB there - 1,142 MiB OVER this page's own 16 GB budget.
:: FLAGS DELIBERATELY OMITTED: --parallel 2 (settled 2026-08-25: +22.0% aggregate,
:: -35.4% per slot with the drafter on - slower for one user),
:: -ctk q4_0 (+0.693% perplexity - with the 2-bit file the window no longer needs
:: buying, so the one card class that could defend that trade no longer has to).

12 GB class · RTX 3060 / RTX 5070 / Arc B580

The honest framing first: this class runs a 27B model poorly in every configuration measured or derived here, and if the exact model is not a requirement, a 14B-class Qwen at 4-bit fits resident and is several times faster. The file, if it must be this model: Q4_K_M with most layers on the processor. The window: -c 112640. The effort levels it offers: all three fit the window — but the recipe ships medium, and here the binding constraint is wall clock, not the window. How much wall clock depends entirely on what you ask for. At 6–8 t/s an ordinary xhigh answer — 2,217 tokens of thinking, averaged over 175 benchmark prompts — takes about 5 minutes. A complete build, the one task measured wanting 61,500–75,800, takes about three hours against one for medium. Earlier editions quoted the three hours as the cost of “an xhigh answer” SUPERSEDED, which overstates ordinary work by roughly thirty times. Ask questions at xhigh freely here; save the overnight run for whole deliverables. Expected speed: 6–8 t/s, derived, and set by your system memory bandwidth rather than by the graphics card. Room left for your desktop: nearly all of the card — almost none of the model is on it.

:: honest framing: this class runs the 27B poorly in every configuration. If the
:: exact model is not required, a 14B-class Qwen at Q4 fits resident and flies.
:: If it must be this model: Q4 with most layers on CPU (~6-8 t/s; speed is set by
:: your system RAM. The ~6-8 t/s assumes TWO sticks of DDR5 in dual channel
:: (~90 GB/s); one stick = single channel = half the bandwidth = half the t/s).
:: file: lmstudio-community/Qwen3.8-27B-GGUF (Q4_K_M — on a bandwidth-bound
:: offload path the smaller Q4 file means fewer GB re-read per token, and the
:: measured perplexity in section 08 favours it too)
llama-server.exe -m Qwen3.8-27B-Q4_K_M.gguf  --alias qwen/qwen3.8-27b
  -c 112640  -ngl 28            :: ~28 of the 65 offloadable layers (64 + the output
                                :: layer — section 11); raise until <500 MB VRAM is free
  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0   :: CPU layers need ~10 GB of real RAM
:: no MTP flags here on purpose: speculation is UNMEASURED on the CPU-offload path.
:: The measurement that would justify adding them is one paired probe with and
:: without --spec-type draft-mtp at this -ngl, on your machine. It costs ten minutes.
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0  --jinja --host 127.0.0.1 --port 1234
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"  :: unlike the 16 GB card
:: this is NOT a window limit — the 112k window (mostly in system RAM) holds xhigh's
:: 61.5-75.8k appetite fine. It is a PATIENCE limit: at ~6-8 t/s an xhigh answer runs
:: ~3 HOURS against ~1 h for medium (section 09). Switch to xhigh for unattended runs.
:: Arc B580: use the Vulkan build (same SYCL caveat as the Arc Pro cards - they are
:: all Battlemage, Intel's Arc B series)

Intel Arc Pro B70 · 32 GB

The file: Q6_K — 22.4 GB, 20.9 GiB resident, near-8-bit quality, which the extra VRAM affords. The window: -c 229376 with the drafter, or the full 262144 with --spec-type none. The effort levels it offers: all three; the window holds xhigh's appetite several times over. Expected speed: 18–26 t/s, derived from bandwidth and unmeasured on this card's chip generation (Battlemage — Intel's Arc B series). VRAM at the top of the window: about 30.5 GiB derived with the drafter off. Room left for your desktop: not measured — every figure for this card is derived, so treat the last gigabyte as unverified.

:: VULKAN llama.cpp build - currently beats SYCL on Battlemage, Intel's Arc B series,
:: which this card belongs to (SYCL has known perf
:: bugs there; IPEX-LLM is archived — do not use it). Re-benchmark after driver updates.
:: file: lmstudio-community/Qwen3.8-27B-GGUF (Q6_K, 22.4 GB = 20.9 GiB)
llama-server.exe -m Qwen3.8-27B-Q6_K.gguf  --alias qwen/qwen3.8-27b
  -c 229376  -ngl 99            :: NOT the full 262144 with a drafter loaded: at the
                                :: MEASURED 45,056 B/token the native window costs ~11 GiB of
                                :: KV, not the 8.5 the q8 arithmetic suggests, and Q6_K + MTP +
                                :: that needs more than 32 GB. For the FULL 262144 here, add --spec-type
                                :: none and drop the MTP line below (~30.5 GiB). Both figures
                                :: are DERIVED from the 3090's constants — unmeasured on Battlemage
  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0   :: verify KV-quant support in your build
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75  :: start at 4, sweep (section 06)
  --jinja --host 127.0.0.1 --port 1234
  --chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}"  :: all three fit, with room to spare at THIS window (229,376).
                                :: An ordinary xhigh answer averages 2,217 tokens of thinking
                                :: (175 benchmark prompts). Only a COMPLETE BUILD has been
                                :: measured wanting 61,500-75,800 - four samples of one task,
                                :: section 09 - and even that leaves ~153,000 here for prompt
                                :: and history. Earlier editions of this comment quoted 122,880
                                :: and ~47,000: copied from the 24 GB recipe, never re-done
                                :: against this window. SUPERSEDED 2026-08-25
:: FLAGS DELIBERATELY OMITTED: --parallel 2 (+22.0% aggregate, -35.4% per slot with
:: the drafter on, measured 2026-08-25 — see section 03's axis table).

Intel Arc Pro B50 · 16 GB

The file: UD-Q2_K_XL — 9.83 GB, 9.154 GiB resident. A 4-bit file does not fit, and at 224 GB/s this card is very slow as soon as any layer has to sit on the processor, so every byte must be resident. The window: -c 65536. VRAM: 13,982 MiB meas on the reference 3090 at this window with this drafter — a memory requirement is a property of the model and the flags, so it transfers to this card even though no speed here does. Room left for your desktop: 606 MiB of the 16 GB. The effort levels it offers: low and medium. xhigh is not offered, on the same window constraint as the NVIDIA 16 GB recipe: 61,500–75,800 tokens of appetite against a 65,536-token window, so it fits on short runs and truncates on long ones. Expected speed: 8–12 t/s der from this card’s 224 GB/s and the bandwidth formula in §04 — no Intel card was ever measured for this page.

Changed 2026-08-25, and this recipe was the one that had no warning on it. It shipped UD-Q3_K_XL at -c 49152 budgeted at “~15.3 GiB” SUPERSEDED, copied from the NVIDIA 16 GB recipe. When that budget was measured rather than derived it came out at ~16,218 MiB, 1,142 MiB over what a 16 GB card has to give (§08). The NVIDIA recipe got a correction note when that was found; this one was missed for a day, which is exactly how a copied recommendation outlives the measurement it was copied from.

:: Q4 does not fit, and at 224 GB/s any layer left on the CPU is very slow - so run
:: the 2-bit file with every byte on the card, Vulkan build. MEASURED 2026-08-25:
:: UD-Q2_K_XL at -c 65536 needs 13,982 MiB; UD-Q3_K_XL at that window needs 16,906
:: and does not fit at all. Section 08 has the full requirement table.
:: file: unsloth/Qwen3.8-27B-GGUF (UD-Q2_K_XL, 9.83 GB = 9.154 GiB)
llama-server.exe -m Qwen3.8-27B-UD-Q2_K_XL.gguf  --alias qwen/qwen3.8-27b
  -c 65536  -ngl 99  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0
                                :: 13,982 MiB measured, leaving 606 MiB of a 16 GB card for
                                :: your desktop. --spec-type none buys back ~1.3 GiB if you
                                :: would rather have headroom than the drafter. A >131k window
                                :: cannot fit resident on this card
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75  :: start at 4, sweep (section 06)
  --jinja --host 127.0.0.1 --port 1234
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"  :: xhigh NOT OFFERED: 61.5-75.8k
                                :: appetite against a 65k window - a WINDOW limit (section 09)

DGX Spark · GB10, 128 GB unified memory

The file: Q4_K_M — 16.5 GB, 15.4 GiB resident. With 128 GB of unified memory, capacity is not the constraint; bandwidth is, at 273 GB/s, so the smallest strong file wins. The window: the full -c 262144. The effort levels it offers: all three — but note the wall clock: the reference 100k-token run derives to about 2–2.8 hours here, so xhigh is an unattended-run setting. Expected speed: 10–15 t/s, derived from bandwidth. Room left for your desktop: not applicable — this machine has 128 GB of memory shared between the chip and the system.

:: llama-server (ARM64 Linux build — no .exe here), same stack as every other card.
:: Bandwidth (273 GB/s) is the ceiling, so the smallest high-quality file wins:
:: Q4_K_M (16.5 GB) over UD-Q4_K_XL (17.6 GB) is ~6% more t/s on a bandwidth-bound
:: box — and the measured perplexity favours it too (section 08).
:: file: lmstudio-community/Qwen3.8-27B-GGUF (Q4_K_M, 16.5 GB = 15.4 GiB)
llama-server -m Qwen3.8-27B-Q4_K_M.gguf  --alias qwen/qwen3.8-27b
  -c 262144  -ngl 99            :: 128 GB unified: full native context is trivial (~11 GiB of
                                :: KV at the MEASURED slope with the drafter on, not the 8.5
                                :: the arithmetic suggests)
  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75  :: start at 4, sweep
                                :: (section 06) — UNMEASURED on GB10; the 3090 gained +45% on
                                :: real code from the same flags
  --jinja --host 127.0.0.1 --port 1234
  --chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}"  :: window holds it; wall clock
                                :: is the cost — ~2-2.8 h for the 100k reference run (section 07)
:: faster vendor alternative: Blackwell-native NVFP4 on vLLM (~1.5x BF16 speed, FP8 KV,
:: 92-97% accuracy) — vllm serve unsloth/Qwen3.8-27B-NVFP4 --max-model-len 131072.
:: llama.cpp cannot load that release (FP8 lm_head); switching stacks buys the FP4 path

Intel Core Ultra iGPU · Arc B390-class

The file: Q4_K_M — 16.5 GB, 15.4 GiB resident; every gigabyte re-read per token hurts at 153.6 GB/s. The window: -c 65536 as an interactive default. This is not a memory limit — an integrated graphics chip borrows system memory — but a practical one. The effort levels it offers: low and medium at this window; xhigh is not offered here for two independent reasons, and both need naming because they have different fixes: the 65,536-token window cannot hold a 61,500–75,800-token thought (raise -c to fix that), and at 5–7 t/s an xhigh answer takes three to four hours (only patience fixes that). Expected speed: 5–7 t/s, derived, and it scales with your memory bandwidth. Room left for your desktop: ample — an integrated graphics chip borrows system memory — but leave the processor and operating system some working memory.

:: llama-server.exe, VULKAN build (SYCL has known perf bugs on current Intel GPUs;
:: IPEX-LLM is archived). RAM is the spec that matters: the B390 tier (Core Ultra X9
:: 388H) ships dual-channel LPDDR5X-9600 = 153.6 GB/s, up to 96 GB shared.
:: Intel's formal Arc-branding floor for B390/B370 is 7,467 MT/s — slower RAM relabels
:: the iGPU as generic "Intel Graphics" — and other Core Ultra configs may ship slower
:: RAM or ONE module (single channel = half the bandwidth = half the t/s). The ~5-7 t/s
:: scales with YOUR bandwidth via section 05's formula; the model fits easily.
:: file: lmstudio-community/Qwen3.8-27B-GGUF (Q4_K_M, 16.5 GB = 15.4 GiB)
llama-server.exe -m Qwen3.8-27B-Q4_K_M.gguf  --alias qwen/qwen3.8-27b
  -c 65536  -ngl 99             :: interactive default, NOT a memory limit: an iGPU borrows
                                :: system RAM, ~57% by default. To raise it: Intel Graphics
                                :: Software >=25.26.1602.2 with driver >=32.0.101.6974 -> the
                                :: GPU's General tab -> "Shared GPU Memory Override" slider
                                :: (~87%; up to ~93% on B390/B370 with current Arc Pro
                                :: drivers, RAM-dependent) -> reboot. Needs >=10 GB RAM, select
                                :: Core Ultra systems. With 96 GB RAM even -c 262144 fits
  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0   :: verify KV-quant support in the Vulkan build
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75  :: start at 4, sweep
                                :: (section 06); UNMEASURED on iGPUs — drop the spec flags if
                                :: a paired probe shows no gain on your machine
  --jinja --host 127.0.0.1 --port 1234
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"  :: xhigh NOT OFFERED, for TWO
:: separate reasons with two separate fixes: 61.5-75.8k of appetite against this 65k
:: WINDOW (fix: raise -c, you have the RAM), and 3-4 HOURS per answer at 5-7 t/s
:: (fix: nothing but patience). Raise -c and effort together for overnight runs only.
:: possibly-faster vendor alternative: Intel's OpenVINO Model Server on the iGPU with
:: OpenVINO/Qwen3.8-27B-int4-ov (OVMS LLM quickstart; OpenAI-compatible /v3 endpoint) —
:: Intel tunes that stack for its own silicon; benchmark both if the t/s matters

Standardized industry metrics

What each recipe costs in electricity, in the units the industry compares on. Read the two plain-language conclusions first: turning the drafter on is the single cheapest energy decision on this page — it cuts the energy per token by 2.52× — and a slower setting is a more expensive setting in exactly the same proportion, because while the model is generating, this card draws essentially the same power whatever it is doing, so the joules per token are just those watts divided by the tokens per second. The whole energy win is throughput. Nothing here saves power; it saves time at the same power, which is the same thing arriving by a less interesting route.

Instrumentation tier for every figure below: in-band GPU board power — read from the card's own sensor rather than from a meter at the wall (NVML, nvidia-smi --query-gpu=power.draw). Inside the number: the graphics chip, its memory, the board's voltage regulators and fans. Excluded and never measured on this machine: power-supply conversion loss, the processor, system memory, drives, chassis fans, the display, and any datacentre overhead. Do not call any of these figures system power or wall power, and do not divide an electricity bill by them.

The idle pair, dated. Settled, with little on screen, first 60 seconds of each idle window discarded (2026-08-23 power matrix): 29.9 W with no server running, 34.1 W with the model resident and answering nothing. A resident model costs +4.2 W — small, and in the physically sensible direction. Earlier editions of this page printed 33.2 W with no server against 30.7–31.1 W loaded, which is backwards; those readings were taken on a board that was still cooling from the previous job. The 34.1 W figure is what the idle-subtracted columns below remove.

Sustained board power is a range, not a constant. Across 2.5 hours of drafter-off benchmark work the board drifted 305.5 → 341.1 W (+11.7%) at constant throughput, constant 77–79 °C and constant memory clock, tracking its own SM clock from 1,453 to 1,606 MHz. Decode at one request at a time is limited by memory bandwidth, so the extra clock bought no tokens and cost about 6% more energy per token. Plan with the band — 306–341 W drafter-off, 338–345 W on the long drafter-on authoring runs, 302–339 W across the short power-matrix arms — or with your own run's mean. Earlier editions printed a flat "~344 W"; that is the top of a range.

Conditions, and the ways each row differs from the recipe it prices. All four energy arms ran on 2026-08-23 at -c 32768 with a 1,458-token prompt and 700 generated tokens, thinking off, no projector, three requests per arm, 500 ms power sampling, 100% coverage. Row [1] and [2]: the recipes serve at 122,880 and 180,224 tokens of window with the projector loaded. Window allocation does not change joules per decode token; prompt depth does, and the depth series is its own set of arms with its own shallow reference point — 3.919 J/token at a 1.5k fill, 4.453 at 28k, 5.598 at 91k. Read that series against its own 3.919, not against this row's 3.743: the two shallow points sit 4.7% apart in energy and 6.4% apart in throughput, both outside the 2.9% floor, because they are different arms on different prompts. The projector is decode-free (0.04–0.09% at a 91k fill), so it changes nothing here. Rows [1] and [2] both decoded faster than the matched sweep did at the same flags — 91.3 against 83.5 for row [1] (9.3% apart) and 106.2 against 93.9 for row [2] (13% apart), unexplained and an open item in the campaign log. Plan speed from the sweep's 83.5 and 93.9; use these rows for energy. The energy ratios survive because every arm in the matrix shares whatever drift this is. Row [4]: this arm ran with the drafter off while the recipe ships n4/p0.75. On UD-IQ4_XS the same drafter cut joules per token by 2.17× (8.104 → 3.743), so recipe [4]'s own figure is expected near 4.1 and is unmeasured — the printed 8.853 is this file's no-drafter cost and is the right number for comparing files, not for pricing this recipe. "Wh per 700-token answer" means exactly that: every arm generated 700 tokens and stopped, so these are not complete-answer figures. Complete-answer energy, for one real authoring task, is in §09. Net columns subtract the dated 34.1 W loaded idle over the covered seconds and are derived; the campaign's own runner subtracted 31.0 W instead, a uniform shift of about 1% that changes no ranking.

Energy per setting: the J/token table

Every knob this page recommends on, priced in energy, so that a setting chosen for speed can be checked for electricity too. The pattern underneath all of it is simple and was verified with no residual left over: joules per decode token equals the watts drawn while decoding divided by the decode tokens per second. That qualifier matters — the mean-watt column in the table above spans each request's whole window, including a short low-power prefill, so it runs 6–11% below the decode figure and dividing it gives the wrong answer. Anything that raises throughput at constant power lowers energy per token in the same proportion. That is why the drafter is an energy feature and why depth is an energy problem.

04

Five model facts that decide every setting

Five things about how this model is built explain almost every recommendation in §03. It stores far less memory per token of context than its size suggests, so long contexts are cheap. Its context limit is 262,144 tokens, and what you will actually run out of is card memory, not model capability. It thinks before answering by default, and that thinking comes out of the same pool as your prompt. Vision is optional at start-up and costs memory rather than speed. And it wants a specific sampling setup that some clients quietly override, which causes the model to repeat itself. Everything below is those five facts with their numbers.

One more fact governs speed rather than settings, and it is worth having in plain words before the arithmetic: with speculation off, a bigger file is a slower file. To write each token, the card has to read the whole model out of its memory once. A file that is 2 GiB larger takes proportionally longer to read, every single token, forever.

The drafter breaks that rule, and it is why this page no longer simply recommends the smallest file that holds its quality. meas Turn speculation on and the ordering can invert: UD-IQ4_XS goes 42.34 → 86.91 t/s (2.05×) while the smaller UD-Q2_K_XL goes 45.66 → 77.01 (1.69×) — so the file that is faster drafter-off is slower drafter-on, by 12.9%. The draft head is quantized along with the model, so a more damaged file drafts worse and gets less back from speculation. The window moves the ordering too (§08). Since every recipe here ships with the drafter on, “smallest that holds its quality” is the right rule for memory and the wrong one for speed.

• Hybrid attention. Most of its 64 layers use Gated DeltaNet, a linear-attention design whose memory use stays constant no matter how long the context gets; only 16 of the 64 are standard attention layers (and those use grouped-query attention with 4 KV heads). The result: the KV cache costs just 64 KiB per token at fp16 — 4.0 GiB at 122k context with an 8-bit cache, 7.5 GiB unquantized — where a dense 27B storing KV in every layer would need exactly 4× as much. Those are the arithmetic, and arithmetic is a floor: what a real server allocates measured 15–29% steeper (§05). Huge contexts are cheap in memory here, and speed falls off gently as they fill rather than collapsing.
• Native 262,144 context. No rope or YaRN tricks are needed below that; the limit you will actually hit is VRAM. The model card lists extension to 1M tokens via YaRN, which is out of scope here: the KV cache alone would run about 34 GiB even at q8_0, by the same arithmetic.
• Thinking is always on, and defaults to xhigh. Reasoning tokens share the context window with your prompt and your output, so a 100k-token thinking run needs a window bigger than 100k (§09). This is also why every speed number on this page states its token regime.
• Vision is optional at serve time — and it costs context, not speed. The BF16 mmproj projector occupies 1,138 MiB of VRAM once loaded — not the 0.867 GiB its file weighs, which is what earlier editions of this page quoted. The independent re-measurement (2026-08-23, same machine, UD-IQ4_XS) measured the resident cost two independent ways and they agreed to the megabyte, and a follow-up pair at 91k of depth reproduced the same 1,138 twice more. At the measured window slope that is about 26,500 tokens of 8-bit window with the coding drafter loaded, 29,900 without it (§05). What it does not cost is speed: at 90,862 tokens of fill, decode with and without the projector differed by 0.04–0.09% (§06) — the only question the projector asks is about window. Serving text-only converts vision you are not using directly into context, but not all the way to the native maximum: with the drafter on, UD-IQ4_XS's measured text-only ceiling is 180,224, and 262,144 fits only with the drafter off, and then only just (§03 prints both variants).
• Official thinking-mode sampling: temperature 1.0 · top-p 0.95 · top-k 20 · min-p 0. Clients that silently send temperature 0 — aider does — cause repetition loops. Override them; §14 shows how, per agent.

The arithmetic that predicts decode speed

Generation speed comes down to one ratio: how fast your card can read its memory, divided by how much it must read per token. Every token requires reading the entire model — about 16.5 GB for a 4-bit K-quant, 14.25 GB for UD-IQ4_XS. So take your card's bandwidth in GB/s, divide by your file's GB, and multiply by an efficiency constant, and that is your speed ceiling in tokens per second. That constant is format-specific, which earlier editions of this page flattened into a single ~0.7: on CUDA it measures about 0.70 for K-quants and about 0.65 for IQ-quants. You can tell which family a file belongs to from its name: a name containing IQ — UD-IQ4_XS, UD-IQ3_XXS, UD-IQ2_S — is an IQ-quant, so use 0.65; a name containing K without IQ — Q4_K_M, UD-Q3_K_XL, UD-Q2_K_XL, Q6_K — is a K-quant, so use 0.70. The independent re-measurement measured 0.649 on UD-IQ4_XS — 936 GB/s ÷ 14.25 GB = 65.7 t/s theoretical against 42.6 measured — and Q4_K_M lands at 0.700: 936 ÷ 16.5 = 56.7 against 39.7 measured. This one calculation predicts every row of the matrix in §07, and it doubles as a fault detector: land far below your format's constant at short context and something is wrong (§11).

05

What fits in 24 GB, and where speed collapses

A 24 GB card has two ceilings, not one, and the difference between them is the most expensive misunderstanding on this page. The first ceiling is how large a window you can set and still have every byte of it live on the card — fully resident, in the words of §02: about 131,000 tokens for the reference configuration. The second is how large a window you can set before the server refuses to start at all: about 213,000. Between those two numbers the server starts happily, answers a short prompt at full speed, and then collapses to a fifth of that speed the first time somebody actually fills the window. Set your window under the first ceiling, not the second, and never trust a ceiling that was checked with a short prompt. The rest of this section is the memory arithmetic that tells you where your own first ceiling is.

The second law of this section: speed against context is a cliff, not a curve. It is flat while the weights and the cache fit in VRAM, and it collapses once anything spills across the PCIe bus — with the magnitude set by how much spilled: about 10× when the weights file itself lands in system memory; 1.1–2× from a 1.9 GB spill of weights answering a short prompt (§11); and 2.5–3.8× once the spilled pages are cache that a deep prompt actually reads (the table below). What decides the size is not how much spilled but how often the spilled pages are touched. Measured on the reference 3090:

That corollary now has a measured face. Probed 2026-08-22 with Q4_K_M (q8_0 KV, drafter on, projector loaded — the 1 GiB-larger UD-Q4_K_XL shifts every ceiling about 30k tokens lower, while dropping --mmproj frees 1,138 MiB and raises them about 26,500) by stepping -c upward with short temperature-0 decode probes: speed holds at 50–55 t/s all the way to -c 212992, turns borderline at 217,088 (41.2 t/s), and collapses at 221,184 (19.5 t/s). But nvidia-smi tells the other half of the story: dedicated VRAM hits its ~24 GB ceiling already at -c 131072 (measured 24,052 MiB). Past that point the KV cache is overcommitted, and the run stays fast only while a short prompt leaves the deep pages untouched. So: about 131k is the practical fully-resident ceiling — fast even when a long run really fills it, though it sits right at the limit, which is why the recipe ships 122,880 — while up to about 213k will load and answer a short prompt. What earlier editions of this page got wrong is what happens next.

Prefill collapses with it — 819 → 183 t/s at 90.9k, 1,085 → 408 at 29.8k — while draft acceptance is unchanged (0.907 against 0.849 at 90.9k), which rules out any drafting explanation: this is purely the KV cache being read across the PCIe bus. The practical reading: a reader who trusted the shallow ceiling gets 26% of the promised speed the first time they open a 90k-token document — 8 t/s on a card that does 50 — and nothing in the server log says why. Never label a window resident or safe on the strength of a short probe; fill it once, deeply, before you believe it.

One caveat on the table's own numbers, added 2026-08-23 by the follow-up round: every cell in it is a single decode probe fired immediately after its prefill, and that is the one moment this card is not at its boost clock — the ±25% band declared in §01 and measured in §11. Re-measured under the cooled protocol, the resident 90.9k row reads 36.6 t/s rather than 30.6, about 20% higher. The spilled column was taken the same way and carries the same uncertainty, so read the penalty ratios as approximate. Nothing about the verdict moves: a 2.5–3.8× collapse is an order of magnitude outside a ±25% band.

The VRAM budget table

The cliff chart says where speed dies; this table says why — and the independent re-measurement (2026-08-23, same machine, UD-IQ4_XS) replaced most of its estimates with measurements. Earlier editions budgeted by arithmetic: 64 KiB per token at fp16, 34,816 bytes per token at q8_0 with block scales, plus "1–1.5 GiB of compute buffers" — the scratch memory a server needs while it is working — and an assumption of "22–23 GiB usable". Every one of those is a floor, not a figure. The campaign instead read llama-server's own dedicated-VRAM counter across 17 fully-resident server loads and fitted two constants per configuration — a fixed base plus a per-window-token slope. The model reproduces all 17 loads to within 127 MiB, so it is what this page now budgets with. (Those 17 are the fully-resident subset of 26 loads in total; the other nine were overcommitted configurations, which the model does not describe. Later measurement rounds loaded further configurations that have never been checked against it.)

Conditions: RTX 3090, 24,576 MiB, UD-IQ4_XS, q8_0 KV, -ngl 99, --parallel 1, llama.cpp build 10502. Worked example derived — §03's the default recipe with vision at -c 122880, if you put the n-max 10 coding drafter on it: 13,232 + 1,138 + 1,008 + 898 + (122,880 × 45,056 ÷ 1,048,576 = 5,280) = 21,556 MiB of server-process VRAM, leaving 3,020 MiB (2.95 GiB) of board VRAM for the desktop and slack. Drop the 898 and the same row keeps 3,918 MiB (3.83 GiB) — which is why §03 ships the vision recipe at n-max 4: the wider drafter setting fits here, it just spends the slack that absorbs a desktop measured anywhere from 133 to 1,181 MiB.

Two rules follow, and both change how you read every ceiling in this guide. First, budget on the server's own report, not on the board VRAM total — the three counters and what each can see are in §02. The nvidia-smi board VRAM figure includes your window manager and your browsers; the campaign watched one identical configuration read 474 MiB apart on the board VRAM — about 11,000 tokens of window — with nothing changing but what was on screen, while llama-server's own number moved by at most 127 MiB. That spread is the whole explanation for "same command, different ceiling". Second, the drafter is a line item, not a freebie. At -c 163840 with vision, its fixed and per-token costs together book roughly 1.8 GiB — more than the vision projector. On a pure prose workload, where speculation was measured worth only about 1.16× (§06), that is the easiest 1.8 GiB on the card to reclaim: --spec-type none.

With those constants in hand, what fits on a 24 GiB card by file and window — the KV column stated as the q8_0 arithmetic floor, so add 15% (drafter off) or 29% (drafter on) for what a server really allocates:

Two footnotes the table earns. First, the q4_0 KV flag, which earlier editions ruled out on mechanism alone and which has now been measured (2026-08-23, same corpus and flags as the q8_0 run): a q4_0 K/V cache costs +0.693% perplexity against fp16 — 6.6413 against 6.5956 — where q8_0 costs +0.309%. The degradation is super-linear in bits: the second halving of the cache costs slightly more than the first (+0.383% on top of q8_0), which is the shape the mechanism predicts and the reason the cut stops paying. That mechanism is unchanged and still unmeasured directly: the problem is not that quantization error accumulates — each cache entry is quantized once — it is that at long context every new token attends over hundreds of thousands of perturbed entries, and retrieval gets unreliable at exactly the 200k-plus lengths a q4_0 cache would enable. What the number changes is the tone. 0.69% is small — smaller than the perplexity gap between the two 4-bit files this page recommends (0.93%) — so "never q4_0" was too strong. q8_0 remains what every recipe ships because it already buys the context ceilings here; q4_0 is a window-extension trade to take knowingly (§08 carries the row and its caveat). Second, prefill: these budgets make a 200k-plus prompt fit, and earlier editions estimated that filling one costs "several minutes at 100k, potentially tens of minutes at 200k+". The independent re-measurement measured it instead, across a 60× span of prompt lengths on a fully-resident -c 131072 server:

Conditions: UD-IQ4_XS, -c 131072 fully resident, q8_0 KV, drafter at n-max 4 / p-min 0.75, no projector, prompts prefixed with a unique identifier so the prefix cache (§02) cannot be reused (2026-08-23). The energy column comes from a separate 2026-08-23 arm at 1.5k / 28k / 91k of fill, not from these five prompts, and its depths are close but not identical. Use about 1,000 t/s as the planning figure: prefill decays only 25% over a 60× longer prompt, because the quadratic term lives on just 16 of the 64 layers (§04) — the hybrid dilutes it. A genuinely full 100k prompt is about 2 minutes, not "several"; extrapolating the same slope, 200k lands nearer 5 minutes than tens of them. Overcommit the window and this number collapses with everything else: 183 t/s at 91k in the correction above.

06

Speed: which flags to set, and what the drafter buys

Six flags belong on every card, and each one is here because a measurement put it here. Put every layer on the graphics card with -ngl 99: writing -ngl 64 looks complete and is not, and it costs about 30% of your speed with no warning anywhere. Serve one request at a time with --parallel 1. Store the memory of the conversation in 8-bit with -ctk q8_0 -ctv q8_0: it halves that memory and costs 0.3% of perplexity. Turn the built-in drafter on with --spec-type draft-mtp: it roughly doubles your speed, changes nothing about what the model writes, and cuts the electricity per token by two and a half times. Choose --load-mode none if system memory is tight and --load-mode mmap if you restart the server often. And whatever number you end up quoting, say which kind of token you counted, because on this model that alone is worth a factor of 1.7.

• All layers on the GPU, or accept the cliff — -ngl 99. llama.cpp counts the output layer as one more layer than the model has transformer layers, so on this 64-layer model -ngl 64 reads as complete and leaves that output layer — a 5120 × ~151k-vocabulary matrix multiplication — on the processor, running for every generated token. Measured cost: 25.7 against 39.7 t/s on Q4_K_M, and independently 29.84 against 42.31 on UD-IQ4_XS — −29.5% — with no memory signature at all (§11). Partial offload is set by system memory bandwidth, not by the graphics card, so a 5080 and a 4060 Ti are predicted to offload at nearly the same 8–12 t/s der — that band is §04's formula run at system-memory bandwidth, not a measurement: neither of those cards has ever been in this machine, and the only offload figure measured here is the 25.7 t/s above, on this 3090. And this is a dense model — every part of it runs on every token — so the trick of parking a model's rarely-used parts on the processor has nothing to target here.
• One slot — --parallel 1, and this is now settled rather than quarantined. Some interfaces default to 2, which halves the window each request gets and doubles the KV cache. Two earlier measurements of running two requests at once disagreed badly — one found +11% aggregate throughput, another found +60.3% — and the second of those ran with the drafter off, so this page refused to use either figure until a decisive test could settle them — it called that refusal a quarantine — and named the matched pair that would run it. That pair ran on 2026-08-25. On UD-IQ4_XS with the drafter on at n-max 10 / p-min 0.5 — the faster shipped setting — three reps each, first post-prefill probe discarded: --parallel 1 gave 82.98 t/s aggregate and 85.79 per slot; --parallel 2 gave 101.25 t/s aggregate and 55.41 per slot — +22.0% aggregate, −35.4% per slot. The +60.3% figure was an artefact of measuring with the drafter off: once the drafter is already saving most of the repeated reading of the model, batching has far less left to save, which was the stated hypothesis for the quarantine and is now measured rather than assumed. Draft acceptance is unchanged across the pair (0.618 → 0.620), so the second slot does not disturb the drafter at all; the whole cost is that each user waits about 55% longer for their own tokens — 12.6 s against 8.2 s for a 700-token answer. Every recipe here still ships --parallel 1, and now for a reason a single user can check: serving one person, one slot is the faster setting (§03).
• 8-bit KV cache — -ctk q8_0 -ctv q8_0, verified twice on two files. Measured wikitext perplexity with the 8-bit cache: 6.5498 against 6.5348 at fp16 on Q4_K_M — +0.23%, inside the error bars — and the re-measurement replicated the effect independently on UD-IQ4_XS at 6.6160 against 6.5956, +0.31%, likewise about one standard error. Half the KV memory for statistically nothing, on both files. It also costs nothing measurable in energy: 8.341 against 8.284 J per decode token, 0.7% apart and well inside the 2.9% floor. The one price that is not nothing is speed: fp16 KV measured 43.19 against q8_0's 42.31 t/s at -c 32768, so 8-bit costs about 2% of decode — when VRAM is genuinely spare, fp16 is marginally the faster cache. The step below is measured too: q4_0 K/V at 6.6413, +0.693% over fp16 (§08).
• The drafter on — --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75. Lossless by construction: every drafted token is verified by the full model before it counts, so quality is never on the table. Measured here it doubles decode on code and cuts energy per token by 2.52× at its best settings. It is not free in memory — 1,008 MiB fixed plus 11.4% of your window (§05) — and on prose it is worth only about 1.16×, which is the one workload where --spec-type none and the memory back is the better answer. The rest of this section is how far it goes and why.
• Load mode — none for tight RAM, mmap for restart-heavy work. Without --load-mode none (which replaces the deprecated --no-mmap) the whole GGUF stays cached in system memory even after the weights are in VRAM — about 15 GB for the 4-bit file. The cost is a slower model load: measured 10–30 s to first health check across dozens of restarts in the campaign proper, and it is storage-dependent. The independent re-measurement ran about 30 restarts on --load-mode mmap instead and measured warm reloads at 6–10 s with decode unaffected (repeat probes agreed to 0.7%) — because at -ngl 99 every weight lands in VRAM either way, so load mode cannot touch decode speed. So: if you are sweeping flags or restarting dozens of times a day, mmap is the better choice. Every recipe in §03 nonetheless ships none, and that is a deliberate default rather than a contradiction: none is the safe assumption about a machine this page cannot see, because the failure it prevents (system memory exhausted by a cached model file) is worse than the one it causes (a load 20 seconds slower). Change it if you know your machine has the memory to spare. The one thing that campaign did not re-verify is the 15 GB saving itself; it remains untested rather than recommended.
• Say which tokens you counted. This model thinks by default, so a probe labelled "code generation" can spend every timed token reasoning about code and return an empty answer field. On code, answer tokens run about 1.7× faster than reasoning tokens on the same server with the same flags. Every speed row below declares its regime.

Flash Attention: you are already using it, and the recipes depend on it

None of the recipes on this page passes -fa, and none of them needs to. This build documents -fa, --flash-attn [on|off|auto] with a default of auto — so a recipe that omits the flag is not running with Flash Attention off, it is letting the server decide. On this machine it decides on meas, and that is not an inference: with -fa unset and -ctk q8_0 -ctv q8_0 the server loads fine, while the same configuration with -fa off refuses to start. Acceptance also reads identically across auto and on — 0.611 shallow and 0.571 deep, to three decimals — and deep decode differs by 0.1 %.

What Flash Attention actually is. It is a different way of computing attention, not a quality setting. Ordinary attention builds the whole attention matrix in memory and then reads it back, so its memory traffic grows with the square of how much context it is attending over. Flash Attention fuses those steps so the large intermediate is never written out at all. The arithmetic is the same; the memory traffic is far smaller, and the saving grows with depth.

llama_init_from_model: quantized V cache requires flash_attn to be enabled
srv    load_model: failed to create_context with model ...
srv    llama_server: exiting due to model loading error

So should you set it? No — leave it alone. Adding -fa on buys nothing over the default on this card (0.1 % at depth, inside noise), and pinning a flag you have not measured on your own backend removes the one mechanism that would have chosen correctly for you. The reason to know about it is diagnostic, not performance: if a recipe here refuses to load and mentions flash_attn, that is this dependency, and the fix is the KV width, not the flag.

One measurement from this run that is not trustworthy, stated so nobody uses it: the prefill figures. Probes reused a cached prefix, so the server reported a full prompt_n against a near-zero prompt_ms and the resulting “prefill t/s” is meaningless. Only the decode, acceptance, VRAM and load-success results from this arm are reported anywhere on this page.

Every speed this page measured, in one table

Read the band, not a number. The floor is set by how fast the card reads its own memory, and everything above that floor is the drafter, so the right row for you is the one whose content and token regime match your work.

All rows measured 2026-08-21 to 2026-08-23 on the reference RTX 3090 at short context, from the server's own timings. The floor row is also the diagnostic threshold, and it is published as a band rather than a point for that reason: it was established across five contents and both token regimes and holds inside 3.6%.

How speed falls as the prompt grows

Speed drops as the conversation gets longer, gently and predictably, and the drop is about 25–30% between an empty window and a 91,000-token one. It is not a cliff — that only happens when the window overflows the card (§05). What matters far more than the depth is which kind of token you are counting: at the same 91k depth, the answers a reader keeps arrive at 64.8 t/s while the thinking that precedes them runs at 36.6. Plan an agent session on the first number and a long xhigh run on the second.

All three series below are UD-IQ4_XS at -c 131072, fully resident, with prompts prefixed by a unique identifier so nothing is served from the prefix cache. The first column is the one to plan against: it was measured 2026-08-23 under the cooled protocol (§02, documented in §11) — the probe that comes back with the prefill discarded, three probes per depth off the cached prefix — while the other two columns are single probes taken right after their prefill and therefore carry the ±25% band from §01.

The last column is not a measurement of the model: it is the same reasoning-series runs divided the wrong way, tokens ÷ total wall-clock, which buries a 26-second or 105-second prefill in the denominator. It is printed here so that a reader who computes 9.2 t/s on a card that decodes at 47.1 recognises their own arithmetic instead of blaming their hardware. Quote the server's own timings, and quote prefill separately (§05). Why the acceptance columns differ so much: the cooled ladder re-used the campaign's red-black-tree prompt, a textbook algorithm the draft head predicts extremely well, while the middle column generated novel code. Acceptance ranks the content, not the protocol — and, as the next part measures, it does not rank speed at all.

Three things to take from it. First, the old "34–44 t/s at agent depth" band understated a real coding session by about 2× — it was a reasoning-token number, and the tokens an agent actually receives run 65–70 t/s at 91k of depth. Second, "acceptance never moves" was wrong in the safe direction: acceptance rises with depth, 0.80 → 0.92 in the reasoning series and 0.60 → 0.65 in the answer series — two independent confirmations. Decode still falls 25–30% across the same span (86.3 → 64.8 on the cooled ladder), which settles the mechanism: the cost is per-token KV reads growing with depth, not the drafter failing. Depth and acceptance are independent axes. Third, the decline is gentle and it is not a cliff — as long as the window is genuinely resident. Overcommit it and the same depths give 18.6 and 8.0 t/s instead (§05).

The mechanism under the two regimes: mean draft length

Line up the cooled ladder against the reasoning column — same file, same flags, different protocols — and the reasoning side sits a near-constant 1.7× below at every depth: 86.3/51.2 = 1.69, 80.2/47.1 = 1.70, 64.8/35.8 = 1.81 — so call it 1.69–1.81×. A ratio that stable across a 60× span of depth is not clock noise, so it was isolated properly (2026-08-23): one server, one 90,894-token prompt, the same n-max 4 / p-min 0.75 flags, cooled probes, the only change being the chat template:

A 1.69× speed difference produced by nothing but the token stream — and acceptance says nothing about it (0.895 against 0.907, essentially tied). The mean-draft-length column is where the difference actually lives. On uncertain reasoning tokens the p-min 0.75 confidence gate keeps cutting the draft tree short — a mean of 2.99 tokens per pass against 4.31 on the answer stream — so acceptance stays high precisely because the gate is discarding the part that would have missed. That makes acceptance blind to the loss. The refinement this page now carries everywhere: acceptance tells you whether a draft was right; mean draft length tells you how fast you will go. It is the same fact that makes the highest-acceptance configuration the slowest one in the sweeps below.

The energy column comes from a different pair of arms on a different day's prompt — 2026-08-23's power matrix, thinking on against thinking off at short context — and is printed beside these rows because it measures the same mechanism on a completely independent instrument: 1.62× in joules per token against 1.69× in throughput. Two instruments, one effect. Note also the floor row: 27.01 t/s thinking-on against 26.95 thinking-off — identical, which confirms once more that content dependence on this page is almost entirely a speculation effect: without a drafter, what you are writing moves decode by at most 3.6%.

The flags cannot buy the difference back. At the same depth with thinking on, --spec-type none gives 27.01, the shipped n4/p0.75 gives 36.62 (1.36×) and the code-optimal n10/p0.5 gives 38.67 (1.43×) — a 5.6% gain where the same flag change is worth 12% on code. So the practical number for the configuration this page ships by default: anyone running xhigh over a 91,000-token document should plan on 37–39 t/s, not the 62–70 t/s the code probes advertise. The deliverable arrives at code speed; the thinking that precedes it does not.

What the built-in draft head is

Qwen3.8-27B ships with a small helper network inside the same file — a multi-token prediction (MTP) draft head — that guesses several tokens ahead so the full model only has to check them, which is much faster than generating each one. The standard K-quant GGUF files (§08's table) already include it, and the head itself always stays at 8-bit whatever the main quantization is. llama.cpp added support in July 2026, and because every guess is verified by the full model before it is accepted, the output is exactly what the model would have written anyway — you only gain speed. It works on any GPU llama.cpp supports.

  --spec-type draft-mtp  --spec-draft-n-max 4  --spec-draft-p-min 0.75
  :: a safe starting point, not the peak — the optimum moves with the CONTENT, and NOT
  :: with the file: n-max 10 / p-min 0.5 wins every code row on BOTH quants (sweep below).
  :: And it is not free: 1,008 MiB + 11.4% of your window, +898 MiB at n-max 10 (section 05).

How much speed it adds depends heavily on hardware and workload — published reports range from +33% to +145%, and one RTX 5090 tester saw 148 average / 203 peak t/s on the NVFP4 tiers. On the reference 3090, generation went from 40.3 (the 08-21 build's cold-start baseline; the canonical 08-22 baseline is 39.7–39.9) to 58–60 t/s — +45% at the default-ish settings — and on the original benchmark prompt, tuning pushed it to 81.7 t/s, +105%, a figure a 2026-08-22 re-sweep appeared to overturn and a 2026-08-23 matched sweep reproduced to the decimal once the token regime was pinned. The warning all these numbers carry: benchmark on your own machine. That 5090 tester found n-max 4 fastest with 6–8 slower; this 3090 agreed on real code yet kept climbing to n-max 10 on its near-ideal prompt. The optimum moves with hardware and content, so there is no universal best value.

DFlash2, and why it lost

There is a second way to draft, and on this machine it was slower than the one already inside the file. DFlash2 predicts a whole seven-token block in a single pass, then picks the most plausible sequence out of the candidates it considered for each position. Like the built-in head, every draft is verified before it is accepted, so the output is identical to running the model alone. The costs are real: a separate 1.14 GB drafter to download (incoai/Qwen3.8-27B-DFlash2-GGUF, about 1.2 GB of VRAM) and — until llama.cpp PR 27342 is merged — building llama.cpp yourself from that branch. On its best benchmark prompt it reached 69–77 t/s; on realistic code its best was 46.5 t/s at n-max 2, below the built-in head's 57.9. That is the whole verdict.

:: shared flags in every run below:
llama-server -c 32768 -ngl 99 --parallel 1 -ctk q8_0 -ctv q8_0 --jinja --port 1235

:: row 1 — baseline: just the model, no speculative flags
  -m Qwen3.8-27B-Q4_K_M.gguf

:: row 2 — built-in MTP, n-max 4 / p-min 0.75 — the everyday pick (the head ships inside the main GGUF):
  -m Qwen3.8-27B-Q4_K_M.gguf --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75

:: row 3 — MTP at n-max 10 / p-min 0.5 — the peak on answer tokens († = measured warm-cache):
  -m Qwen3.8-27B-Q4_K_M.gguf --spec-type draft-mtp --spec-draft-n-max 10 --spec-draft-p-min 0.5

:: row 4 — DFlash2 (separate 1.14 GB drafter via -md; needs the PR 27342 build):
  -m Qwen3.8-27B-Q4_K_M.gguf -md Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-type draft-dflash --spec-draft-n-max 7

:: rows 5-6 — NVFP4 files with their embedded MTP head (same MTP flags):
  -m Qwen3.8-27B-NVFP4-MTP-HIGH.gguf --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75
  -m Qwen3.8-27B-NVFP4-MTP-VERY-LOW.gguf --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75

:: what each speculative flag means:
::   --spec-type            which drafting method: draft-mtp (built-in head) or draft-dflash (external drafter)
::   -md / --model-draft    path to the external drafter GGUF (DFlash2 only)
::   --spec-draft-n-max     how many tokens to draft per step - the main tuning knob, sweep 2-12 on your machine
::   --spec-draft-p-min     drafter confidence floor 0-1; below it, drafting pauses for that step

That table says two things. First, speculation is real free speed: the re-swept everyday flags measured 57.9 t/s on realistic code, and the temperature-1.0 xhigh sweep sustained 48.8 t/s across a 73,000-token thinking run — putting the 100k reference run at roughly 34 minutes, down from 45. Second, the NVFP4 rows show that the quantization and the drafter interact: speculative decoding makes the GPU check several drafted tokens at once, and that batch-checking is exactly the work NVFP4's software fallback does slowest. So on cards older than Blackwell, pair speculation with normal K-quants, not NVFP4.

Tuning the drafter: three sweeps, two reversals, and one acquittal

The settings advice is short and it is at the end of this part, in the "So what should you set?" box: use n-max 10 / p-min 0.5 where the memory is spare and the work is code, and n-max 4 / p-min 0.75 everywhere else. What takes the space is the story of how this page got there, because it went wrong twice on the way and both mistakes are ones any careful person could repeat. An 18-configuration sweep measured one answer; a second sweep the next day appeared to demolish it; a third, run 2026-08-23 with the token regime finally pinned, reinstated the first and retired the demolition. All three stay on the page, in order.

Two knobs control drafting behaviour, and both mattered more than the defaults suggest.

The comparison table above used the settings each method's authors recommend. Then the campaign swept --spec-draft-n-max across both methods, same prompt, temperature 0, warm cache — a different run state from the cold-start table, so absolute numbers shift a few t/s in either direction: this sweep's baseline reads 39.8 against the table's 40.3, and comparisons belong inside one chart, not across them. The result contradicted both recommendations, and then a second sweep on realistic content contradicted the first:

Three findings, all measured 2026-08-21:

• On this prompt — and only on prompts that draft like it — the tuned built-in head won outright: n-max 10 with p-min 0.5 gave 81.7 t/s, +105% over baseline, beating DFlash2's best (77.6 at n-max 4–5). No drafter download and no source build — but not "no VRAM cost", which is what this bullet used to say. The independent re-measurement (2026-08-23, same machine, UD-IQ4_XS) isolated the built-in head with an on/off pair and measured 1,008 MiB fixed, plus 5,120 B per token of window — 11.4% of whatever -c you set — plus a further 898 MiB for n-max 10 over n-max 4. At -c 163840 that is roughly 1.8 GiB, more than the vision projector costs. The head's own weights are only 334.7 MiB; the rest is draft-path allocation. --spec-type none frees every byte of it, and the server then logs model has unused tensor blk.64.* -- ignoring to prove the head was never loaded.
• --spec-draft-p-min is the hidden second knob. At n-max 10, lowering it from 0.75 to 0.5 was worth +8 t/s (73.5 → 81.7); at n-max 4 the difference was small. A high confidence floor keeps pausing exactly the long drafts that a big n-max exists for.
• DFlash2's recommended n-max 7 was not its own best setting here — 4 and 5 both beat it (77.6 and 77.5 against 74.0). Every recommendation in this space, including this page's, is one machine's measurement.

Conditions: UD-IQ4_XS, -c 32768, -ngl 99, q8_0 KV, --parallel 1, temperature 0 / top-k 1, an 83-token novel-JavaScript prompt, 700 predicted tokens, a fresh server per configuration, reasoning tokens. accepted per target pass = draft_n_accepted ÷ (predicted_n − draft_n_accepted), both fields from the server's own timings.

Rank configurations by accepted tokens per target pass, not by acceptance rate. n-max 4 / p-min 0.75 accepts a near-perfect 89.9% and still loses, because a shallow draft caps how much a single verification pass can deliver. But the metric is a ridge, not a straight line: n16 accepts the most per pass (4.88) and is 8% slower than n10, because drafting deeper costs real time. The same reversal shows up at the extreme: on the verbatim-copy row, n-max 4 accepted a flawless 100.0% (424 of 424) and ran 97.0 t/s while n-max 10 accepted 99.0% and ran 148.7 — 3.96 accepted per pass against 10.54.

Conditions: -c 32768 -ngl 99 --parallel 1 -ctk q8_0 -ctv q8_0 --jinja, enable_thinking:false, temperature 0 / top-k 1, 700 predicted tokens, a fresh server per configuration (the speculative flags are load-time) plus a discarded warm-up, two probes per server, agreeing to about 1%. Answer lengths varied by about 1% between configurations despite temperature 0 — the usual CUDA batch-shape reduction-order non-determinism, with every stream in a file sharing its opening 120 characters. "Identical output regardless of speculative flags" is true in principle and not quite literally true on this build.

Four readings, and the first one retires a claim this page shipped. Same winner, same loser, same shape on both files — n10/p0.5 is fastest on each, n2/p0.75 is the slowest speculating configuration on each, and the ranking between them matches. There is no per-file optimum to split; the split earlier editions published was a token-regime difference wearing a file's name. Acceptance is a property of the draft head, not of the quantization: at six of the seven configurations the two files land within 1.6 points of each other (96.5/96.7, 93.3/94.2, 89.7/88.6, 61.4/60.0, 76.8/75.4), and the seventh — n6/p0.5 at 75.8/72.1 — is 3.7 points apart. The campaign log's own summary says "within 1.6 points at every configuration"; the table above shows that is true of six of them, and this page reports the exception rather than inheriting the rounder claim. The quantization does not change how well the head predicts — it changes how fast the target model verifies. UD-IQ4_XS is faster at every configuration, and gains more from a wider drafter (+7.5% with no drafter, +19.6% at n4, +14.9% at n10). And the shipped default leaves 11% on the table on UD-IQ4_XS and 15% on Q4_K_M for this workload. The one genuine difference is of degree, not kind: UD-IQ4_XS climbs steadily to n6 while Q4_K_M flattens and dips there (69.8 → 66.4), because its verification step is expensive enough that a wide, low-acceptance tree stops paying sooner.

07

What the same run costs on other cards

Only one row of this section was measured: the RTX 3090. Every other card's number is the same arithmetic from §04 — bandwidth divided by file size, times the efficiency constant for that file's format — applied to the card's published bandwidth. That arithmetic predicted the measured 3090 row correctly, which is the only reason to trust it elsewhere, and it is still a prediction. The pattern that matters more than any individual number: how much memory a card has decides which strategy you can use; how fast that memory is decides how long you wait. A card with enormous bandwidth and too little memory loses to a slower card that fits the model, because the moment one layer lands on the processor the whole calculation changes to the processor's memory speed.

The figures below are for the reference run: about 2,000 tokens of prompt and about 100,000 tokens of thinking and output, with each card using the settings that keep quality highest. All of them exclude speculative decoding, which varies per system but helps a lot: on the reference 3090 it moved this run to roughly 34 minutes from 45. Treat the drafter as moving a card up this chart — furthest on predictable content — rather than as a new baseline.

† the 3-bit rows trade quality for that speed — a different model for benchmark purposes (§08). They also trade window: on 16 GiB the resident 3-bit file caps context near 49,152 tokens by §05's budget arithmetic, so a single run longer than 100k tokens must either checkpoint across windows or take the offload path.

System-memory assumption for every offload and shared-memory row: those t/s figures are derived at dual-channel bandwidth — two sticks: desktop DDR5-5600 ≈ 90 GB/s, the integrated-GPU row's LPDDR5X-9600 ≈ 153.6 GB/s. A single stick runs single-channel at half that bandwidth — halve the t/s and double the hours, and slower DDR5 grades scale down proportionally. If your machine reads slower than this table, check that assumption before blaming the table: Task Manager → Performance → Memory shows your speed and slots used, and §04's formula predicts your number — your GB/s ÷ your file's GB × 0.70 for a K-quant or 0.65 for an IQ-quant. Read the family off the filename: a name containing IQ (UD-IQ4_XS, UD-IQ3_XXS, UD-IQ2_S) is an IQ-quant and takes 0.65; a name containing K without IQ (Q4_K_M, UD-Q3_K_XL, UD-Q2_K_XL, Q6_K) is a K-quant and takes 0.70. Rows where the model is resident on a discrete card are unaffected; their bandwidth is the card's own.

08

Which file to download

On a 24 GB card, download UD-IQ4_XS from unsloth for everyday use, or Q4_K_M from lmstudio-community if you want the highest measured quality and can accept that it leaves only about 0.3 GiB of the card for your desktop. Two files of the same size from different quantizers really can differ in quality, so the choice is not arbitrary — but between these two the measured difference is 0.9% of perplexity, which is smaller than the uncertainty of the measurement itself, while the size difference is 2.1 GiB and buys you either the vision projector or about 50,000 more tokens of window. If you have a 16 GB card, take UD-Q2_K_XL — the memory each file and window needs is now measured rather than estimated, and it is the 2.9-bit file, not the 3-bit one, that fits a 65,536-token window with its drafter still switched on (the requirement table below); if you have 32 GB, Q6_K; and if you have a Blackwell card and can run vLLM, the NVFP4 build is the one that runs natively there. If you have 12 GB, the answer is no — not a smaller quantization of this model, but a smaller model. And the 24 GB answer is the interesting result of this revision: the right file now depends on how big a window you want, and there are three answers rather than one. Up to about 98,000 tokens, stay on UD-IQ4_XS. At about 131,000, UD-Q3_K_XL decodes 20% faster on this page's fill and ties the reference file on accuracy. At the full native 262,144, UD-Q2_K_XL is the faster one and the only one that can still speculate. All three are measured; the ladder that settles the quality half starts at the eight-rung table below, the memory and speed each configuration needs is in the requirement table, and the card-by-card verdicts are in the last part of this section.

Units, because they bite: download pages list decimal GB while VRAM budgets are binary GiB, and the gap is about 7%. The 17.6 GB UD-Q4_K_XL is 16.39 GiB resident, which is why "17.6 GB of weights + 4.25 GiB of cache" still fits inside 24 GiB (§05). Every GiB figure in the table above is the GB figure divided by 1.073741824.

Does FP4 beat INT4? A scored smoke test

A common belief says floating-point number formats always beat integer formats at the same bit width. It did not survive the test here: all three 4-bit files scored the same on one benchmark, and on the other the integer-grid file answered two more questions than either floating-point file. Twenty questions cannot rank healthy files — that is what the rest of this section and §10 are about — but it is enough to say the belief is not a rule you can apply blind.

First, what the two formats actually are. A 4-bit quantization stores each weight as one of only 16 allowed values; the formats differ in where those 16 values sit:

The three quantizations above were graded for correctness on GSM8K and MATH-500, 20 problems each, greedy decoding — always take the most likely next token, so the same prompt gives the same answer every time (§02) — identical prompts, a 4,096-token budget, with answers that ran out of budget mid-thinking counted wrong and reported as truncations. One framing rule before the numbers: at 20 problems this is a smoke test, not a ranking — each question is worth 5 points (§02). Its job is to catch a broken file: a bad conversion or a mangled chat template collapses a score by 20–40 points or more, which even 20 questions detects reliably. It cannot rank healthy files, whose true gaps are a point or three (the arithmetic is in §10). Measured on the reference 3090:

What the numbers say:

• "Floating point always wins" did not survive the test. All three tied on GSM8K; on MATH-500 the integer-grid K-quant answered two more problems than either FP4 file. At 20 samples that edge is weak evidence — but it is the opposite direction from the belief. What actually decides 4-bit quality is how cleverly a format spends its bits — block sizes, scale precision, which layers stay at higher precision — and mature K-quants are very good at that. Both families use fine-grained scaling; the float-versus-integer grid underneath matters far less than people assume.
• The two NVFP4 tiers were indistinguishable at this sample size — which is what noise looks like, not evidence of equivalence. The file structure does predict a tie: the family shares one byte-identical NVFP4 backbone, with tiers differing only in the surrounding layers. The surprise is where VERY-LOW's savings actually landed: on these maths prompts at n-max 10 its draft head drafted noticeably worse (accepted length 3.33 against about 4.8) — a speed cost, not an accuracy one. The penalty is configuration-dependent: re-measured 2026-08-22 at n-max 4 / p-min 0.75 on code generation, both tiers accepted identically (0.83) and VERY-LOW was the faster file (54.6 against 48.2 t/s) — smaller weights, less memory traffic per token.
• Sample-size honesty: each problem is worth 5 points. Every gap in this table is one or two questions wide. Treat it as "no meaningful quality difference, mild edge to the K-quant" — not a ranking. The same prompts were also run through a different serving stack and grader version, and individual cells moved by up to 10 points, which means measurement variance is the same size as the differences being measured. A one-run quantization ranking published without error bars is not evidence, whoever published it.

How the files rank on perplexity

Accuracy at 20 questions cannot separate healthy files, but a per-token measure can, because it feels quantization damage at every scored token instead of collecting one right-or-wrong bit per question. Perplexity is defined in §02; the run below scores 294,912 token positions — 36 chunks × 8,192, forced by the corpus, the tokenizer and -c 8192 — not the "~330k" earlier editions of this page printed, which was an unsourced estimate about 12% high. Measured 2026-08-22 on the reference 3090 with llama-perplexity over the full wikitext-2-raw test set:

Read it with three limits attached. One corpus. Wikitext rewards plain-text prediction while unsloth's Dynamic calibration targets chat-style data, so the ordering among the top files may not carry over perfectly to instruction-following work. The scored-position count is tokenizer-bound. A later run on the same corpus files recorded 297,193 tokens under Qwen's tokenizer and 295,216 under Gemma's, both at 36 chunks — so 294,912 is a count of scored windows (36 × 8,192), not of corpus tokens, and it moves with the tokenizer. The practical consequence is a hard rule: perplexity is never comparable across model families. And the reference file reproduces. A separate 2026-08-23 run re-measured the UD-IQ4_XS figure as a gate before further work and returned 6.5956 against an expected 6.5956 — bit-identical, which is what pins the error bars and the chunk count.

Two robust findings survive all three limits: both NVFP4 tiers trail the K-quants — the same direction the scored table's MATH-500 column hinted, reached by an independent measurement — and none of the unsloth Dynamic variants beat the plain lmstudio Q4_K_M on this corpus, including at identical size. Two cheap metrics agreeing beats one expensive one.

The same corpus, the same 36 chunks and the same error bars also price the cache instead of the weights — the one setting this page used to argue about from mechanism alone. Measured 2026-08-23 on UD-IQ4_XS, every other flag identical to the runs above (-c 8192 -fa on -ngl 99), on a corpus file byte-verified identical to the one the fp16 and q8_0 baselines used:

Three readings, and the third is the one that keeps this honest. Degradation is super-linear in bits — the second halving costs slightly more than the first (0.383 against 0.309), so the cut stops paying rather than scaling. Both are small next to the model quantization: the Q4_K_M-to-UD-IQ4_XS gap on this same corpus is 0.93%, larger than q4_0's entire cache penalty, so anyone who chose the smaller weights file on quality grounds has already accepted a bigger hit than moving to a 4-bit cache would add. And the error bars overlap: ±0.045 on each of two independent estimates gives ±0.063 on their difference, so the fp16-to-q4_0 gap of 0.0457 is about 0.7 of one standard error and a single 36-chunk run does not resolve it on its own. (That combined figure, not the ±0.045 on a single estimate, is the right yardstick for every file-against-file comparison on this page, including the 0.061 gap between the two 4-bit files above.) the direction is consistent with the q8_0 point and with theory, and that is all it is. Two limits on how far to carry this. It was measured at -c 8192, so it says nothing about the long-context retrieval failure that is the actual argument against a 4-bit cache at 200k-plus (§05's mechanism footnote — still unmeasured). And q4_0 was not faster in any way that matters: 266 s against 298 s of perplexity wall-clock is a bandwidth effect on a prefill-shaped workload, not a decode result. Nothing here argues for changing the default; it argues for stating the price instead of the prohibition.

The quantization ladder: eight files of one model, four instruments

This is the table the rest of this section was missing. One model, squeezed to eight sizes from 13.3 GiB down to 5.8 GiB, every file measured the same way on the same day. Three instruments read the table below — perplexity, task accuracy and the empty-answer count — and a fourth one, added 2026-08-25, runs the code each file wrote (further down). It answers the question people actually ask — how small can I go before it stops being the same model? — and it answers it in more than one voice, because the instruments disagree about where the damage starts and each is right about something the others cannot see. Where two of them agree turns out to matter as much as where they differ.

Read the columns like this. Perplexity is a per-token measure of how surprised the model is by ordinary text: it feels a little damage at every rung and is the earliest warning, but a file can lose perplexity and still do your work. Accuracy Mean is the composite of three graded benchmarks — GSM8K, HumanEval and MBPP, 25 questions each, 75 items in total — and it is the only column in the reader's own units. The empty-answer column counts the times the model returned nothing at all, and it is kept separate from truncations because the two are different failures: a truncation hit the token cap, an empty often terminated normally and simply emitted zero characters, which no truncation counter can see. Those last two columns are the functional detectors that §01 counts as the third of this campaign's instruments — the automated checks for what a score cannot see (§02). The two vocabularies name one thing.

Conditions, and they differ between the two scored columns. Perplexity meas: frozen wikitext-2-raw test set, 36 × 8,192 = 294,912 scored positions per file — the same count and corpus as the table above it — -ngl 99 -c 8192 -fa on --load-mode mmap, fp16 KV, llama.cpp build 10502. Accuracy and the two failure columns meas: a frozen suite — fixed once and never edited again, so every file is asked the identical questions — 1cdf54f8eb9d3f8f, GSM8K + HumanEval + MBPP at n=25 each, greedy, seed 42, --max-tokens 16384, -c 32768, q8_0 KV, reasoning_effort=low, and no drafter — which turns out to matter, and is dealt with in the next part. "Silent" empties are the ones that did not hit the cap: they terminated normally and returned zero characters, so no truncation counter ever saw them. Means are printed at the harness's own precision; what they can be read to is the subject of the next two paragraphs.

So what can be said, arm against arm, is whatever a paired McNemar test says. It compares the two files question by question on the same 75 items and counts only the ones they disagree about — b is the number the left file got right and the right file got wrong, c the reverse, and p is how often a gap this large would turn up by chance if the two files were really equal (§02). The exact test, counting a difference in either direction:

The one sentence that reconciles a falling perplexity curve with a flat accuracy curve. Look at the c column: it is zero in six of the nine rows. The smaller file almost never wins a question the larger one lost, and in every comparison against the 4-bit reference file the losses run in one direction only. So quality is falling at every step down the ladder; it is simply below what 25 questions per set can resolve until the cliff arrives. The flat accuracy column is not evidence that the files are equal. It is evidence that this instrument cannot see a gap this small, which is what §10 said before the ladder ran.

Where the boundary is, stated as an interval because that is all the data supports. UD-IQ2_S at 2.481 bits per weight ties the reference file. UD-IQ2_XXS at 2.153 does not. The boundary therefore lies between 2.48 and 2.15 bits per weight, and 25 questions per set cannot place it more finely than that. Anyone quoting a single number for where this model breaks — including a future edition of this page — is quoting something the measurement does not contain.

Two limits to carry out of this table. Every step is resolved on perplexity, and the smallest one is not close: the 4.223-to-3.895 step moves perplexity by 0.1735, about 2.8× the ±0.063 combined error bar this section uses for any file-against-file difference derived, and every step below it is larger. And a cross-model reference point, because "how small before I should just use a smaller model?" is the real question underneath. On the identical frozen suite, gemma-4-12B-it-QAT-Q4_0 — 6.497 GiB at 4.65 bits per weight, within 4% of UD-IQ2_XXS's download size — scored 73.30 with 19 truncations, against the 2.153-bit 27B's 78.70 with one. At an equal weight budget the crushed big model won, which is worth knowing before assuming that a smaller model at higher precision is automatically the safer trade. Conditions differ between those two arms (the Qwen ran at reasoning_effort=low with a q8_0 cache; gemma ran at its own defaults, having no effort knob), and that is a disclosed asymmetry, not a fair fight.

The empty-answer table, in full

The ladder above prints the empty count in one narrow column. It has earned a table of its own, because it is the cheapest instrument in this whole campaign and the only one of the three that draws a usable line. Cheapest, literally: it costs no GPU time at all — every number below was counted from transcripts that were already sitting on disk when the accuracy run finished. And it draws a line where the other two cannot: perplexity falls at every step from the very top, so it never has a flat stretch that could mark a beginning, while the paired accuracy test still says "tie" a full rung below the point where empty answers first appear.

Three words in plain language before the numbers, because they name three different things. An empty answer is a reply with no characters in it — you asked, and nothing came back. At cap means that particular empty reply had run out of its token budget and was cut off; the server records that as a truncation, so it appears in a log where somebody might see it. Silent means the opposite, and it is the one to worry about: the model ended the reply by itself, in the ordinary way, and handed back nothing — an answer that finished normally and returned zero characters, so no truncation counter can see it and nothing in your logs reports a problem at all. That is why the two are separate columns rather than one. Every row is n=75: three benchmark sets — GSM8K, HumanEval and MBPP — at 25 questions each, the identical 75 questions put to every file.

Conditions meas: the frozen suite 1cdf54f8eb9d3f8f, greedy at temperature 0 / seed 42, --max-tokens 16384, -c 32768, q8_0 KV, reasoning_effort=low, no drafter — the same eight arms that produced the accuracy column above, read back from their saved artifacts rather than from a counter. How this table relates to the ladder's truncation column. "At cap" counts only the empties that were cut off; the ladder's Trunc column counts every answer that was cut off, empty or not. At the bottom rung the two read 18 and 20 — so 18 of that file's 20 truncated answers came back with nothing in them.

The finding, and it is a clean one. The empty-answer rate is exactly zero at every rung down to and including 2.912 bits per weight — the same rung where the perplexity curve turns and the last one that ties the reference file on the paired test — and then it rises without going back down: 2, 3, 5, 28. It notices damage at 2.481 bits, a full rung above where the paired accuracy test can resolve anything at all. At 2.481 the paired test returns p=0.0625 and says "tie"; the first verdict of "different" does not arrive until 2.153. The empty column had already stopped reading zero one rung earlier, for no GPU time.

How to read it, and how not to. The useful reading is zero against not-zero, and the shape of the rise. The difference between 2 empties and 3 out of 75 is not a ranking and must not be used as one — the same sample-size limit that governs the accuracy column governs this one. What the column does carry is a threshold, and it is the reason this page recommends nothing below 2.912 bits per weight even though the paired test would still license 2.481.

The median-answer column is a second detector hiding inside the first, and it finds a different failure. Median answer length holds inside a band of 416–502 tokens all the way down to 1.994 bits per weight — seven of the eight rungs barely move how long an answer is — and then jumps to 932 at 1.835. That is the rung that stops terminating: it is not answering worse at the same length, it is failing to stop, which is why 20 of its 75 answers ran into the cap and why that one arm took two and a half hours where the next longest took thirty-six minutes. A quality curve would never have predicted it, and neither would the empty count on its own.

A fourth instrument: we ran the code instead of reading it

Every test above scores text. Perplexity scores it per token, the accuracy suite scores whether the answer is right, and the repetition screens check whether it has degenerated into a loop. None of them ever executed anything. So we did meas: each file had been asked to finish a JavaScript Dijkstra implementation, and every one of those programs was rebuilt with the prompt's own opening lines and run under node v24.15.0.

Six of the ten programs run. Four do not. The line falls in a place worth knowing:

• Down to 2.481 bits per weight the code runs — UD-IQ4_XS, UD-Q3_K_XL, UD-IQ3_XXS, UD-Q2_K_XL and UD-IQ2_S each build the graph, run the search and print a shortest path with its cost.
• Below that, none of it runs. UD-IQ2_XXS throws TypeError: Cannot read properties of undefined. UD-IQ1_M and UD-IQ1_S do not even parse.

This does not change what this page recommends, and that is the point of reporting it. The recommendation floor here is 2.912 bits per weight, set by the empty-answer column. The executed floor is 2.481 — one rung below the recommendation. A fourth instrument, aimed at something none of the others measure, landed inside the margin the other three had already left. UD-Q2_K_XL writes code that works, which is the most direct answer this page can give to whether a 2-bit file is a real daily driver.

The failures arrive together, which is the useful part. The prompt said to continue the file and to emit no markdown fences. Every file that ran followed both instructions. Every file that failed broke them — restarting the program from scratch, or wrapping it in the fences it had been told not to use. At the bottom of the ladder, following instructions and producing working output stop being separate skills. That is a better early warning than any single score, and you can watch for it yourself without a benchmark: a model that starts ignoring the shape you asked for is telling you something about the code it is about to hand you. n=1 per file — one task, one language — so read this as a threshold, not a rate.

And a caution about a rule of thumb you will see everywhere. The usual advice is that a smaller model at more bits beats a bigger model at fewer bits. On this one task it went the other way: gemma-4-12B-QAT-Q4_0 at 4.651 bits per weight produced code that does not run — it opened a helper function inside a method, never closed the method, and declared the next one as though it had — while Qwen3.8-27B at 2.912 bits produced a working program. One sample is not a refutation and this page does not offer it as one. It is a reason to run the substitution on your own workload rather than accept either rule of thumb: the comparison is cheap, and it does not always land where the slogan says.

What each file and window actually needs

Every recommendation on this page ends in the same small piece of arithmetic, so here is the table that lets you do it yourself instead of trusting a verdict. Find your file and the window you want, read what it needs, and subtract your own desktop from your card's total. What is left over is your slack (§02).

The memory column transfers to any card. The speed column is this 3090's and nobody else's. How much memory a configuration needs is a property of the model, the window and the flags — the same file at the same -c asks for the same bytes on a 5080 as it does here — so a 16 GB owner can plan against these figures directly, and that is the one thing about a smaller card this page can state as measured. How fast it then decodes is a property of this card's memory bandwidth. No speed on this page was ever measured on any other card, and none can be.

Conditions, all cells meas (2026-08-25, the reference RTX 3090, board total 24,576 MiB): -ngl 99 -fa on --parallel 1 -ctk q8_0 -ctv q8_0 --jinja, reasoning off, and text-only — there is no --mmproj among the recorded flags, which matters because the projector is the fourth determinant of any ceiling on this page (§05); where the drafter is on it is --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75, the flags §03's vision recipe ships. Every window was deep-filled to about 90% of itself with ordinary English prose from the wikitext-2-raw test set — not probed shallowly, which is the mistake §05 documents — the first probe after the prefill was discarded because the card is still raising its clock (§11), and the figure printed is from two settled probes. Memory is board VRAM, so it contains this machine's desktop as well as the server. The three arms at the full -c 262144 are a separate deep fill and are in the table two parts below.

What the drafter costs, and a cross-check worth having. Turning speculation on costs a consistent 1,208–1,660 MiB across these arms, and the cost grows with the window rather than with the file — which is exactly what §05's two-constant model predicts, since the drafter books 1,008 MiB fixed plus 5,120 B per window token. Subtracting the pairs above gives 1,208–1,210 MiB at -c 32768, 1,284–1,338 at 65,536 and 1,660 at 131,072 derived, against 1,168, 1,328 and 1,648 predicted derived — agreement within 44 MiB, well inside that model's own 127 MiB worst residual. That model is fitted at ordinary windows and it is not reliable at the top of the native window: at -c 262144 it under-predicts the measured requirement by about 1,213 MiB, because it does not carry the compute buffers (below).

Which file is fastest changes with the window — and with what you are writing

The speed column of the table above holds a result that is worth separating out, because it contradicts the way almost everyone reasons about quantization. With the drafter on, a different file reads fastest at different windows. Not a different file per card, or per workload — per window, on the same card, on the same content, with the same flags. Here is that column on its own, with a fourth window added, all arms drafter-on and deep-filled with prose:

Read down the columns and the highest reading keeps moving: UD-Q2_K_XL at 32,768, UD-Q3_K_XL at 65,536, UD-IQ4_XS at 98,304, UD-Q3_K_XL again at 131,072 — three different files across four windows. Before that becomes a claim about which file is fastest, here is how much of it the measurement actually supports. At -c 32768, -c 65536 and -c 98304 the three files sit inside a narrow band — 3.6, 2.7 and 3.2 t/s from top to bottom der — and at those windows only UD-Q3_K_XL was loaded twice. Those three columns are bands, not rankings, and this page does not order the files inside them. One column is different, and it is the one the recommendation rests on.

At -c 131072 the separation is wide and it reproduced across independent server loads. Each file was loaded twice, from scratch, with five settled probes per load: UD-Q3_K_XL read 41.44 then 42.04 (standard deviation 0.36), UD-IQ4_XS 34.66 then 34.70 (0.16), UD-Q2_K_XL 29.46 then 29.41 (1.07). Every file landed within 1.4% of its own first load. The gap between the top two is 20% — 41.74 against 34.68 der — which survives a 1.4% reproduction spread comfortably. UD-Q3_K_XL was loaded twice at -c 98304 as well — 32.86 then 32.95 — which matters for the mechanism below. On the two decimal places: they are the harness's own precision, not a claim that a cell is stable to 0.01 t/s. The band any cell can be read to is the reproduction spread printed here — 0.16 to 1.07 t/s — and at the three windows where no second load was taken, the band is unknown and the column is not ranked.

Why a file can decode faster at a deeper window. For a plain decoder this is impossible, and the impossibility is the reason the result was doubted: more context means more cache to read per token, so speed falls. A speculating decoder is not a plain decoder. What you actually measure is the model's own decode rate divided by how many tokens it commits per verification pass, and that second quantity belongs to the drafter — specifically to mean draft length, which §06 already establishes is the thing that ranks a configuration while draft acceptance is not. Watch UD-Q3_K_XL move between two neighbouring windows:

Acceptance falls, draft length rises, and throughput follows the draft length. An acceptance of exactly 1.00000 sitting beside a short draft is not a triumph — it is the confidence floor cutting the draft off early. --spec-draft-p-min 0.75 tells the drafter to stop guessing the moment it is less than 75% sure (§02), so on content it finds hard it proposes almost nothing, and everything it does propose survives checking. You get a perfect score on a nearly empty exam, and you go slowly. This is the same fact §06 measured on the reasoning stream, arriving from a completely different direction.

One more disagreement, and it is the sharpest evidence in this section rather than an embarrassment. The drafter pair below also measures UD-IQ4_XS against UD-Q2_K_XL at -c 32768, and it ranks them the other way round: 86.91 t/s against 77.01, the 4-bit file 12.9% ahead. The table above, at the same -c, reads 41.35 against 43.19 with the 2.9-bit file ahead. Neither number is wrong and neither is the general answer, because two conditions differ between them: that pair generated novel code into a nearly empty window, while every row here is prose filling about 90% of the window. Change the content and change the depth and a speculating decoder reorders itself. If you take one thing from this part, take that — the ordering belongs to the workload, not to the file — and then go and measure your own.

Which file for which card — and, on 24 GB, for which window

The three parts above cover quality, memory and speed at ordinary windows. Two further measurements finish the picture, and both of them changed an answer this page had already published. The first is that the whole ladder was scored with the drafter off — and the drafter is on in every recipe this page ships. The second is what happens when you actually fill the model's full 262,144-token window instead of arguing about it from arithmetic.

“Shipped” is two settings, not one. §03's launcher carries both: n-max 4 / p-min 0.75, the conservative pair, on four of its picks, and n-max 10 / p-min 0.5, the faster pair, on one. The table below was measured at the faster pair. Earlier editions called each of them “the shipped” setting in different places SUPERSEDED, which is ambiguous and is now stated once, here. The ranking inverts when you turn the drafter on. With no drafter the smaller file is faster, which is the obvious result and, for anyone following this page's recipes, the wrong one. Measured on 700-token novel-code generations at -c 32768, q8_0 KV, thinking off, temperature 0, three settled probes with the first post-prefill probe discarded:

Drafter off, the 2.9-bit file is 7.8% faster, exactly as fewer gigabytes per token predicts. Drafter on — which is how every recipe in §03 runs — the 4-bit file is 12.9% faster despite being 45% larger. The mechanism is the draft head: it is quantized along with everything else, so it degrades with bit width. Acceptance falls 0.611 → 0.551 and mean draft length falls 5.70 → 5.08, a 10.9% drop against an 11.4% drop in throughput — draft length predicting throughput again, to within half a point, exactly as §06's measurement says it does. What the bigger file keeps in speculation — 2.05× against 1.69× — is worth more than everything the smaller file wins by being smaller.

Replicated 2026-08-25, after a reader challenged it. meas The argument against this table is a good one: UD-Q2_K_XL reads 31 % fewer bytes per token, so bandwidth alone says it must win. It was therefore re-measured from scratch by a second implementation in a different language, against the original script’s conditions taken verbatim — same prompt, same flags, same probe protocol. All four arms landed inside 1.3 % (86.91 → 86.58 and 77.01 → 76.97), and both acceptance figures reproduced exactly to three decimals (0.611 and 0.551). The bandwidth argument is right about bytes and still loses: what speculation returns to the less-damaged file is worth more than what the smaller file saves by being small.

And a condition this page was under-stating. The first two replication attempts read 73.71 t/s for the 4-bit file instead of 86.9. An earlier version of this paragraph blamed -fa on together with --reasoning off SUPERSEDED — that was wrong, and measuring it (§05) showed why: -fa defaults to auto, auto resolves to on, and those runs already had Flash Attention. The whole gap is --reasoning off, which changes what the model emits and therefore what the drafter has to guess — acceptance moved 0.523 → 0.611 with it. Every speed figure on this page is measured with reasoning off; benchmark your own card with reasoning on and you will read roughly 15 % lower, and the gap will look like hardware when it is a regime difference.

Both files were probed at -c 32768, so this pair ranks them at shallow context — which is the range the recommendation table's first row covers, and no further. That row's boundary is now about 98,000 tokens, and it comes from the depth series above rather than from this pair: at -c 131072 a third file overtakes both of these. Do not confuse it with UD-IQ4_XS's 180,224-token text-only ceiling (below), which is a statement about how much window that file can hold with a drafter aboard, not about which file is fastest inside it.

The general lesson is worth more than this pair: sweep at the recipe you ship, not at a clean-room default. The ladder was run drafter-off for clean determinism and, for the configuration anyone actually runs, it produced the wrong ordering. Carrying the shipped flags from the first arm would have cost nothing and would have had the right answer in hand from the start.

Then the second measurement, at the other end of the window. All three arms below were deep-filled to 218,233 real tokens — 83% of the full 262,144-token window, not a shallow probe — on the card's 24,576 MiB of board VRAM, with the first post-prefill probe discarded. Each is judged against the desktop reserve defined in §02, and it is worth restating which half of it is measured and which is a choice: the 1,181 MiB worst case of the desktop's own share is measured, the 127 MiB of load-to-load variation is measured, and the 1,796 MiB reserve built by adding them is this page's own derived threshold — the amount of slack it wants a configuration to keep before calling it usable with a screen attached, not an amount the operating system demands.

At the full window the 2.9-bit file beats the 4-bit file on every axis at once: 34% faster at depth, 962 MiB more slack, and it can run speculation at all, which the 4-bit file cannot. That last point is the one that decides it — UD-IQ4_XS reaches 23,821 MiB with the drafter already off, so there is nothing to trade. This measures what earlier editions of this page asserted from arithmetic alone, and the number now says it: full native context on the 4-bit file needs the card free of any graphical session.

Two honest costs ship with that recommendation. Filling 218,233 tokens took 424 seconds at 514.7 t/s of prefill — seven minutes before the first token of the answer. This is a long-document configuration, not an interactive one. And the two-constant VRAM model in §05 under-predicts this window by about 1,213 MiB — it predicts 19,353 and 21,647 MiB against measured 20,567 and 22,859 — because it does not count the compute buffers — the scratch memory the server needs while it is actually working, on top of the weights and the cache. §05 already says its arithmetic is a floor rather than a budget; this measures how big the gap between floor and reality gets at the top of the window, and it is big enough to matter. A reader budgeting from the formula alone would think the drafter-on arm kept 2.9 GiB when it keeps 1.7.

Putting all of it together — the ladder, the empty-answer audit, the requirement table, the depth series, the drafter pair and the full-window trial — gives this. Every "fits" below means fits and still leaves this page's 1,796 MiB desktop reserve free, which is a stricter test than fitting on the card:

The boundary of what this machine can say, and it is sharper than earlier editions drew it. This machine owns one 24 GB card and has never loaded any of these files on anything else, so the 16 and 12 GB rows have to be split into three kinds of statement rather than dismissed as one. Quality is measured and it transfers — a file's perplexity and its benchmark answers do not depend on which card decodes them, so the accuracy tie between UD-Q2_K_XL and the 4-bit reference file is a measurement a 16 GB owner can rely on directly. The VRAM requirement is also measured, and it also transfers, because how many bytes a configuration asks for is a property of the model, the window and the flags rather than of the card: the 13,982 and 16,906 MiB figures above were read off this 3090 and are what those same configurations will ask for on any card (§08). The arithmetic that turns a requirement into a verdict is derived and its steps are printed in the cells: card total, minus the reserve, against the requirement. The speed is not measured, and on a card this machine does not own it never can be — every t/s in this section is this 3090's memory bandwidth. How fast UD-Q2_K_XL runs on a 5080 or a 4080 is not known here at all, and §15 lists it as a gap rather than filling it with an estimate.

UD-IQ4_XS against Q4_K_M — the two files this page recommends

These two files are the whole decision on a 24 GB card, and the honest summary is that they are tied on quality and not tied on anything else. At 13.3 GiB, UD-IQ4_XS is 2.1 GiB smaller than Q4_K_M, matches it on perplexity to within the measurement's own error bar, and — because decode is limited by memory bandwidth — is faster everywhere it was measured: prefill 1365 against 1303 t/s, plain decode 42.97 against 39.99 with no drafter, the same maths benchmark with speculation 73 against 63, the matched code sweep 93.9 against 81.7 — the first and last pairs measured on the same prompt in the same sweep. It also costs about 8% less energy per token. The saved memory becomes context or vision.

The measured ceilings for the smaller file, each row naming all four determinants (file · drafter flags · projector · desktop), and each one lower than earlier editions of this page published, because the re-measurement filled the windows instead of probing them shallowly: with vision and the n-max 4 drafter, 163,840 is fully resident (996 MiB slack); with vision and the n-max 10 drafter the same window keeps only 537 MiB, which is less than a busy desktop holds, so 131,072 is the largest window that still leaves a desktop comfortable (2,426 MiB slack) and 122,880 is what §03 ships. Text-only with the n-max 10 drafter the ceiling is 180,224 (847 MiB slack) — not the ~196k the projector arithmetic implied: 196,608 leaves 415 MiB and 212,992 leaves 280, with decode already sagging 56.6 → 51.9 → 49.2 across those three on a short probe. Do not try to read the budget model out of those three slack figures: the board VRAM total is saturating, so the growth goes somewhere the board VRAM figure cannot show — the server's shared memory use across the same three windows is 478, 638 and 744 MiB, which is the spill beginning before the board VRAM slack runs out. The full native 262,144 fits only text-only and drafter-off, at 23,216 MiB with 1,360 MiB of board VRAM left — and that figure was read at load rather than at depth: filled to 218,233 real tokens on 2026-08-25 the same configuration reaches 23,821 MiB and keeps only 755, which is 1,041 MiB short of this page's 1,796 MiB desktop reserve, so the window now needs the card free of any graphical session by measurement rather than only by arithmetic (§08). Earlier editions said shallow probes "stayed at 65–70 t/s all the way to the full native 262,144 with no collapse anywhere" — that is precisely the probe artefact §05 now documents: fill that same configuration to 91k of real tokens and it delivers 8.0 t/s.

So why does Q4_K_M keep being named a default anywhere? Because there are two defaults with two scopes. Q4_K_M is this page's measurement base and its maximum-quality pick — perplexity leans slightly its way, it is the file most of the 08-21 and 08-22 numbers were taken on, and IQ-format decode support varies more across backends (everything here was measured on CUDA). UD-IQ4_XS with vision at -c 122880 is the reference machine's shipped daily default (§03, menu row 1). Short version: context-hungry or speed-hungry, take UD-IQ4_XS; maximum measured quality at 122k with little on screen, stay on Q4_K_M. Either is defensible, and the caveat on the Q4_K_M recipe's quality argument above applies to this paragraph too.

The limit of everything above: none of it tested a long agentic loop

Every number on this page comes from single-turn prompts of at most 16,384 tokens. You ask once, the model answers once, and the answer is scored. That is not how a coding agent works. An agent calls a tool, reads the result, calls another, and keeps going for dozens of turns — and a model that goes wrong once in fifty turns will fail an agent while scoring perfectly here. So the smaller files on this page are recommended for what was measured, and are untested for agentic work.

09

Reasoning effort: what each level costs and buys

This model has a dial that decides how long it thinks before it answers, and the honest finding is that the dial moves hours, not scores. On a 175-prompt benchmark suite run on this machine — all seven sets now scored, the last two by the blind judge panel added on 2026-08-24 (§09) — the three levels scored 80.2, 80.3 and 79.7 out of 100, a spread of 0.6 points at a sample size where a single benchmark cell is worth about ±16 points, which is a tie — while the wall clock went from 1.0 to 1.5 to 2.7 hours and the average answer grew from 830 to 2,217 tokens. So: use medium when you are waiting at the keyboard, and xhigh when you are not. This page ships xhigh as the default in every recipe whose window can hold it and whose speed makes the wait sane — the 12 GB recipe's window holds it and still ships medium, because at 6–8 t/s an answer takes three hours — for one reason that the benchmark suite cannot measure and the smaller judged tests could: on complex single-file tasks, xhigh was the only level that delivered 100% of the specification with no fatal defect, twice out of two. Buy it as insurance against an incomplete answer, not as a higher score. And not for prose: the two open-ended sets, once judged, are the only place in this whole campaign where the levels separate at all, and they separate against xhigh — it lost to medium on 14 of 25 MT-Bench prompts and won on 3 (§09). For writing, medium.

The strongest evidence: a 175-prompt suite, three levels, a tie

Measured 2026-08-23 on the reference 3090: five scored benchmark sets at 25 questions each, per effort level, greedy decoding, seed 42, no drafter (to keep the scores clean), 525 generations in 8.48 hours of GPU time. Two further sets — ALPACA and MT-Bench — needed a judge, and for a long time this page had none, so it published a five-of-seven index and said so. They are now scored (2026-08-24), by a blind three-seat panel described in §09 below. Both indices are kept: the five-set one because every earlier comparison on this page is against it, and the seven-set one because that is what the benchmark protocol actually specifies. They are comparable only with another run using the same sets.

Conditions: UD-IQ4_XS, llama.cpp build 10502, q8_0 KV, -ngl 99, --parallel 1, temperature 0 / top-k 1 / seed 42, no speculative decoding. This table is a merge of two caps, and it matters which cell came from which. Six cells were re-run at --max-tokens 32768 with -c 65536 — those are the ones that had truncated: MATH-500 at low and at xhigh, and HumanEval and MBPP at medium and at xhigh. The other nine scored cells are the original run at --max-tokens 16384 with -c 32768, carried across unchanged. What licenses carrying them is not an argument but the byte-comparison in §10: 139 of 139 untruncated prompts reproduced identically when the cap and the window were both raised. Decode speed, and which instrument read it: the benchmark harness reports 42.2 / 42.0 / 41.9 t/s as an unweighted mean over requests, while the server's own token-weighted timings read 38.4–40.2 t/s for the same arms. The harness figure is pulled up by short, shallow answers. The two are never averaged together on this page (§15). The three scorer names in that table are defined in §02: execution pass@1 is the share of problems whose first generated program runs and passes its tests, and ROUGE-L F1 measures how much of a reference summary's wording a generated summary reproduces. MeetingBank's ~22 is a property of the metric pairing rather than a failure: the model writes the 60–120-word summary the prompt asks for — measured across all 75 MeetingBank generations at 79–129 words, mean 105 — while the reference is a ~40-word resolution title, so ROUGE-L F1 is structurally capped. It is flat across the three arms and contributes no signal. Note also that mixing a ROUGE score into a mean of percentages is what makes this an index rather than an accuracy.

The two sets a machine cannot score, and what a judge found in them

Five of the seven benchmark sets can be marked by a computer: there is a right answer, or the code runs or it does not. The other two — ALPACA and MT-Bench — ask for open-ended writing, and the only way to score writing is to have something read it. For most of this campaign nothing suitable was available, so this page did the honest thing and published a five-of-seven index while keeping the transcripts. On 2026-08-24 those transcripts were finally read.

The reader was a panel of three independent Claude Opus 5 judge seats, working blind: every answer carried an opaque identifier, the mapping from identifier to effort level was sealed in a file no seat could open, and each seat received the answers in its own shuffled order so that ordering effects could not line up across seats. All three seats rated all 150 answers, on the standard 1-to-10 single-answer rubric, giving 450 ratings with none missing. Ratings convert to the 0–100 scale as (rating − 1) ÷ 9 × 100.

Conditions. MT-Bench's generations are the 16,384-cap ones described above; it never truncated. ALPACA's xhigh answers come from the raised-cap re-run — one of its 25 answers spent its whole budget inside a reasoning block and returned nothing, so the budget rule required a re-run at 32,768. The re-run reproduced it exactly: the same answer consumed all 32,768 tokens and again returned nothing, while the other 24 came back token-for-token identical. That is not a budget shortfall, it is a generation that does not terminate, so the remedy is exhausted and the score is final, carrying one disclosed non-terminating answer that all three seats rated 1. Because the re-run replaced one arm's answers, all three ALPACA arms were re-judged together, so the three numbers come from one session rather than two. Judging protocol, packet builder and scorer: scripts/bench/judge-panel.py; every rating and the sealed key are kept with the run data.

The finding, and it is the only thing in this campaign that separates the effort levels at all. Because the same 25 prompts were put to every level, the arms can be compared prompt-by-prompt rather than only mean-against-mean, which is a far more sensitive test. Doing that (a bootstrap: the same set of per-prompt differences is re-scored 20,000 times with the prompts drawn at random and with replacement, and the range the answer falls in 95% of the time is reported): on MT-Bench, medium beats xhigh by 0.51 rating points, interval +0.21 to +0.80, and the count is lopsided — xhigh wrote the better answer on 3 prompts, the worse on 14, and tied on 8. All five other pairings are ties, ALPACA included.

What the seats caught that no mechanical scorer could. The value of a judge is not only the score; it is the failure modes a right-answer test is blind to. Three kinds turned up, and all three matter more than the point differences above.

Confident invention. The travel-writing prompts drew fluent, well-structured answers containing things that are simply not true — a “Hawaiian $2 bill”, a claim that the first surfers ever to ride a wave did so at Sunset Beach, a misdated battle, a hula centre that does not exist, and mistranslated Hawaiian words. A Chicago-transit prompt produced invented Metra line names, expressways matched to the wrong interstate numbers, and a count of “14 L lines”. A finance prompt turned a 1-for-4 bonus issue on 1,000 shares into 1,500 shares. None of this is detectable by perplexity, by an exact-match scorer, or by running code.

Degeneration well short of any limit. One low MT-Bench answer, asked for a short story, began spelling out numbers and never stopped — “one hundred and one, one hundred and two…” — and ended without a story. It used 1,682 tokens, nowhere near a cap, so nothing in the harness flagged it; all three seats rated it 1. One xhigh ALPACA answer expanded into 100 near-duplicate list entries that collapsed into nonsense, and another repeated the same two words three times each to pad a list to the requested length.

A failure that belongs to the model, not the setting. On one ALPACA prompt demanding a full sentence, low and medium independently returned the same bare noun phrase, and every seat rated both a 3. When two settings fail a prompt the same way, the effort dial is not the variable.

xhigh costs 1.84× medium's wall clock on this suite and produces 1.81× the tokens — the same ratio, because decode speed is flat across the three levels. That is a different multiplier from the one earlier editions of this page printed: on a single long authoring task, xhigh cost about 4× medium's wall clock. Both are measured and neither is wrong; the multiplier is task-shaped, not a constant, and it grows with how much room the task gives the model to keep thinking.

The other evidence: judged deliverables, at n=2

The suite above measures whether the model gets short, self-contained questions right. It does not measure whether it produces a complete working deliverable, and that is where the effort dial visibly earns its keep. Measured 2026-08-22: the same ~1,700-token coding task — build an animated aquarium page — run twice per level at temperature 1.0, with the six resulting pages graded blind against the specification by independent reviewers.

† the first run's 4,096-token cap cut 5 of xhigh's 200 answers mid-thinking; re-run at a 16,384 cap it scored 95.0% with zero truncations, so the budget had cost about one point. Only the xhigh arm needed re-running, because greedy decoding is deterministic and the never-truncating arms are byte-identical under any larger cap — a licence that is no longer an argument but a measurement, 139 of 139 prompts reproduced byte-for-byte (§10). One scope warning that earlier editions of this page did not attach loudly enough: "16,384 proved sufficient" was true on GSM8K. On the seven-benchmark suite above, the same 16,384 cap truncated xhigh eight times in the scored sets, and three prompts exceeded even 32,768. Size your cap against your hardest set, not against GSM8K. The GSM8K percentages in this table also carry the grader caveat from §08 and should be re-graded before they are quoted again.

The findings from the judged runs, stated plainly. The dial changes the answer sharply at one end and hardly at all at the other, and neither is where you would guess. The sharp change is between low and medium, and it moves with task difficulty: low matched the others on short maths, then shipped fatally broken JavaScript on the complex page twice out of two, with one broken run burning more tokens than medium. That claim needs its evidence base attached, and earlier editions of this page did not attach it — it is two runs at temperature 1.0, on one task. The independent re-measurement (2026-08-23, same machine, UD-IQ4_XS) ran the same task once per level and got the opposite defect direction: its low page rendered correctly while its medium page shipped a real initialisation bug — the resize routine ran after every creature array was built, so every mobile creature spawned in the top-left corner, violating the specification's "distribute them across the tank". Three runs across two campaigns, both directions observed. What is solid is that a fatal defect is a live risk at every effort level at n=1 — not that low specifically manufactures them. Above that point the change is small: medium and xhigh overlap on score, and medium's own run-to-run spread beats the between-level gap. Only xhigh delivered 100% of the specification with zero fatal defects on both runs, and it won both blind head-to-head rankings. Its edge is completeness insurance, not average score — and the 175-prompt suite above says the same thing from the other side: on short scored questions, there is no score to win.

Your window sets an effort ceiling

Effort has a second price besides time: thinking tokens live inside the context window (§02), so a window that cannot hold a level's thinking cannot offer that level at all. Measured appetite on one hard task: xhigh's appetite is a distribution, not a number. Earlier editions quoted "73–76k" from this page's two completed runs; pooling those with the re-measurement's two samples of the same task gives 61,500, 73,100, 75,800 and one run that hit a 65,536-token cap and returned nothing usable. Read it as 61,500–75,800 straddling 65,536 — which is the operationally important part: the re-measurement's re-run at a 120,000-token cap then wanted only 61,476 tokens, fewer than the cap the previous sample had blown through. A cap sitting near the middle of that spread looks generous and is a truncation machine. Size caps and windows against the distribution's upper tail, never its middle.

Set it server-side in llama.cpp with --chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}"; graphical front-ends usually expose it as a reasoning-effort setting. llama-server ignores a per-request reasoning_effort field, which is why every recipe in §03 sets it at launch and why the multi-agent setup in §14 needs two servers to route between two levels.

What each level costs in electricity

A local card's tokens are free but its electricity is not quite. Measured with a power log integrated over each generation window (2026-08-23, one run per level, with xhigh measured twice — once at a 64k cap that truncated and once at 120k that completed), an answer to the same task costs 20.6 Wh at low, 36.0 at medium and 104.8 at xhigh. Those are board joules only, so they are not a household electricity figure and must not be treated as one (§15). The number worth noticing is not the total but the rate: energy per token rises with effort, because the thinking stream decodes more slowly than the answer stream and the board draws the same watts either way.

Conditions: RTX 3090, UD-IQ4_XS, -c 131072, no projector, drafter at n-max 4 / p-min 0.75 on, temperature 1.0 / top-p 0.95 / top-k 20, n=1 per level, one HTML-authoring task, mixed regime (thinking and answer tokens both timed and both counted), 1 Hz power sampling with no clock or power-state columns, idle not subtracted in the gross columns. Instrumentation tier: in-band GPU board power (NVML); the power supply, processor and the rest of the machine are excluded and unmeasured. All four gross Wh figures were independently re-integrated from the raw logs on 2026-08-23 and reproduced twice: the original rectangle-rule method returned the originally published 20.55 / 35.96 / 120.21 / 104.9 Wh to within 0.05%, and a trapezoid integration with interpolated edges — which is what the four values printed above are — agrees to within 0.15%. Either way they may be cited without hedging; the net columns subtract the dated 34.1 W loaded idle and are derived. The first-minute clock ramp makes each of these runs look 0.01–0.63% better than its settled rate, which is the right sign and a negligible size, so no ramp correction is applied — but it is a 17–20% artefact on 10-second probes, which is why short probes on this page are read differently (§01).

Three things hide inside that table. First, energy per token rises with effort — 4.26 → 5.18 → 6.13 J — but effort is not the cause. The mechanism is arithmetic: joules per token equals watts divided by tokens per second, and it predicts these four numbers as 4.24 / 5.16 / 6.60 / 6.13 against the measured 4.26 / 5.18 / 6.60 / 6.13, with no residual at all. The thinking stream simply decodes more slowly. The control that proves it: on the 25-question arms, where all three levels decoded at the same drafter-off speed, energy per token was flat to 0.3% (7.921 at xhigh against 7.947 at medium) while answers differed 3.1× in length. Effort changes joules per answer, not joules per token. Second, energy-delay product spans 1.57e7 to 5.52e8 — a factor of 35 across three levels, where energy alone spans 5.8×, because a slow setting is punished twice by that metric. Third, the 120 Wh row returned nothing. That xhigh run hit its 65,536-token cap mid-document: 5.8× low's energy for zero deliverable. The completed run beneath it cost less — 104.84 Wh — because a cap that fits lets the model stop when it is finished. Wall clock is still the bill that matters; truncation is the one that wastes it entirely.

Which level to use for which kind of work

Within your window's ceiling, the trade is time. Pay it knowingly, and drop one level when you are sitting there waiting.

10

How many benchmark questions are enough?

Enough for what? That is the whole question, and it has an arithmetic answer. A 25-question benchmark can tell you a model file is broken and nothing finer. A 25-question benchmark can tell you a setting has not collapsed. Telling two healthy 4-bit files apart, where the real difference is one to three points, needs thousands of questions and days of machine time — which is why nobody ranks healthy files by accuracy scores, and why this page ranks them by perplexity instead. Before you believe any benchmark number, including the ones on this page, ask three things: how many questions, what was the token budget, and how many answers were cut off. The rest of this section is those three questions with numbers on them.

A worked example from this page's own data. A GSM8K score of 85% at 20 questions carries a 95% confidence interval of roughly 64–95% — a thirty-point window around a table where every score sits within ten points of every other. One flipped question moves that score 5 points. Healthy 4-bit files differ by perhaps 1–3 true points; inside that window, such a gap is invisible.

The arithmetic is unforgiving because sampling noise shrinks with the square root of the sample count: halve the gap you want to detect and you need four times the samples. What that costs on this 3090 — paired evaluation on the same questions, temperature 0, about 80% base accuracy, 95% confidence at 80% power, about 1,200 tokens per sample at about 55 t/s with the drafter on:

The energy column is derived at the measured 6.13 J per decode token for a drafter-on run at 55.8 t/s (§09), in-band GPU board power only. Two measured reference points for calibration, both from the 2026-08-23 sweep with the drafter off and real answer lengths rather than a 1,200-token assumption: 25 MATH-500 questions at low cost 126.2 Wh, and 50 questions at medium cost 165.9 Wh. Real benchmark work costs more than the table's assumption because real answers are longer than 1,200 tokens and because a drafter-off run spends about 7.9 J per token instead of 6.1.

The first row is the one job a local accuracy run does well, and it is the only claim §08's scored smoke test makes. The last row is why nobody ranks healthy files by accuracy: the run that could would take days per file and about twenty kilowatt-hours. And one failure mode the table cannot show: a saturated benchmark discriminates nothing at any sample size. On the 2026-08-23 suite, GSM8K returned 100% for all three effort levels. No amount of extra questions from that set would have separated them, because the set had stopped measuring anything about this model. Check for a ceiling before you buy more samples.

The other way a benchmark lies: the token budget

Sample size is not the only silent corrupter. A thinking model spends tokens reasoning before it answers, and a fixed max_tokens cap silently converts its hardest questions into wrong answers. The run does not error; the score just drops. This page has the cleanest possible demonstration of it, because the same measurement was run twice and the two runs disagree about which setting is best.

Same model, same prompts, same grader, same machine, same day. The only thing that changed is how many tokens each answer was allowed. The entire xhigh penalty was the cap, because by the grading rule a truncated answer scores zero, and xhigh truncated eight times against one and two for the other levels. Two individual prompts make it concrete: one MATH-500 problem needed 18,273 tokens and was correct; one HumanEval problem needed 17,025 and was correct. Both scored zero at 16,384. And the failure is worse than a cut-off answer: 11 of the 12 truncated generations came back with an empty answer field (eleven of the twelve are the scored-set truncations counted above; the twelfth is an ALPACA prompt, which was unscored at the time and is now scored — the judge panel rated that empty answer 1 out of 10 on all three seats, and the raised-cap re-run reproduced it exactly — the same answer consumed all 32,768 tokens and again returned nothing, §09), because the runaway happens inside the reasoning block, so the model never emits its end-of-thinking marker and never reaches an answer at all. Three xhigh prompts still exceed 32,768 and are genuine non-terminating loops rather than long solutions; they are reported as truncations rather than hidden.

The tempting fix is the wrong one: "re-run using only the questions that did not truncate." That selects the question set based on one arm's behaviour — dropping precisely the hard items where a quality difference could live — and quietly changes the question from "which setting scores better?" to "which setting scores better on easy questions?". It also breaks comparability with every published number on the same benchmark. The correct fix is to raise the budget, not to shrink the test, and to re-run only the arms and datasets that actually truncated.

11

Why it sometimes runs at half speed

Four things on this machine produce the same complaint — the command that did 40 tokens per second yesterday does 25 today — and the server starts cleanly and reports nothing wrong in all four cases. Two are real slowdowns you can fix: the card quietly ran out of memory and spilled part of the model into system memory, or the command left one layer on the processor. One is not a slowdown at all but a client that never delivered your request. And the fourth does not slow anything down at all: it makes your own measurement read up to 25% low, so the server is fine and the number is wrong. Before any of them, though, comes a class of problem whose fix is never in the server command at all, so it is stated first.

Failure class 0 · the client contract

These are the failures a reader most often blames on the model, and every one of them is fixed in a different file from the server command. They belong together because they share one cause: the model has no awareness of its own limits and cannot budget its own tokens. Your configuration is the only thing enforcing them.

• One token pool. prompt + thinking + output ≤ context. A 100,000-token thinking run inside a 32,768-token window fails by arithmetic, not by bug (§02).
• Cut-offs are client-side. The server generates until it is told to stop. A truncation comes from your client's max_tokens, its request timeout, or Ctrl+C. Size max_tokens so the whole run fits one response, and check that your client's context setting matches the server's -c — those two numbers drifting apart is the classic silent failure (§14 shows the setting per agent).
• A cut generation cannot resume. With OpenAI-compatible interfaces there is no seamless continuation: the thinking is gone and the next request starts over. This is why a cap placed near the middle of the thinking-appetite distribution is a truncation machine rather than a saving (§09).
• Timeouts outlive defaults. A long xhigh run outlasts most default client timeouts; the server log shows one as Client disconnected. Stopping generation.
• A signature that separates client from server in one look. Check the llama-server log for the task. If the server logged no task at all, the request never left your machine and nothing about the model is implicated. If the server logged the task and returned something empty, that is the token-budget trap above, not a transport problem.

Failure 1 · the silent VRAM spill

The NVIDIA driver does not refuse an allocation that exceeds VRAM — by default it quietly overflows into shared system memory across the PCIe bus. On the reference machine, two open browsers held 2–3 GB of VRAM while the 122,880-token configuration needed about 22 GiB (§05), and about 1.9 GB spilled. The server loaded without a single warning and decoded at 20–35 t/s — and stayed there no matter how far the context was shrunk, because the weights stay spilled either way. This is §05's cliff in its most treacherous form: the cliff is invisible to the server. Nothing errors, nothing is logged, everything just runs at half its speed or worse.

The signature lives in Task Manager: dedicated GPU memory pinned at 23.5–24.0 / 24.0, shared GPU memory growing, processor use around 70% from driver paging. The fix is freeing the VRAM. The prevention is making the failure loud: NVIDIA Control Panel → Manage 3D Settings → Program Settings → llama-server.exe → CUDA — Sysmem Fallback Policy → Prefer No Sysmem Fallback. With that set, an over-budget load fails with an out-of-memory error at start-up instead of running slow all day.

Failure 2 · the -ngl off-by-one

llama.cpp counts the output layer as one more "layer" than the model has transformer layers. This model has 64, so -ngl 64 reads as complete — and leaves the output layer, a 5120 × ~151k-vocabulary matrix multiplication that runs for every generated token, on the processor. Measured cost: decode 25.7 against 39.7 t/s, prefill 106 against 290 t/s (server-log prefill on the short test prompt — a different probe from llama-bench's, which reads about 1300 on this card), GPU utilisation 53–67% instead of about 90%, and a stack of processor threads pinned doing the vocabulary multiplication. Independently replicated on a second file: the independent re-measurement (2026-08-23, same machine) measured UD-IQ4_XS at 29.84 t/s with -ngl 64 against 42.31 with -ngl 99 — −29.5% — on a different quantization. The Q4_K_M pair above is a steeper −35.3%; call the penalty 30–35% of your decode speed. The detail that makes it dangerous: the load succeeded and board VRAM read 15,256 against 15,661 MiB, near enough identical. There is no memory signature. The only signature is the speed — which is why §04's bandwidth figure for your file is the number to check against. Every recipe in §03 says -ngl 99 for exactly this reason: any number past the real count means "everything", with no off-by-one to get wrong.

The diagnostic commands that actually work on Windows

One tooling trap cost this page's own debugging session real time: on Windows, nvidia-smi's per-process listing shows [N/A] or "Insufficient Permissions" for memory, and its memory.used counts only dedicated VRAM — it cannot see the spill. The tools that exposed both failure modes:

# totals + who is on the GPU (dedicated only - the spill is NOT visible here):
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

# the spill itself - total shared GPU memory in use (PowerShell):
(Get-Counter '\GPU Process Memory(*)\Shared Usage').CounterSamples |
  Measure-Object CookedValue -Sum   # divide Sum by 1MB; growing = spilling

# per-process dedicated commitment - this is what caught the compositor (dwm)
# holding 3.6 GiB during a browser-UI session:
(Get-Counter '\GPU Process Memory(*)\Dedicated Usage').CounterSamples |
  Where-Object { $_.CookedValue -gt 200MB }

# or simply Task Manager - Performance - GPU: the dedicated/shared split and,
# under Performance - Memory, your RAM speed and "Slots used" (the section 07 channel check)

Failure 3 · PowerShell 5.1 silently drops your screenshot

One more Windows trap, and it is the kind that looks like a model problem. The independent re-measurement (2026-08-23, same machine) posted a 1440p screenshot to the vision endpoint from Windows PowerShell 5.1 and got nothing back — no answer, no error worth reading. Invoke-RestMethod cannot POST the roughly 261 KB body that a 1440p PNG data-URI produces. The identical request from Python succeeded in 3.8 s.

The diagnostic signature is the one from Failure class 0, and it separates client bugs from server bugs in one look: check the llama-server log for the task. If the server logged no task at all, the request never left your machine — nothing about the model, the projector, or --mmproj is implicated, and re-tuning any of them is wasted time. The fix is the transport: use Python's requests, curl, or PowerShell 7 or later for anything carrying a base64 image. Keep PowerShell 5.1 for the counters above, where it is excellent.

Failure 4 · the first probe after a long prefill — a 45% measurement swing

This one does not slow your server down; it makes your stopwatch lie, and it has already cost this page one published finding (§06's retired 15% projector cost). Measured 2026-08-23 while trying to settle that finding: four runs of one identical configuration read 18.27, 18.82, 19.21 and 26.60 t/s — the highest 45.6% above the lowest, with nothing changing but what the card had been doing beforehand. Every one of those was the probe fired immediately after a roughly 105-second prefill. This is the measurement that produces the ±25% band declared once in §01, and the two figures are the same measurement stated two ways: half of a 45.6% end-to-end spread is ±22.8%, rounded to ±25%.

The nvidia-smi samples taken before and after each prefill say why, and it is not heat. The first load of a session starts on a genuinely idle card — 56 °C, 225 MHz, 34 W — and its prefill drives the SM clock to a full 1,455 MHz boost. Every later load starts while the card is still winding down from the previous server's teardown and model load — 63–66 °C, 315–435 MHz, 62–136 W drawn at 1% utilisation — and its prefill only reaches 900–990 MHz. The decode probe that follows inherits whatever clock state that was. With the drafter on the same effect is milder but present: post-prefill probes read 54.3–60.0 against a settled 62.7.

The fix is a protocol — this page's cooled protocol (§02) — and it costs about a minute per data point. Fire the prompt, discard the decode timing that comes back with it, wait 30–45 s, then send several short repeat requests that re-use the cached prefix and report the median of those. Under that protocol repeat probes on this machine agree to 0.4% — which is the only reason a 0.04% difference between two configurations was measurable at all. The same ramp is a power caveat: a prefill fired at a cold board draws unrepresentative watts over an unrepresentative duration, which is why 10-second power probes read 277–287 W against a sustained 306–341 (§03).

Two traps that only affect people taking measurements

Both were found while producing this page, and both are the kind of mistake that yields a plausible number rather than an obvious error.

• "The server is down" is not "the GPU is idle." A settled idle window measured after the last job read a mean of 37.8 W against a median of 31.1, because five short excursions to 121–124 W punched through it — each one the desktop rendering plot images, at 0–14% GPU utilisation, minutes after llama-server had exited. If you measure idle power, measure it with the desktop quiet and discard the first 60 seconds while the board is still cooling. The settled figures on this page (29.9 W with no server, 34.1 W with the model resident) were taken that way; the earlier 33.2 W and 34.6 W readings were not, which is how they came out above the loaded figure.
• Never charge a whole token count against a partly covered power log. Joining request timings to a power log that started mid-run, and crediting each request its full token count, produced a physically impossible 5.32 J per token and 59.8 t/s on a card that cannot do either with the drafter off. The fix is to count only requests whose entire window lies inside the log — 115 of 150 in that case. Any per-arm energy join across a log boundary needs the same filter.

12

Vision: sending screenshots to the model

This model can look at images if you start the server with the optional projector file, and on a 24 GB card that is a real option rather than a compromise: it costs 1,138 MiB of memory and, measured carefully, nothing at all in speed. What it does cost is context. A 1440p screenshot occupies about 3,600 tokens of your window and a 4K one about 8,244, so a feedback loop that keeps ten screenshots around has spent 36,000 tokens before the model writes anything. Use 1440p, set --image-min-tokens 1024 and --image-max-tokens 10580, send each picture as base64 data inside an ordinary request (that is the standard way to put an image into a text request), budget max_tokens generously, and clear old screenshots between iterations. One honest limit up front: everything in this section is a single demonstration each, and the control that would prove the model is actually using the image was never run.

What was demonstrated end to end on the reference 3090, 2026-08-22: Chrome captured a 4K screenshot (3840×2160, a 2.3 MB PNG) of a test page — an animated canvas aquarium — and the model, served with the configuration below, named the visible creatures individually, identified the weakest visual element (kelp drawn as wire-thin lines), and wrote working replacement drawing code with tapered, phase-offset swaying blades. Round trip: about 39 s, of which the 4K image cost 8,244 prompt tokens. n=1

Image cost is resolution, not file size — and it obeys an exact law. The vision tower uses patch 16 with a spatial merge of 2, so one visual token covers a 32 × 32-pixel block: tokens = pixels ÷ 1024 + 2, the +2 being the start and end markers. The independent re-measurement (2026-08-23, same machine) confirmed it to within 2 tokens at two resolutions — 720p measured 922 against 920 predicted, 1440p measured 3,602 against 3,600. The 4K row needs the rounding stated to match: 3840×2160 divided flat gives 8,102, while rounding each dimension up to whole 32-pixel blocks (120 × 68) gives 8,162 against the measured 8,244 — 1.0% high on the block form, 1.8% on the flat one, so quote the block form and expect about a percent of slack at 4K. Earlier editions interpolated the 1440p and 1080p rows and ran 28–30% high; only the 4K row was ever measured. Corrected:

Rounds are recomputed from the corrected costs rather than carried over: a full xhigh cycle wants 80,000–90,000 of the 122,880-token window (§09), leaving 33,000–43,000 for images, and the column is that remaining window divided by the row's cost. The old rows inherited their error into this column — "~7–9" for 1440p was arithmetic on a token count 30% too high.

The two budget flags behave differently, and it is worth knowing which one actually saves you anything. --image-max-tokens is a real cap that bites, as above. --image-min-tokens is only a floor: it lifted a 720p shot from 922 to 1,077 tokens and left the 1440p shot at 3,602, untouched. It raises small images and does nothing to anything above the floor — so --image-min-tokens 1024 costs you nothing on the screenshots you care about, and is not a lever for shrinking them. The default cap is 4,096 tokens, which by the law above is about 4.2 megapixels: that is exactly why a 1440p shot (3,602) passes through untouched at default flags while a 4K one does not, and why this page's vision configuration raises the cap to 10,580 for 4K-class detail. (4K handling upstream still has open quirks, so 1440p remains the recommended resolution — and it is now measured rather than interpolated.)

The serving configuration, and the capture command that feeds it:

llama-server -m Qwen3.8-27B-UD-IQ4_XS.gguf  --alias qwen/qwen3.8-27b
  -c 122880  -ngl 99  --parallel 1  --load-mode none  -ctk q8_0 -ctv q8_0
  --mmproj mmproj-Qwen3.8-27B-BF16.gguf  --image-min-tokens 1024  --image-max-tokens 10580
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75
  # n4/p0.75, matching section 03's recipe [1]: with the projector loaded, the wider
  # n10 drafter's 898 MiB comes out of the slack a desktop needs. Measured live at
  # these flags: ~3 GiB of board VRAM left over with a desktop up. An older live check
  # of
  # this configuration read 71.3 t/s short-context, but its token regime was never
  # recorded, so this page does not use it as a speed band - plan on the 83-86 t/s
  # measured on answer tokens at these same flags (sections 03 and 06);
  # the projector itself books 1,138 MiB of that (section 05)
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0  --jinja  --host 127.0.0.1 --port 1234

# capture the page your code just rendered, using Chrome's no-window mode. The flag
# below is spelled --headless, but it has NOTHING to do with the graphics card or
# with "headless" as section 02 defines it: it only means the browser draws no
# window on screen. It does not free any VRAM.
chrome --headless --disable-gpu --window-size=2560,1440 ^
  --screenshot=shot.png --virtual-time-budget=6000 file:///path/to/page.html

Multi-image works, and it can compare versions. Measured: three screenshots of three different aquarium implementations in one request — 4K, 1440p, 1080p — cost 13,913 prompt tokens with a 74-second round trip, and the model ranked them by visual richness correctly and flagged the third as "broken / nearly empty… looks like the animation failed to spawn its fish". That page is the same low-effort output whose code was blind-scored 40/100 in §09 for freezing on its first frame — two independent reviews reaching the same verdict, though again without a withheld-image control. That 13,913 is also an unplanned third confirmation of the corrected token law: 8,244 + 3,602 + ~2,040 = about 13,890, some twenty tokens from the measurement once the text prompt is counted. Under the old interpolated figures the same three shots should have cost about 15,500 — this page's own multi-image measurement had been contradicting its own table all along.

Which coding agents can carry a screenshot to the model

All five were driven from a script, with no interactive terminal session, against this server with the same 2.3 MB 4K PNG and a question only answerable by seeing it (2026-08-22, one attempt each). The encouraging result: no agent hallucinated sight. Every failure was an honest "I see no image", which is the failure mode you want, because the alternative — an agent that describes a picture it never received — is undetectable from the transcript. Two of the five needed configuration before they passed, and in both cases the default behaviour was to silently send the image as text.

One attempt per agent, one image, one question — so this table says "the plumbing works", not "this agent is good at vision". The image-attachment matrix at scale, across resolutions and multi-image requests, was never run.

13

When 27B is enough, and when it is not

A well-tuned local 27B is genuinely good at bounded work: answering factual questions, writing and explaining ordinary code, summarizing, and completing a self-contained task you can check when it finishes. It gets less reliable as a task gets longer, as the knowledge gets rarer, as the number of simultaneous requirements grows, and as the number of unchecked steps between you and the result increases. The decline is gradual rather than sudden, and much larger models are further along the same decline rather than free of it. The practical rule: give this model work you will look at, and escalate work that has to be right without you looking. This section is the evidence for that rule — and unlike the rest of this page, most of it is argued from published literature rather than measured here.

What this page did measure about capability: a five-set, 175-prompt benchmark suite at three effort levels, composite index 82.1 / 80.5 / 81.3 with GSM8K saturated at 100 and code execution pass rates of 92–100% on HumanEval and 84–92% on MBPP (§09). That is a real capability measurement on this file and this machine, and it is not comparable to the vendor's published figures for this model: different precision (4-bit against full), different harness, different sampling, different prompt formatting. Nor does it speak to the five territories below, all of which are about behaviour over long horizons that a 25-question set cannot reach. One agentic pipeline was also built and validated end to end on this machine — 22 model calls, 9 minutes 36 seconds of wall clock on one sandboxed repair task against real code — and it is worth reporting its outcome rather than only its existence: 0 of 96 target tests turned green, while all 561 previously-passing tests kept passing. The harness worked; the fix did not. That is one sample, and it proved the plumbing and the cost rather than the capability; the 10-task, three-effort sweep that would have measured capability was cut against a four-hour time gate on an eight-and-a-half-hour projection.

A model does not need trillions of parameters to act as an encyclopedia. High-frequency knowledge — geography, mainstream history, standard biology, common coding syntax — compresses comfortably into 27B (a model stores on the order of 2 bits of factual knowledge per parameter), which is why local factual question-answering feels frontier-grade. What frontier models buy with their scale looks like a difference in kind but is built from differences in degree: per-step reliability improves smoothly with capability, and because a long task succeeds only if every step does, those smooth gains compound into cliff-like gaps on exactly the work below — as reasoning engines, software architects, and autonomous agents. The boundary runs through five territories:

• Long-horizon reasoning. Direct questions are effortless at 27B; what degrades is the 30-step deduction, the formal proof, the architecture analysis — work where one early flaw silently poisons everything downstream: agent success decays exponentially with task length, and transformers solve deep compositional problems by pattern-matching that collapses as the reasoning graph grows — a collapse that reaches even frontier models at sufficient depth, so the frontier extends the horizon rather than escaping the regime. This page's judged runs located the local edge precisely: at xhigh, the 27B delivered 100% of a complex single-file specification, twice. Multi-file systems, where the flaws hide in the seams between files, are where the footing gives — and note from §09 that no read-then-edit task against existing code was ever run here, so that boundary is argued rather than measured.
• The long tail of knowledge. 27B keeps what training data repeated often — fact recall tracks how many times a fact appeared in pre-training — and frontier models retain far more of the rare: the niche legal precedent, the legacy framework, the unusual drug interaction (the studies measure trivia-style entity facts; these are the practical analogue). But the tail deficit shrinks rather than disappears — scaling fails to appreciably improve memorization of tail facts, and closing the gap by scale alone would take many orders of magnitude more model. The danger is the failure mode: on obscure ground no model reliably says "I don't know" — training and benchmarks reward confident guessing over abstention — and a small model simply has more gaps for the fluent guess to fill. Treat confident answers about rare domains as unverified drafts, or ground them with retrieval: retrieval-augmented small models beat plain models orders of magnitude larger on exactly these facts.
• Ultra-long recall. A whole codebase or a 500-page audit in one prompt is two problems wearing one coat: holding the text, which this card's 122k window does (§05), and keeping every cross-document dependency straight, which improves with model scale — in RULER's within-family test, a 34B degrades far less than a 6B trained on the same context — but scale mitigates rather than cures: only about half of models claiming 32K or more actually handle 32K, information buried mid-context is used far worse than at the edges, and once retrieval requires inference instead of literal matching, most long-context models fall below half their short-context accuracy by 32K.
• Multi-constraint instruction following. "Five custom interface rules, strict typing, an exact schema, and ten edge-case tests — simultaneously." Rule stacks like that degrade small models first, and quietly: the dominant failure at high instruction density is silent omission — the output stays a coherent, compliant-looking document — and violations are hard enough to spot that benchmarks need model judges to find them. Every model declines as constraints stack, so fidelity under heavy constraints is one of the clearest capability dividends — frontier reasoning models hold near-perfect through about 150 simultaneous instructions where small models decay exponentially — but it buys later, slower degradation, not immunity: even the best fall to about 68% at 500 instructions.
• Autonomous agentic reliability. An agent that calls tools, reads its own error logs, and self-corrects across dozens of steps multiplies its per-step error rate through every step — measured agent success declines exponentially with task length, like a half-life — and worse than independent errors predict, because models self-condition on their own earlier mistakes, amplifying drift into failure. Frontier agents push the horizon from minutes toward hours, but at 50% success — at their quoted horizon they still fail half the time. This is why the setups in §14 shine for supervised sessions and wobble on multi-hour autonomous loops, and why §09's defect risk matters most in exactly that mode, where nobody is watching the output.

This table is a summary of the literature above plus this page's own judged runs; no cell in it is a measurement of a frontier model on this machine.

14

Coding agent setups

Every recipe in §03 ends with an OpenAI-compatible server, so any OpenAI-compatible agent can connect with the same three facts: the address http://localhost:1234/v1, the model name the server reports (qwen/qwen3.8-27b, set by --alias), and the context and output limits from your -c setting. Two things go wrong often enough to state once for all five agents below. Tell the client the model's real limits — if its context setting and the server's -c drift apart, generations get cut off silently. And override the temperature — several clients send 0 by default, which puts this model into repetition loops; it wants 1.0 with top-p 0.95. The five setups below are ordered by popularity, and each was installed, configured exactly as printed, and run through a live request against the reference server on 2026-08-22.

OpenCode · the most-starred coding agent

# install:  npm i -g opencode-ai   (or the curl installer / brew)
# config: ~/.config/opencode/opencode.json (or opencode.json in the project)
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llamacpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": { "baseURL": "http://localhost:1234/v1", "apiKey": "dummy" },
      "models": {
        "qwen/qwen3.8-27b": {
          "name": "Qwen3.8 27B (local)",
          "attachment": true,
          "modalities": { "input": ["text", "image"], "output": ["text"] },
          "limit": { "context": 122880, "output": 106496 }
        }
      }
    }
  }
}
# run:  opencode --model llamacpp/qwen/qwen3.8-27b   (or pick via /models)
# the attachment/modalities pair is REQUIRED for images: without it OpenCode
# silently reads a referenced PNG as text bytes (section 12's agent table)
#
# THE TWO NUMBERS ABOVE ARE TIED TO YOUR SERVER'S -c. Derive them, do not copy:
#   context = the server's -c value exactly
#   output  = -c minus your longest expected prompt; 106496 = 122880 - 16384.
# For recipe [2] at -c 180224 that is context 180224, output 163840;
# for recipe [3] at -c 262144, context 262144, output 245760.

Two gotchas from the documentation: baseURL must sit inside options, not at the provider root, and the provider identifier must be a custom name ("llamacpp", "local") — reusing a built-in provider name breaks resolution. (llama.cpp example)

aider · the git-native pair-programmer

The most detailed walkthrough of the five — its metadata-and-limits pattern is the template every other client repeats in its own configuration format.

Step 1 · Install aider

# needs Python 3.9+ — any one of these:
python -m pip install aider-install && aider-install   # official installer (recommended)
pip install aider-chat                                 # plain pip
uv tool install --force --python python3.12 aider-chat # uv, isolated environment

Step 2 · Point aider at the server

Aider's openai/ model prefix means "any OpenAI-compatible server" — llama-server and OpenVINO Model Server both qualify. Set two environment variables (the key is required by aider; make it match the server's --api-key if you set one, otherwise any placeholder works):

# Windows (cmd / batch)                              # Linux / macOS
set OPENAI_API_BASE=http://localhost:1234/v1         export OPENAI_API_BASE=http://localhost:1234/v1
set OPENAI_API_KEY=dummy                             export OPENAI_API_KEY=dummy

Step 3 · Tell aider the model's limits

Aider knows nothing about a local model until you describe it. Create .aider.model.metadata.json in the directory you launch aider from, or in your home directory. Match max_input_tokens to the server's -c value — these two numbers drifting apart is the classic silent failure:

{
    "openai/qwen/qwen3.8-27b": {
        "max_input_tokens": 122880,   // = the server's -c, exactly
        "max_tokens": 106496,         // = -c minus your longest prompt (122880 - 16384)
        "input_cost_per_token": 0,
        "output_cost_per_token": 0,
        "supports_vision": true
    }
}

The model name after openai/ must match what the server reports at /v1/models — llama-server sets it with --alias, so every recipe in §03 serves as qwen/qwen3.8-27b.

Step 4 · Fix aider's defaults for a thinking model

Aider sends temperature 0 by default, which the model card advises against for thinking mode and which is a known repetition risk on reasoning models — this page's own audit of ten long greedy transcripts found 10 of 10 clean, so treat it as a risk worth avoiding rather than a certainty. It also needs to know to strip thinking blocks before parsing edits. Create .aider.model.settings.yml next to the metadata file:

- name: openai/qwen/qwen3.8-27b
  use_temperature: 1.0
  reasoning_tag: think
  extra_params:
    top_p: 0.95

Step 5 · Launch

aider --model openai/qwen/qwen3.8-27b --edit-format diff --timeout 3600

• --edit-format diff — the model writes only changed sections instead of whole files. At local speeds this is the biggest quality-of-life setting there is: a 60 KB file edit drops from minutes to seconds. Diff mode also degrades gracefully on cut-offs, because complete sections still apply.
• --timeout 3600 — long thinking runs outlive default client timeouts; the server log shows a timeout as Client disconnected. Stopping generation.
• First-run sanity checks: the model-warning banner should be gone (the metadata file was found), and the banner should read diff edit format. If edits arrive mangled, the fallback is removing --edit-format diff to use whole-file mode.
• Vision: with supports_vision: true and the server launched with --mmproj, /add screenshot.png works and the local model will see it (§12).

Qwen Code CLI · Qwen's own terminal agent

# install (Node.js 20+)
npm install -g @qwen-code/qwen-code
# configure via environment variables — put them in ~/.qwen/.env to persist:
OPENAI_BASE_URL=http://localhost:1234/v1
OPENAI_API_KEY=dummy
OPENAI_MODEL=qwen/qwen3.8-27b
# then just run:
qwen

The key must be present even for a local server ("Missing credentials" otherwise), and set the context window explicitly in its settings if offered — Qwen Code's built-in defaults assume the cloud model's limits, not your -c value. For images, declare the model in settings.json under modelProviders with capabilities: {vision: true}; an environment-variables-only setup silently drops them (§12). (setup notes)

Pi coding agent

# custom providers live in ~/.pi/agent/models.json:
{
  "providers": {
    "llamacpp": {
      "baseUrl": "http://localhost:1234/v1",
      "api": "openai-completions",
      "apiKey": "dummy",
      "compat": { "supportsDeveloperRole": false, "supportsReasoningEffort": false },
      "models": [
        { "id": "qwen/qwen3.8-27b", "name": "Qwen3.8 27B (local)",
          "contextWindow": 122880, "maxTokens": 106496, "reasoning": true,
          "input": ["text", "image"] }
      ]
    }
  }
}
# one-shot from the command line, no interactive session:
#   pi --model llamacpp/qwen/qwen3.8-27b --no-session -p "task"
# contextWindow = the server's -c; maxTokens = -c minus your longest prompt.

The compat flags matter: they tell Pi to send a plain system role and to skip the reasoning_effort request parameter, which is the dialect llama-server expects — effort is a server-side setting here (§09). Pi enforces maxTokens on the client side, so size it like aider's; this is the client whose default cap causes the mid-generation cut-offs described in §11. (documentation)

DeepSeek Harness · plugin-based agent framework (developer preview)

# install globally, then run the web UI (an npx cold cache can hang for minutes):
npm i -g @deepseek-ai/dsh
dsh web
# custom OpenAI-compatible providers live in $DSH_HOME/settings.yaml.
# BOTH blocks are required: registering the provider is not enough — without the
# routing block the harness silently keeps its cloud default and fails with
# MISSING_CREDENTIAL (verified end-to-end on the reference machine, 2026-08-22):
agent-default-model:
  provider: llamacpp
  model: qwen/qwen3.8-27b
llm-pi-ai:
  providers:
    llamacpp:
      api: openai-completions
      baseURL: http://localhost:1234/v1
      apiKeyEnv: LLAMA_API_KEY        # set LLAMA_API_KEY=dummy to match --api-key
      compat:
        supportsDeveloperRole: false  # llama-server wants a plain system role
        maxTokensField: max_tokens
      models:
        - id: qwen/qwen3.8-27b
          input: [text, image]        # vision works when the server runs with --mmproj
          contextWindow: 122880       # match the server's -c value exactly
          maxTokens: 106496           # REQUIRED: dsh enforces this client-side (same
                                      # schema as Pi). Without it the small default cap
                                      # cuts long thinking replies mid-run — the symptom
                                      # is "Output token limit reached ... send continue"
                                      # on any xhigh answer (section 11)

The harness's headline trick is routing: it plans with heavy reasoning, then fans the plan out to sub-agents that execute with light reasoning. Two things to know when backing it with llama-server. First, llama-server ignores a per-request reasoning_effort field (the same dialect issue as Pi above) — effort is fixed server-side with --chat-template-kwargs, so either run one server at xhigh (this page's quality-first default), or run two llama-server instances — one xhigh planner on one port, one low executor on another — and register both as separate models in settings.yaml so the harness can route between them. Second, its parallel sub-agents each hold their own context: with --parallel 1 (every recipe here) requests queue rather than run concurrently, which is the right default on a single 24 GB card for memory reasons — raising --parallel multiplies the KV cache and walks you off §05's cliff. It is now also the right default for latency reasons, which was an open question in earlier editions and is not any more: the matched pair of 2026-08-25 measured two concurrent slots at +22.0% aggregate throughput and −35.4% per slot with the drafter on (§03), so a harness whose sub-agents you are waiting on finishes each one faster with a single slot. For non-thinking "fast mode" runs, Qwen's official instruct sampling is temperature 0.7 · top-p 0.8 · presence penalty 1.5.

15

Sources, instruments, and what was not measured

This section exists so that a reader who distrusts one number can find out exactly what produced it. It has four parts: which instrument read each class of number; where every external fact came from and how strongly it is sourced; a numbered list of the places this page corrected itself; and a register of what it never measured at all, so that nobody mistakes an omission for a result. It ends with one command you can run to check whether your own setup matches this machine.

Which instrument produced which number

External sources, graded

Each bullet is marked [P] primary document, [A] arithmetic on published specifications, or [C] carried from a dated prior fact-check on this page.

• [P] §13 task boundaries — long-tail knowledge: Kandpal et al., ICML 2023, PopQA / Mallen et al., ACL 2023, knowledge-capacity scaling laws, Why Language Models Hallucinate · long-horizon reasoning: Faith and Fate (NeurIPS 2023), emergent-abilities mirage (NeurIPS 2023), METR time horizons, agent half-life (Ord, 2025) · long context: RULER (COLM 2024), Lost in the Middle (TACL 2024), NoLiMa (ICML 2025) · constraint stacking: FollowBench (ACL 2024), IFScale, ComplexBench (NeurIPS 2024) · agentic reliability: METR blog, self-conditioning in long-horizon execution
• [P] Arc Pro B70 — 32 GB, 256-bit, 608 GB/s: Intel datasheet, corroborated by Tom's Hardware and Puget Systems
• [P] RTX 50-series bandwidth (5090 1792, 5080 960, 5060 Ti 448 GB/s) — GamersNexus, TechSpot; the RTX 3090's 936 GB/s is NVIDIA's own GA102 whitepaper figure (384-bit × 19.5 Gbps)
• [P] DGX Spark GB10: 128 GB unified LPDDR5X, 273 GB/s — Tom's Hardware review, LMSYS
• [P] Arc B390-class iGPU (Core Ultra): dual-channel LPDDR5X-9600 mandate, 153.6 GB/s, ≤96 GB shared, shipping in Core Ultra X9 388H laptops — Intel ARK, TechPowerUp, Tom's Hardware, HotHardware, shipping laptop list
• [C] NVFP4 on Blackwell: vLLM-only, about 1.5× BF16, 92–97% accuracy, FP8 KV — Unsloth documentation, NVIDIA forums, Kaitchup. Vendor claims, unmeasured here
• [C] Quantization quality claims (unsloth Dynamic v3, Qwen's AD-IQ3_S, the lmstudio Q6_K comparison) — HF discussion #65, kingy.ai
• [C] The xhigh overthinking anecdote (22,276 thinking tokens, 21 minutes, for one simple drawing) — Simon Willison
• [P] Intel software stack: IPEX-LLM archived January 2026 — the repository itself; Vulkan beats SYCL on Battlemage — llama.cpp #22413, B70 benchmark notes, Phoronix
• [P] Model card, OpenVINO builds, drafting support — Qwen/Qwen3.8-27B, int4-ov, int8-ov, qwen38-mtp recipe and benchmarks, llama.cpp discussion 27164, NVFP4-MTP-GGUF card (whose Blackwell-only claim this page's Ampere measurements contradict)
• [P] DFlash2 block drafting and path selection — inco.ai announcement, llama.cpp PR 27342, drafter GGUFs
• [A] Every non-3090 row of §07: bandwidth ÷ file size × the format constant (0.70 K-quant, 0.65 IQ-quant), using the bandwidth figures above

This page's own measurement runs

• 2026-08-21 — llama.cpp build 5ecbe1ac1 (PR 27342), CUDA 12.8, RTX 3090: the drafting comparison (baseline / built-in head / DFlash2 / NVFP4 on Ampere) and the 18-configuration n-max and p-min sweep, all temperature-0 600-token code generation, plus llama-bench prompt-processing and generation figures. Also the FP4-against-INT4 scored comparison: GSM8K and MATH-500, 20 each, greedy, 4,096-token budget, llama-server build 10502, graded with chinkeong/benchmark
• 2026-08-22 — RTX 3090: the context-ceiling probe (stepped -c with short temperature-0 probes), the reasoning-effort comparison at temperature 1.0 with blind judging, the acceptance demonstration at n-max 10 / p-min 0.5, the wikitext-2 perplexity table over six files, the GSM8K n=200 runs per effort level and per file, the 10-configuration drafting re-sweep, the two half-speed failure modes, and the vision demonstration. Reference recipe: Q4_K_M, q8_0 KV, drafter on, projector loaded
• 2026-08-23, independent re-measurement round (00:19–02:19, ~2 h GPU) — the source of most corrections dated 08-23 in §04–§12: the drafter's VRAM bill, the two-constant VRAM model, the measured window slope, the deep-fill collapse, the answer-against-reasoning token regimes, depth on the shipped recipe, the vision token law, per-effort energy, appetite as a distribution, chat-template forensics, the perplexity position count, the PowerShell image-POST trap, and the bandwidth citation fixes. Same RTX 3090, driver 596.36, Windows 11 Pro 26200, i5-13600KF, 31.8 GiB DDR4-3200 dual-channel; llama.cpp build 10502 (version 0.1.2-dev, commit 0adcc3bb5, Clang 20.1.8, binary dated 2026-08-19); Qwen3.8-27B-UD-IQ4_XS.gguf (14,252,845,984 B) with mmproj-Qwen3.8-27B-BF16.gguf; --load-mode mmap, port 1235, --parallel 1, q8_0 KV, -ngl 99. Run without sight of this page, so where its numbers differ from an 08-21 or 08-22 row, the difference is either a genuine condition change or a correction, and each is labelled as one or the other where it lands
• 2026-08-23, follow-up round (02:48–04:29, ~1.7 h GPU) — same machine and build. Four measurements: (1) the matched drafter-flag sweep — 7 configurations × 2 files × 2 probes, -c 32768, thinking off, temperature 0 / top-k 1, 700 tokens, fresh server per configuration, on a novel 149-token rate-limiter prompt chosen to avoid the acceptance inflation a textbook algorithm causes (§06). (2) the projector-at-depth pair — byte-identical 90,862-token prompts, loads ordered A-B-B-A, post-prefill probe discarded, 45 s cooldown, 5 probes per load with the drafter off and 4 with it on, plus the cooled depth ladder and the thinking-on-against-off isolation that produced the mean-draft-length result. (3) llama-perplexity at -ctk q4_0 -ctv q4_0, flags identical to the campaign's cache series, on a corpus file hash-verified identical to the fp16 and q8_0 baselines'. (4) a no-GPU repetition audit of 10 long greedy transcripts: 10 of 10 clean, the single detector flag a false positive, and the one incomplete file cut by its 65,536-token budget rather than looping
• 2026-08-23, effort benchmark sweep (04:40–13:19, 8.48 h GPU) — the evidence behind §09 and §10. Suite hash 1cdf54f8eb9d3f8f, 175 prompts (7 sets × 25) × 3 effort arms = 525 generations, greedy at temperature 0 / top-k 1 / seed 42, --max-tokens 16384 with a rule-driven re-run of only the truncating datasets at 32,768 and -c 65536, no speculative decoding, q8_0 KV. ALPACA and MT-Bench ran unscored at the time — transcripts kept, and scored on 2026-08-24 by the judge panel below. Includes the determinism check (139 of 139 byte-identical) and the three grader fixes
• 2026-08-24, judge panel (zero GPU) — the evidence behind §09. The kept ALPACA and MT-Bench transcripts from the three effort arms, 150 answers, read blind by three independent Claude Opus 5 seats for 450 ratings with none missing or partial. Opaque salted identifiers; the identifier-to-arm mapping sealed in a key file no seat reads; a separate shuffle seed per seat; the standard 1–10 single-answer rubric; ratings normalized (r−1)/9×100. Arm comparison by 20,000-resample paired bootstrap over the per-prompt differences, seed 42, six comparisons. Builder, scorer and paired test: scripts/bench/judge-panel.py. Instrument limit, stated with the result: the answers are Qwen's and the judge is Claude, so no model grades its own output — but the judge and this page's author are both Claude models, a correlated instrument, which is why three seats run and the seat spread is published
• 2026-08-23, energy joins (zero GPU) — two analyses computed entirely from logs already on disk, using scripts/power/attribute-power.py. The first joined a 500 ms power log (17,715 samples, 10:48–13:19) to the sweep's per-request timings, counting only the 115 of 150 requests whose whole window lay inside the log, and produced the drafter-off reference of 7.884 ± 0.307 J per decode token plus the sustained-power drift anomaly. The second re-integrated the 08-22 effort arms' own 1 Hz logs and reproduced all four published Wh figures to within 0.05%, adding joules per token, tokens per kWh and energy-delay product per level
• 2026-08-23, power matrix (13:21:49–14:05:20, 43.5 min of logged span, 19 arms, 100% sample coverage) — thirteen of those arms carry full per-arm energy tables (the drafter ladder, three quantizations, two cache precisions, both token regimes, three depths); the other six are the two idle arms, the two concurrency arms, and the two power-cap arms that were skipped. The source of every figure in §03 and §03: settled idle with and without a server, the three-point drafter ladder, three quantizations, two cache precisions, both token regimes, three depths, and the concurrency pair. In-band NVML board power at 500 ms, three requests per arm, 700 generated tokens each. The two power-cap arms were skipped for lack of an elevated shell; the runner's own net columns used a 31.0 W idle constant rather than the measured 34.1 W loaded idle, a uniform shift of about 1% that changes no ranking, and the net columns published here are re-derived at 34.1 W
• 2026-08-23 to 08-24, quantization ladder (14:13 → 12:35, ~6 h GPU) — complete. Nine files of this model ranked on perplexity over 294,912 scored token positions each, under one identical condition set, with functional detectors beside every rung. Its gate re-ran the UD-IQ4_XS reference file and reproduced 6.5956 bit-identically twice at 0.000% drift before starting, which is what pins §08's error bars, and a same-size pair resolved a 4.27% quality gap against ±0.045 error bars — so the rig both reproduces and still discriminates. It also recorded that the same corpus tokenizes to 297,193 tokens under Qwen's tokenizer and 295,216 under Gemma's, which is the source of the tokenizer caveat. The full treatment is now in §08: the eight-rung table, the point where the perplexity curve turns, the accuracy ladder, the empty-answer audit and the paired tests. This entry's earlier "not yet on this page" caveat is retired by the two rounds below
• 2026-08-24 into 08-25, accuracy ladder over the same files — eight arms run across the day; most took under thirty minutes each (1,705 s and 1,766 s on two of them) and the bottom rung took 8,965.8 s — two and a half hours, more than four times the next-longest arm, because at 1.835 bits per weight the model stops terminating. The evidence behind §08's Mean, empty and truncation columns and behind every paired McNemar row. Frozen suite 1cdf54f8eb9d3f8f — GSM8K + HumanEval + MBPP, 25 questions each, 75 items per file — greedy at temperature 0 / seed 42, --max-tokens 16384, -c 32768, q8_0 KV, reasoning_effort=low, no speculative decoding, eight files scored on the identical items so that every arm-against-arm claim is paired. Two instrument failures are recorded rather than quietly fixed: a grader crashed on a bare #### with an empty tail, a shape no file above 2.15 bits had ever produced (fixed to grade it wrong; no recorded score could change, because any earlier run reaching that path would have crashed and none did); and an empty-answer count was first published from the truncation counter and was wrong — UD-IQ1_M reported zero truncations while returning nothing to five questions. Both columns are now read from the saved artifacts by work/ladder-repcheck.py, which is why they are separate columns. A cross-model arm on the identical suite — gemma-4-12B-it-QAT-Q4_0 at 6.497 GiB — ran the same day under disclosed condition asymmetry
• 2026-08-25, the four rounds that turned the ladder into a recommendation — (1) the drafter on/off pair on both candidate files: 700-token novel-code generations, -c 32768, q8_0 KV, thinking off, temperature 0, three settled probes per arm with the first post-prefill probe discarded, which found the ranking inverting between drafter-off and the shipped drafter-on recipe. (2) the full-native-window trial: three arms at -c 262144, each loaded, VRAM read as a drafter on/off pair, then deep-filled to 218,233 real tokens — 83% of the window, 424 s of prefill at 514.7 t/s — and probed with the first post-prefill probe discarded. A self-caught instrument error belongs with it: the probe script averaged prompt_n across settled probes, which read 4 because probes 2 and 3 hit the server's prompt cache; the real fill is in the server's own prompt eval time line and the decode figures are at depth as published. (3) the requirement sweep behind §08's memory-and-speed table and §08's depth series: three files — UD-IQ4_XS, UD-Q3_K_XL, UD-Q2_K_XL — × three windows (-c 32768, 65536, 131072) × the drafter on and off, plus a fourth window at -c 98304 drafter-on, each window deep-filled to about 90% of itself with wikitext-2-raw prose, board VRAM read at that depth, first post-prefill probe discarded and two settled probes taken. At -c 131072 every file was loaded twice from scratch with five settled probes per load, and at -c 98304 UD-Q3_K_XL was loaded twice, which is what settled the ordering after it had been published, withdrawn and reinstated (the episode is recorded at §08 rather than only in a log). The condition on that round is stated wherever its numbers appear: the fill is prose, the drafter's confidence gate is content-dependent, and the ordering is therefore published as a measurement on prose rather than as a property of the files. (4) the execute probe (§08), which used no GPU at all: every file's probe-A JavaScript, already on disk, rebuilt with the prompt's own prefix and run under node v24.15.0 at a 15-second timeout. Six of ten programs run; the three files below 2.481 bits per weight do not, and neither does the cross-model gemma arm at 4.651. It moved a published number — the “functional floor” had been read off four detectors that never executed anything — and it is the cheapest round in the whole campaign. The same day's matched --parallel pair closed the concurrency question (§03)
• Methodology version. This page follows the campaign methodology as of 2026-08-25: 26 numbered rules plus four review gates, a two-voice writing law, a recipes-first structure, and a standardized-energy-metrics mandate. Two rules were amended by the work in this revision and the amendments are visible in §08's tables rather than only in a log. Rule 20 now requires empty answers and truncations as separate columns, because the truncation counter is structurally blind to an answer that terminates normally and returns nothing — a blindness that produced a false statement in an earlier draft of that very table. Rule 25 now requires a sweep to be run at the recipe that ships: the whole quantization ladder was measured with the drafter off and gave the wrong file ordering for the configuration every recipe on this page uses. The methodology is pinned here for the same reason the build is pinned to commit 0adcc3bb5: a page that names a rule number without naming the ruleset cannot be checked later

The twenty-nine places this page corrected itself

Each entry is a claim an earlier edition of this page made and this edition retires, with the measurement that forced it. They are kept visible rather than quietly deleted, because a page that hides its corrections teaches its readers to trust the wrong things.

1. "ALPACA and MT-Bench are unscored by design." They were unscored by circumstance — no judge was available — and calling that a design choice dressed a gap up as a decision. A blind three-seat panel scored all 150 kept answers on 2026-08-24, the headline index became a seven-set index, and the pair turned out to be the only instrument in the campaign that separates the effort levels at all: xhigh lost to medium on 14 of 25 MT-Bench prompts. The gap that remains is real and now stated as a gap — the judge and this page's author are both Claude models. §09, §15
2. "The built-in draft head costs no VRAM." It costs 1,008 MiB fixed, plus 5,120 B per window token, plus another 898 MiB at n-max 10 — about 1.8 GiB at a 163,840-token window. §05, §06
3. The vision projector was budgeted at its file size (~0.9 GB). It occupies 1,138 MiB resident, measured four times. §04, §05
4. Large windows were declared safe on the strength of short probes — 196,608 shipped with a claimed ~3 GiB of slack, and 262,144 shipped as verified. Filled properly, 196,608 leaves 415 MiB, the real text-only ceiling with the drafter is 180,224, and 262,144 needs the drafter off. §05, §08
5. "Past the resident ceiling, speed degrades progressively." It collapses, and only when real tokens land there: 3.82× at a 91,000-token fill, from 30.6 to 8.0 t/s. §05
6. The projector was thought to cost about 15% of decode at depth. Paired probes measured 0.04–0.09%: it costs memory and nothing else. §06
7. One universal 0.7 bandwidth-efficiency constant. It is format-specific: 0.70 for K-quants, 0.649 measured for UD-IQ4_XS. §04, §07
8. Perplexity was said to score "~330k token positions". It scores 294,912 — 36 chunks × 8,192 — and that count is tokenizer-bound, so it is a count of scored windows rather than of corpus tokens. §08, §10
9. 1080p and 1440p image costs were interpolated and ran 28–30% high. The measured law is tokens = pixels ÷ 1024 + 2; a 1440p shot costs 3,602, not "~4.7k". §12
10. 81.7 t/s was demoted to "best case, not reproducible on real work". It reproduces at 81.71 on a deliberately novel code prompt; the demotion was an unlabelled token regime. §06
11. The drafter flags were split per file. A matched sweep across both files ranks n-max 10 / p-min 0.5 first on each; acceptance lands within 1.6 points at six of the seven configurations and 3.7 at the seventh, so it is a property of the draft head and not of the quantization. §06
12. Configurations were ranked by draft acceptance. Across configurations acceptance inverts: the 96.5%-acceptance setting is the slowest speculating one. Mean draft length — strictly, accepted tokens per verification pass — is what ranks them. §06
13. "Acceptance never moves with depth." It rises, 0.80 → 0.92 across a 60× span, in two independent series. §06
14. The agent-depth band was published as 34–44 t/s. That was the reasoning stream; the tokens an agent receives run 65–70 t/s at the same depth. §06, §03
15. "The board pulls a flat ~344 W." Sustained draw is a range — 306–341 W drafter-off — drifting 11.7% with its own SM clock at constant throughput, constant temperature and constant memory clock. §03
16. Idle was published as 33.2 W with no server against 30.7–31.1 W loaded — physically backwards, and the reason was a board still cooling. Settled with little on screen: 29.9 W with no server, 34.1 W with the model resident. §03
17. "Never use a q4_0 KV cache." It costs +0.693% perplexity against fp16 — real, super-linear in bits, and smaller than the gap between two respectable 4-bit weight files. A knowing trade, not a prohibition. §06, §08
18. "16,384 proved sufficient for every xhigh thought." True on GSM8K only. On a seven-set suite the same cap truncated xhigh eight times, and three prompts exceeded even 32,768. §09, §10
19. An effort sweep read as "quality falls as effort rises" (81.3 / 80.5 / 77.3). The entire penalty was the token cap; at 32,768 the arms read 82.1 / 80.5 / 81.3, a tie. §09, §10
20. "low ships broken code, 2 of 2 runs" was printed as a flat claim. A second campaign saw the fatal defect at medium instead, and on the 175-prompt suite low was the highest-scoring arm. What is solid is that a fatal defect is a live risk at every level at n=1. §09
21. "Two concurrent requests gain +60.3% aggregate throughput." That arm ran with the drafter off, and this page quarantined it against an older ~+11% drafter-on figure. The matched drafter-on pair ran on 2026-08-25 and the quarantine is lifted: +22.0% aggregate and −35.4% per slot (82.98 → 101.25 t/s aggregate, 85.79 → 55.41 per slot), acceptance unmoved at 0.618 → 0.620. The older figure was much closer to right, for the reason the quarantine named — the drafter has already saved most of the repeated reading of the model that batching would otherwise save. Every recipe still ships --parallel 1, now on evidence rather than on caution. §03, §06
22. "Quality falls about 1% of perplexity per gigabyte down to 2.9 bits, then five times that — so stop at about 9 GiB." The advice stands and the reasoning behind it did not. The per-gigabyte figures were right, but 2.9 bits per weight is not the point after which quality degrades — it is the last rung that still performs like the full-quality file. Accuracy on 75 paired items is flat from 4.22 bits through 2.91 and still a tie at 2.48, and empty answers are exactly zero down to 2.91. §08
23. The whole quantization ladder was swept with the drafter off, and it gave the wrong ordering for the configuration every recipe ships. Drafter off, UD-Q2_K_XL is 7.8% faster than UD-IQ4_XS and the ladder reads "swap your daily file". Drafter on, UD-IQ4_XS is 12.9% faster despite being 45% larger, because the draft head degrades with bit width. Sweep at the recipe you ship. §08
24. Full native context on UD-IQ4_XS was said, from arithmetic alone, to need the card free of any graphical session. It is now measured: deep-filled to 218,233 real tokens the 4-bit file reaches 23,821 MiB with speculation already off, leaving 755 MiB — 1,041 MiB short of the 1,796 MiB this page reserves for a desktop. UD-Q2_K_XL holds the same window with the drafter at 1,717 MiB of slack and decodes 34% faster there. §08, §05
25. "Board watts fell as the drafter got wider — the win compounds." They did not. That claim came from the whole-window mean watts (325.2 → 308.4 → 302.4), which fall because each request also contains a short low-power prefill and setup segment drawing 101–139 W, and that segment is a larger share of a shorter run. Power while actually decoding is flat: 344.6 → 341.7 → 341.0 W, a 1.0% change. The 2.52× energy saving is the 2.50× throughput gain and nothing else — a duller mechanism and the true one, and the same distinction is why joules per token divide by the decode watts, not by the column headed mean load. §03
26. The word "cliff" named two unrelated things. It is this page's term for the collapse that happens when a window overflows the card — a real discontinuity, 3.8× at a 91k fill — and it was also being used for the quality step between low and medium, which is a step in a small sample and not a discontinuity in anything. The second use is gone. §05, §09
27. The short-context speed band was published as 71–84 t/s. Its lower endpoint came from one live check of the vision recipe whose token regime was never recorded — the one thing this page says no speed number may lack. Rebuilt from two measurements that do carry their regime: 83.5 t/s on a novel code prompt at -c 32768 and 86.3 at a 1,458-token fill under the cooled protocol, both answer tokens at the shipped flags. The band is now 83–86 t/s, and the 71.3 figure is kept only as a labelled loose end. §03, §12
28. The drafter-off floor was printed as 39.7–43.8 t/s, and prose speculation was credited with 1.16×. Both were inherited from a summary bullet in the campaign log that this page checked against the log's own table and found wrong. 43.80 t/s is a drafter-ON prose figure, so it cannot be a floor endpoint; the measured drafter-off floor for this file is 41.46–42.97 across five contents and both token regimes. And 43.80 ÷ 41.55 is 1.05×, not 1.16× — the 1.16× belongs to n4/p0.75's 48.35 t/s on the same content, which is what the log's own "best vs floor" column says. §06
29. Three memory budgets converted MiB to GiB by dividing by 1000. The 262,144-token window costs 9,984 MiB (9.75 GiB, not "9.98"), that configuration totals 23,216 MiB (22.7 GiB, not "23.2"), and the worked vision example totals 21,556 MiB (21.05 GiB, not "21.6"). The slack figures those pages quoted were right; the GiB labels were not. §05

What was not measured

Stated plainly so that nobody mistakes an omission for a result. Each entry names the measurement that would close it and roughly what it would cost.

The reproduction check

One command, one number, one pass band. If your machine returns something inside the band, your build and your card behave like the one every measurement on this page came from. If it returns something below the band, the diagnostic table in §11 tells you which of the two half-speed failures you have.

:: 1. start the server with the drafter OFF — this measures the bandwidth floor,
::    which is the one number that does not depend on your content:
llama-server.exe -m Qwen3.8-27B-UD-IQ4_XS.gguf -c 32768 -ngl 99 --parallel 1 ^
  -ctk q8_0 -ctv q8_0 --spec-type none --jinja --host 127.0.0.1 --port 1235

:: 2. send TWO identical requests and discard the first (it inherits the cold-clock
::    state that section 11 measures at up to 25%). Each request: 700 tokens,
::    temperature 0, top-k 1, thinking OFF, a short novel code prompt.
::    ONE LINE - a line break inside the -d body will not survive cmd:
curl -s http://127.0.0.1:1235/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"qwen/qwen3.8-27b\",\"max_tokens\":700,\"temperature\":0,\"top_k\":1,\"chat_template_kwargs\":{\"enable_thinking\":false},\"messages\":[{\"role\":\"user\",\"content\":\"Write a sliding-window rate limiter class in Python.\"}]}"

:: 3. read the SECOND run's decode speed from the server's own log line:
::      eval time = ..... ms / 700 tokens ( ... ms per token, NN.NN tokens per second)
::    Use that number. Do NOT use tokens divided by wall-clock (section 10).