Quality gates¶
conversion_report.json is written next to every conversion and is the artifact
to read before trusting an output directory. This page explains what each tier
means, what actually fails a run, and how to re-check an artifact later.
The tiers¶
| Tier | Question it answers | Deterministic | Fails the conversion |
|---|---|---|---|
weight |
Did the bytes on disk survive decode, transform and serialization? | Yes | Yes |
numerical |
Does the assembled model load and produce finite logits? | Yes, needs MLX | Only when run via verify --load |
perplexity |
How much accuracy did the source quantization cost? | No | Never |
preserve |
Were the source GGUF blocks re-expressed without requantizing? | Yes | Yes (preserve mode only) |
The separation exists because a single cross-runtime perplexity number cannot distinguish "the converter is wrong" from "4-bit is lossy". A correct converter should show a PPL delta against FP16; failing a build for that teaches nobody anything. Weight fidelity, by contrast, has an exact expected answer.
Reading the weight gate¶
"fidelity": {
"weight": {
"status": "pass",
"tolerance": 0.0006,
"dtype": "float16",
"tensors_compared": 290,
"tensors_total": 291,
"bit_exact_tensors": 154,
"max_relative_l2": 0.00031,
"worst_tensor": "model.layers.0.mlp.down_proj.weight",
"failures": [],
"failure_count": 0
}
}
toleranceis the dtype's unit roundoff with headroom:float166e-4,bfloat165e-3,float321e-7. Half-precision output legitimately loses a few parts in 10⁴ relative to an fp32 reference; anything beyond that is not rounding.tensors_comparedvstensors_totalis the honest coverage number. Fused (Phi-3 QKV), permuted (Llama rotary) and split (MLAattn_k_b) tensors are counted astransformedinconversion.skippedand are not claimed as verified — they are covered by the architecture fixture tests instead.bit_exact_tensorsis non-zero only when the source dtype and output dtype match and no transform ran.- A tensor recorded as
identitywhose stored shape no longer matches the source is a hard failure (layout_mismatch), because the artifact and its stated plan disagree. - A tensor recorded as
reshaped(Qwen3.5's depthwiseconv1d, which GGUF stores squeezed to 2-D) is still value-compared: the source is reshaped to the stored shape when the element counts match, so a transposed kernel fails the gate instead of being excused as "transformed".
The comparison is against gguf.quants.dequantize applied to the raw GGUF
bytes, not against the converter's own decode helper, so a bug in the converter
cannot hide behind a matching bug in the oracle.
Preserve tier¶
--quant-source preserve re-expresses the Q4_0 / Q4_1 / Q5_0 / Q5_1 / Q4_K /
Q5_K / Q6_K / Q8_0 blocks as MLX affine groups (w = scale * code + bias, fp16
scale and bias, 32-value groups). Where the block layout is expressible it
copies the same integer codes and only rounds the scale and bias to fp16.
The governing rule is one line: a block format maps exactly only when its
scale granularity divides 32. Q4_0/Q4_1/Q5_0/Q5_1/Q8_0 cover exactly
32 values, and Q4_K/Q5_K cover 256 with 32-value sub-blocks, so all of them
qualify. Q2_K/Q3_K/Q6_K store a scale (and for the first two, a min) per
16 values and cannot.
The legacy 32-value types are the easiest exact mappings, not the hardest:
| Type | Stored | Decode | MLX scale |
MLX bias |
Rel. L2 |
|---|---|---|---|---|---|
Q4_0 |
d |
d*(q-8) |
d |
-8*d |
0 |
Q4_1 |
d, m |
d*q+m |
d |
m |
0 |
Q5_0 |
d, qh, qs |
d*(q-16) |
d |
-16*d |
0 |
Q5_1 |
d, m, qh, qs |
d*q+m |
d |
m |
0 |
All four are bit-exact, including on arbitrary code bits (checked against
gguf.quants.dequantize). The one layout detail worth naming: the 5th bit of
Q5_0/Q5_1 lives in qh, and qh is read as a single little-endian
uint32 with value i taking bit i — not as four independent bytes.
tests/test_preserve_legacy.py also proves that the writer never calls
dequantize for these types, by replacing the symbol in both modules that could
reach it with a function that raises and requiring the conversion to succeed.
That is scoped to the writer: the preserve audit decodes on purpose, because
it is the independent oracle, so the test runs with the audit off.
Q5_K is preserved exactly, like Q4_K: it keeps a 6-bit scale and min per 32
values — exactly MLX's group — and its 5-bit code is the low nibble of qs plus
one high bit from the block's qh plane. MLX stores 5-bit groups as five words
per 32 values, so the codes land unmodified. Skipping this width is expensive:
the pinned Qwen3.5-0.8B Q4_K_M is 36 Q5_K tensors, and sending them to FP16
turned a 1.08x artifact into 1.45x.
Q6_K is the exception, and it is the reason this tier has two halves. Q6_K
scales 16 values at a time; MLX's narrowest group is 32, so a group would need
two scales. Only 0.7% of groups in a real model have matching pair scales, so
"copy the codes or give up" meant giving up on 99% of the tensor and writing
it as FP16 — which turns 0.82 bytes/value into 2 and makes a Q6_K model 2.4x
the size of the GGUF it came from. Instead those tensors are re-fitted at the
source width and gated on their own ceiling:
| Tier | What happened | Ceiling |
|---|---|---|
exact |
codes copied; only fp16 scale/bias rounding | 1e-3 |
inexact |
re-fitted onto the 32-wide grid | 2.5e-2 (6-bit), 8e-3 (8-bit) |
The preserve gate passes only when every measured tier passes, so the two
numbers can never cover for each other. --preserve-fallback chooses the
width for the inexact tier; 6bit is the default because it keeps the source
width:
| Fallback | Q6_K artifact | Relative L2 |
|---|---|---|
6bit (default) |
1.067x the source bytes | 1.7-1.8e-2 |
8bit |
1.371x | 4.4-4.9e-3 |
float16 |
2.4x | 0 (lossless, 2.44x bytes) |
Measured on Llama-3.2-1B (bartowski Q4_K_M and Q6_K, 113 quantized tensors). Fitting is per group: the lower squared error wins between the source lattice and a least-squares refit from two starting points, and groups whose pair scales do agree keep their codes untouched.
"preserve": {
"gate": "pass",
"tolerance": 0.001,
"fallback": "6bit",
"tensors_compared": 9,
"aggregate_relative_l2": 0.00774,
"size": {"source_bytes": 1294336, "output_bytes": 1397760, "ratio": 1.08},
"preserved_by_qtype": {"Q4_K": 7, "Q5_K": 1, "Q6_K": 1, "Q8_0": 1},
"preserved_by_bits": {"4": 7, "5": 1, "6": 1, "8": 1},
"group_size": 32,
"tiers": {
"exact": {"status": "pass", "tolerance": 0.001, "tensors_compared": 8,
"max_relative_l2": 0.000241},
"inexact": {"status": "pass", "tolerance": 0.025, "tensors_compared": 1,
"max_relative_l2": 0.0112}
},
"kept_float16": {"counts": {"F32_has_no_affine_reexpression": 3}},
"tensors": [
{"name": "model.layers.0.self_attn.q_proj.weight", "source": "Q4_K",
"bits": 4, "tier": "exact", "group_size": 32, "relative_l2": 0.00021,
"max_abs_error": 0.00012, "bit_exact": false}
]
}
Note that the top-level tolerance bounds the exact tier only; inexact
tensors are bounded by tiers.inexact.tolerance. A tensor is never compared
against a ceiling its format cannot meet.
What preserve refuses, and why it says so¶
Two families cannot be re-expressed, and both are rejected by name rather than falling through to a generic label:
| Source | Reason reported | Cause |
|---|---|---|
Q2_K, Q3_K |
sub_block_scale_is_per_16_values |
same 16-versus-32 wall as Q6_K, but with 2-3 code bits a refit loses real precision instead of bounded precision |
IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_NL, IQ4_XS |
codebook_has_no_affine_form |
w = d * kvalues[code] against a fixed non-uniform table, so there is no scale*code + bias form |
TQ1_0, TQ2_0 |
ternary_not_an_integer_grid |
ternary levels, not an affine integer grid |
MXFP4 |
block_float_has_no_affine_form |
per-block exponent, no shared scale |
Q8_1, Q8_K |
intermediate_type_never_stored_in_a_file |
ggml kernel-internal types |
This is the practical answer for iMatrix. An importance-matrix-guided
quantization is a recipe, not a layout: it selects which tensors get which
IQ type. Those IQ types are what a conversion sees, and none of them is
affine, so an iMatrix model converts but does not preserve. The conversion
still succeeds — the tensors are written as float — and the run prints how many
tensors were affected and which block types caused it, so the size ratio is not
mistaken for a mapping bug. A qtype_name of None for an unknown type is
reported differently (Q9_FAKE_has_no_affine_reexpression), so a genuinely new
format is never disguised as a known rejection.
Per-tensor relative L2 against gguf.quants.dequantize:
| Source | Relative L2 | Why |
|---|---|---|
| Q8_0 | 0 (bit-exact) | 8-bit codes map to 8-bit affine codes with scale = d, bias = -128d |
| Q4_K | 2.0e-4 - 7.7e-4 | fp16 rounding of the per-32 scale and min only |
| Q5_K | 2.1e-4 - 7.0e-4 | same, with the 5th code bit taken from the block's qh plane |
| Q6_K (pair scales agree) | ~2.2e-4 | same, within one 32-value group |
| Q6_K (pair scales disagree, 6-bit) | 1.6-1.8e-2 | not representable on a 32-wide grid; re-fitted |
| Q6_K (pair scales disagree, 8-bit) | 4.4-4.9e-3 | four times finer grid, 1.37x the bytes |
Each range spans Llama-3.2-1B (bartowski, 113 quantized tensors) and the pinned
Qwen3.5-0.8B Q4_K_M (98 Q4_K, 36 Q5_K, 17 Q6_K, 36 Q8_0). The upper ends
are per-tensor maxima, not typicals — Qwen3.5-0.8B's worst exact tensor is
7.69e-4, which is what the 1e-3 ceiling is sized against. The block type, not
the layer, decides which band a tensor lands in: in that same file a Q6_K
ffn_down measures 1.58e-2 while the Q4_K ffn_down one layer over measures
6.82e-4, and every Q8_0, ssm_alpha and ssm_beta tensor round-trips at
exactly 0.
Only block types with no affine re-expression at all (F32 norms, non-matrices,
unaligned rows) are decoded to float, and the report names the reason. The
fail-closed rule extends to architectures: only llama, qwen2, qwen3,
mistral, and the Qwen3.5 family (qwen35 plus the qwen3_5 / qwen3.5 /
qwen36 / qwen38 aliases) are supported, and any other architecture raises an
error naming it rather than silently requantizing.
Qwen3.5 is hybrid: only the quantized matrix projections of a linear-attention
block are preservable. A_log, dt_bias, conv1d and the gated norm have no
affine matrix form, so they stay float and are counted in
kept_float16.counts.F32_has_no_affine_reexpression. Two layout rules make
that output loadable at all, and both are recorded in tensor_layout:
| Source tensor | On-disk MLX form | Recorded as |
|---|---|---|
blk.N.ssm_a |
log(-ssm_a) — GGUF stores the decay rate, MLX wants its log |
transformed |
blk.N.ssm_conv1d.weight |
(channels, kernel) → (channels, kernel, 1) |
reshaped |
blk.N.ssm_norm.weight, *_norm.weight |
unchanged | identity |
The norms are deliberately not de-shifted: llama.cpp applies its +1
centering to every norm except linear_attn.norm.weight, and MLX-LM's
qwen3_5 sanitize only re-applies that offset when it sees an HF-layout
conv1d. Since this converter emits MLX layout, the GGUF norms are already the
values the runtime wants.
Source tensors without an output¶
Not every source tensor becomes an output tensor, and an unexplained gap is a silent correctness hole. The report names each one with a reason:
"source_tensors_without_output": [
{"name": "blk.4.nextn.attn_k.weight", "reason": "mtp_head_not_consumed"},
{"name": "rope_freqs.weight", "reason": "reproduced_from_config"},
{"name": "mystery.weight", "reason": "unmapped"}
]
| Reason | Meaning |
|---|---|
reproduced_from_config |
the tensor's effect is recreated from metadata (llama3 rope table → rope_scaling) |
mtp_head_not_consumed |
a speculative-decoding head (mtp.* / nextn.*) that the text runtime never runs |
unmapped |
no adapter claimed the tensor — the case to investigate |
A non-empty unmapped list is a defect report, not a normal outcome: the
conversion still completes (the tensor list is data, not policy), but the
artifact deserves a verify --load before it is trusted.
Re-checking an artifact¶
gguf2mlx verify --model-dir ./mlx-model --weights --source model.gguf
gguf2mlx verify --model-dir ./mlx-model --load
--weights replays the audit from the report's recorded tensor_layout without
converting again. --load is the numerical tier: it imports mlx_lm, loads the
directory, runs one short forward pass, and requires every logit to be finite.
Both exit non-zero on failure, so either can gate a release or a CI job.
A failure on a shared runner can come from the GPU rather than the weights, and the two such failures need different remedies:
| Driver error | Meaning | What happens |
|---|---|---|
…ErrorTimeout |
the command buffer lost its scheduling slot | retryable: true; the matrix and publish --verify-load retry once, because the same bytes usually pass |
…ErrorHang |
the Metal context is wedged | environment: gpu_hang, not retryable — the remedy is to re-run the job, since an in-process retry cannot recover |
Non-finite logits and a missing runtime are never classified as either, so a real defect still fails on the first attempt instead of hiding behind a second one.
That distinction exists because the nightly went red twice on the gemma2
entries — once with Caused GPU Timeout Error, then with
Caused GPU Hang Error where the in-process retry hung again and cost ten
minutes for nothing. Both times the weight gate had already passed, mlx was
the same 0.32.4 the previous green run used, and gemma2-q4_k_m passed on the
same commit; nothing in the report let a reader tell any of that apart from a
broken conversion.
After mlx_lm.convert quantization, the report is carried into the quantized
directory with artifact.stage = "mlx_lm_quantized": the weight audit describes
the FP16 staging output, and the 4-bit weights are explicitly not covered by
it. Use --load on the quantized directory.
Nightly load matrix¶
tests/load_matrix.json pins real public GGUFs by repository, filename and
SHA-256. With GGUF2MLX_LOAD_MATRIX=1, each entry is downloaded, hash-checked,
converted, weight-gated, loaded with mlx_lm and exercised. The
Load matrix workflow fans the entries out as one job per (architecture,
qtype) on Apple Silicon: a converted FP16 tree is 3-5x its source GGUF and a
single runner has ~14 GB free, so running the whole sweep in one job ends in
ENOSPC that looks exactly like a conversion bug.
Run a single pinned case locally with
GGUF2MLX_LOAD_MATRIX=1 GGUF2MLX_LOAD_MATRIX_ENTRY=llama-q4_k_m pytest
tests/test_load_matrix.py -q.
The matrix currently covers Llama-3.2-1B, Qwen2.5-0.5B, Qwen3-0.6B, Gemma-2-2B, Phi-3-mini and Qwen3.5-0.8B across Q4_0 / Q4_K_M / Q8_0 where published. MoE architectures are excluded from the nightly set because no ≤2B public fixture exists; their evidence remains the pinned real-model runs in validation.
The Qwen3.5 entry is pinned (unsloth/Qwen3.5-0.8B-GGUF, Q4_K_M,
bd258782…) and runs with quant_source: preserve, so it simultaneously
exercises the hybrid layout and the preserve tier.
All twelve entries pass, locally on Apple Silicon (2026-10-26) and in CI
(2026-10-28): conversion, the weight gate, the preserve gate where applicable,
mlx_lm.load and a finite-logit forward pass. For the Qwen3.5 case
specifically: weight gate 115/133 with a maximum relative L2 of 2.85e-04,
preserve gate 170 exact / 17 inexact, a 1.084x artifact.
The CI green took three dispatches rather than one: the first full sweep came
back 11/12 because gemma2-q8_0's runner GPU hung, and the entry then passed in
9m02s on a fresh job. That is the case the timeout-versus-hang split below was
written for.
What the Qwen3.5 case found¶
It was worth pinning. Five defects only a real checkpoint exposed:
| Symptom | Root cause | Fix |
|---|---|---|
Unsupported qwen35 tensor: blk.0.post_attention_norm.weight |
llama.cpp names Qwen3.5's post-attention norm post_attention_norm, not ffn_norm; the mapper only knew the Llama spelling |
dedicated mapping to post_attention_layernorm.weight |
Missing 1 parameters: language_model.lm_head.weight |
build_config forced tie_word_embeddings: false for Qwen3.5 while the file has no output.weight and upstream ties |
tie/untie follows the tensor list, as for Qwen3 |
| artifact 1.446x the source | 36 Q5_K tensors had no affine re-expression and fell back to FP16 |
Q5_K → 5-bit affine with the source codes copied; 1.084x |
| wrong decay / unloadable kernel | A_log was only restored for a tensor named ssm_a (real name blk.N.ssm_a), and the squeezed conv1d kernel was never widened |
blk.N.ssm_a → log(-x), kernel → (channels, kernel, 1), gated norm left unshifted |
tokenizer.ggml.pre='qwen35'; using legacy BPE splitting |
llama.cpp names the hybrid family qwen35 while Qwen2/2.5/3 report qwen2; the value was unlisted, so the tokenizer fell to legacy splitting and kept the default NFC normalizer, which no Qwen tokenizer has |
QWEN_PRE_TOKENIZERS and QWEN_FAMILY_ARCHES route every Qwen ID to the same no-normalizer path |
The last one is the reason the load matrix asserts tokenization rather than just a successful load: the model loaded, produced finite logits, and still would have tokenized text differently from the reference. A warning in the matrix log was the only signal.
What the first full sweep found¶
The first run of all thirteen cases (2026-10-21, one job, main @ 63fe89c)
passed 4 of 11 e2e cases. Everything it caught was invisible to the unit
suite, which is the point of the gate:
| Symptom at load | Root cause | Fix |
|---|---|---|
Received 1 parameters not in model: rope_freqs.weight (llama ×3) |
llama.cpp carries llama3 rope scaling as a precomputed table with no llama.rope.scaling.* metadata, and the mapper passed the tensor through. The config also lost the scaling, so positions would have rotated wrongly even after dropping the tensor |
infer factor / original_max_position_embeddings from the table, verify by regenerating it, emit rope_scaling: llama3, drop the tensor |
Missing 1 parameters: lm_head.weight (qwen3) |
build_config forced tie_word_embeddings=False for qwen3 while Qwen3-0.6B ships no output.weight |
tie/untie follows the tensor list for every architecture |
Unsupported gemma2 tensor: blk.0.post_attention_norm.weight |
the Llama mapper only knows two norms per block; Gemma 2/3 has four (attn_norm, post_attention_norm, ffn_norm, post_ffw_norm) and ffn_norm is the pre-FFN norm |
dedicated Gemma 2/3 mapper plus the -1 de-shift for both post norms |
25165824 requested and 0 written (phi3 Q8_0) |
disk exhaustion: twelve downloads plus their FP16 outputs on one 14 GB runner | one job per entry, and drop the converted tree after each case |
Re-run after the fixes, all 11 entries green on
run 39505897872
(main @ 40ef9f5, 2026-10-21, 11 jobs × ~5 min wall clock, ~11 min total):
| Entry | Weight gate | Tensors compared | Bit-exact | Max relative L2 | Tolerance |
|---|---|---|---|---|---|
| llama-q4_0 | pass | 114/146 | 32 | 2.04e-04 | 6e-04 |
| llama-q4_k_m | pass | 114/146 | 32 | 2.08e-04 | 6e-04 |
| llama-q8_0 | pass | 114/146 | 32 | 2.07e-04 | 6e-04 |
| qwen2-q4_0 | pass | 291/291 | 122 | 2.07e-04 | 6e-04 |
| qwen2-q4_k_m | pass | 291/291 | 121 | 2.18e-04 | 6e-04 |
| qwen2-q8_0 | pass | 291/291 | 121 | 2.15e-04 | 6e-04 |
| qwen3-q8_0 | pass | 310/310 | 113 | 2.08e-04 | 6e-04 |
| gemma2-q4_k_m | pass | 183/288 | 0 | 2.08e-04 | 6e-04 |
| gemma2-q8_0 | pass | 183/288 | 0 | 2.10e-04 | 6e-04 |
| phi3-q4_k_m | pass | 195/195 | 65 | 2.08e-04 | 6e-04 |
| phi3-q8_0 | pass | 195/195 | 65 | 2.07e-04 | 6e-04 |
Every case also loaded with mlx_lm and produced finite logits. The compared
fraction is the honest part of the table: llama excludes the 32 rotary
Q/K projections (row-permuted) and Gemma 2 excludes its 104 de-shifted norms,
because the audit only compares tensors written byte-for-byte. Both transforms
are covered by fixture tests, but they are not value-checked here.
What is still not covered¶
--direct-quantwrites packed codes, scales and biases rather than plain tensors, so it is described bytensor_decisions(schema 1 fields) and the layout fixtures, not by the plain-tensor weight audit.- The numerical tier is not run during conversion: it needs MLX, and conversion deliberately works on Linux without it.
- Preserve mode records the mapping error, not the error the source quantization already baked in: a tensor can be bit-exact and still be lossy relative to the original float model.