gguf2mlx¶
GGUF → HF-style safetensors that MLX-LM and mlx-vlm can load.
Explicit architecture contracts, no silent precision downgrade, fail-closed when a
layout is not supported.
PyPI package · Source · Issues
What it does¶
GGUF is a distribution format; MLX-LM and mlx-vlm expect a Hugging Face-style
directory. gguf2mlx bridges that for explicitly supported architectures by
reading GGUF metadata and tensors, rebuilding the MLX layout, and failing closed
when a layout is not actually supported — there is no silent "Llama fallback".
Use it when a model ships GGUF but no MLX checkpoint. If the original HF weights
exist, prefer mlx_lm.convert, which avoids inheriting the GGUF's quantization
error.
Default output is FP16/BF16
--quantize re-quantizes from that copy. To keep the bytes already in the
GGUF, use --quant-source preserve — see
Quality gates.
Install¶
pip install gguf2mlx # installs MLX + MLX-LM on Apple Silicon
pipx install gguf2mlx # isolated CLI; or: uv tool install gguf2mlx
# extras: [vlm] / [publish] add Hugging Face Hub, [all] everything, [dev] toolchain
Only numpy, gguf and safetensors are required, so conversion runs on Linux
and CI. MLX is needed only for --quantize.
Quick start¶
# GGUF -> FP16 MLX-LM directory
gguf2mlx --input model.gguf --output ./mlx-model
# GGUF -> 4-bit with bounded memory (no FP16 staging directory)
gguf2mlx --input model-Q4_K_M.gguf --output ./mlx-model-4bit \
--quantize --direct-quant --q-bits 4 --q-group-size 64 --q-mode affine
# Reuse the GGUF blocks instead of quantizing the FP16 copy again
gguf2mlx convert --input model-Q4_K_M.gguf --output ./mlx-model-preserved \
--quant-source preserve
from mlx_lm import load, generate
model, tok = load("./mlx-model")
print(generate(model, tok, prompt="Hello from MLX", max_tokens=32))
--force replaces a non-empty --output, --json prints one machine-readable
result object, and -v/-q control stderr diagnostics.
Trusting the output¶
Every conversion writes conversion_report.json and gates on weight fidelity:
each written tensor is compared position-wise against an independent decode of the
source GGUF, with a per-dtype relative-L2 tolerance. Fused or permuted tensors are
reported as transformed rather than silently counted as verified.
gguf2mlx verify --model-dir ./mlx-model --weights --source model.gguf
gguf2mlx verify --model-dir ./mlx-model --load
Perplexity is recorded but never gated: cross-runtime PPL mixes quantization noise with conversion error, so it cannot prove a converter correct. Nightly, the pinned load matrix converts real GGUFs and proves each one loads and runs.
Limits worth knowing¶
--quantizeis a second quantization. Conversion cannot restore precision already lost in the source GGUF, and tokenizer fidelity depends on GGUF metadata.- Mixed-precision MoE output is experimental and needs a JANG-compatible runtime; use uniform output when interoperability is unverified.
- No universal speed claim versus llama.cpp — measure on your hardware.
Where to go next¶
| If you want to… | Read |
|---|---|
| Check whether your model converts | Supported conversion matrix |
| Understand the pipeline | How conversion works |
| Convert a vision model | VLM conversion |
| Push to the Hugging Face Hub | Publishing |
| Audit a conversion | Quality gates |
| See real-run evidence | Validation logs |
License¶
Apache-2.0. Repository traffic statistics are tracked separately in
barrontang/gguf2mlx-traffic
(GPL-3.0-or-later); see Stats.