How conversion works¶
Pipeline stages, streaming/sharding behaviour, and offline tokenizer reconstruction.
Split out of the top-level README on 2026-10-20; content is unchanged apart from heading levels.
- Read GGUF metadata and detect the architecture
- Validate that the architecture has a supported adapter
- Build
config.jsonusing source metadata and selected dtype - Export tokenizer assets from GGUF metadata
- Decode tensors to the selected floating precision (text conditioning uses FP32 intermediates for BF16 output)
- Remap tensor names into the target Hugging Face / MLX-LM layout
- Optionally run
mlx_lm.convert --quantizeinto a second staged directory - Write the selected output directory atomically
If any required step fails, conversion fails and the staged output is cleaned up.
Offline tokenizer reconstruction¶
Language-model conversion writes tokenizer.json, tokenizer_config.json,
special_tokens_map.json, vocab.json, and, when available, merges.txt and
chat_template.jinja directly from the GGUF header. No Hugging Face model or
tokenizer download is needed for this step.
- BPE (
gpt2,bpe) preserves vocabulary IDs and merge order. Recognizedtokenizer.ggml.preprofiles include GPT-2, Llama-3 and Qwen2; these differ in number splitting and Unicode normalization. - SentencePiece BPE (
llama) reconstructs score-ranked merges rather than incorrectly interpreting the vocabulary as Unigram. - WordPiece (
bert,wordpiece) reverses BERT's GGUF word-start/continuation representation and honorsnormalizer.lowercase,normalizer.strip_accents, CLS/SEP IDs and insertion flags. - Unigram (
t5,unigram,spm,sentencepiece) retains token scores, byte fallback, whitespace-prefix/removal flags and any precompiled character map. Legacy SentencePiece aliases on Gemma retain score-ranked BPE behavior. - Matching embedded
tokenizer.huggingface.jsontakes precedence over rebuilding, including its original normalization, added-token and post-processing rules. Explicit GGUF special-token IDs are preserved; absent roles are inferred only from known token names/types, not arbitrary vocabulary positions.
Each format is built by its own function in src/gguf2mlx/tokenizer.py
(build_bpe, build_unigram, build_wordpiece), not by one shared branch
ladder, and the document is validated before it is written. The separation
matters because the four pipeline blocks — normalizer, pre-tokenizer, decoder,
post-processor — are inferred rather than read from the file: a branch added
for one family used to be able to change what a later family received, and the
order of the branches became part of the contract by accident. The rebuild is
output-identical to the previous implementation across 13,824 combinations of
model, architecture, tokenizer.ggml.pre, byte fallback, space prefix,
whitespace flags and precompiled charsmap.
Token strings, scores and token types are header metadata arrays, not the
embedding tensor: weights alone cannot recover a vocabulary. Missing vocabulary,
invalid IDs, duplicate entries, invalid merges, or missing/nonfinite scores for
score-based models fail rather than creating a misleading minimal tokenizer, and
each failure names the GGUF key that is wrong (tokenizer.ggml.scores: requires
a finite score for every token). The emitted document is then checked for the
defects that would otherwise surface at load time or not at all: non-contiguous
token IDs, merges naming a piece outside the vocabulary, duplicate added-token
IDs or contents, and pipeline blocks without a type.
Unknown BPE pre-tokenizers emit a parity warning and retain legacy splitting.
GGUF may omit original normalization or added-token options, so reconstruction
does not guarantee exact native/original tokenizer parity for every model.
The VLM path still requires matching processor/tokenizer assets as documented above.
Metadata semantics follow the llama.cpp GGUF definitions and tokenizer implementation.
SmolLM2-135M uses the Llama tensor layout with tied embeddings. Llama GGUF conversion detects a missing separate output head and restores exported Q/K rotary projection ordering. A local HF -> FP16 GGUF -> MLX conversion was validated against the original model cast to FP16: all 272 parameter tensors, logits on six tokenizer/Unicode/code prompts, and two greedy generations matched exactly. Preserve the original tokenizer JSON in GGUF when exporting to avoid losing source normalization semantics. This is conversion parity, not a broad model-quality benchmark.
Conversion reads GGUF tensor data through memory-mapped views and decodes/transforms one tensor (or one required fusion group) at a time. Output payloads are spooled to disk immediately rather than accumulated in a shard-sized array dictionary. Shard boundaries use converted payload bytes, including quantized weights, scales, and biases; the index records their actual total size.
--max-shard-gb accepts a positive, finite size in GiB. Defaults remain 4.5 GB
(decimal) for floating-point LLM conversion and 256 MiB for direct quantization
and VLM conversion. A tensor larger than the limit occupies a shard by itself;
safetensors headers add a small amount beyond the payload limit. This setting
controls the converter's streaming stage, not a subsequent non-direct mlx-lm
quantization stage.
Memory is bounded by the largest active tensor/fusion group and transformation or quantization scratch space, not strictly by the shard limit. Header/index metadata grows with tensor count, and OS-managed memory-mapped pages may also appear in RSS. Finalizing a shard uses a bounded copy buffer and temporarily requires up to one extra shard of disk space. Failed conversions still discard the staging directory instead of publishing a partial model.