VLM conversion¶
LLaVA-1.5, LLaVA-NeXT, Qwen2-VL, Qwen2.5-VL, Qwen3-VL and MiniCPM-V 4.7 adapters: direct and hybrid paths, flags, and known limits.
Split out of the top-level README on 2026-10-20; content is unchanged apart from heading levels.
VLM conversion is separate from MLX-LM conversion. Six explicit adapters produce
HF-style directories for mlx-vlm:
- LLaVA-1.5 (
model_type: llava): Llama + CLIP (QuickGELU,layer_norm_eps=1e-5) + two-linear GELU projector. - LLaVA-NeXT / 1.6 (
model_type: llava_next): the same CLIP-L/336 tower plus anyres tiling. Four things decide whether the directory computes the trained function, and three of them are silent when wrong — the top-levelmodel.image_newlineparameter, the block count, the feature layer that follows from it, and the CLIP MLP tensor roles. See LLaVA-NeXT layout. - Qwen2-VL (
model_type: qwen2_vl, GGUFqwen2vl): Qwen2-VL language model with M-RoPE, ViT and 2x2 patch merger. Reassembles split Q/K/V in order and the two temporal Conv2D kernels into Conv3D; merger normalization and MLP always come from GGUF, even in hybrid mode. - Qwen2.5-VL (
model_type: qwen2_5_vl, GGUFqwen2vl): the same GGUF tensor names as Qwen2-VL — llama.cpp drives both through one graph builder — but a different function: RMSNorm ViT blocks with no norm bias, a SiLU gatedffn_gate/ffn_up/ffn_downMLP, an RMSNormln_qmerger norm, andclip.projector_type=qwen2.5vl_merger. Aliasing the two therefore yields a model that loads and emits finite logits while computing the wrong function, so the adapter rejects a Qwen2-VL export (and vice versa) instead of converting it. - Qwen3-VL (
model_type: qwen3_vl, GGUFqwen3vl): fused biasedattn_qkv, LayerNorm ViT blocks with bias, a GELU-tanhlinear_fc1/linear_fc2MLP, a Conv3d patch embedding with bias, a learnedpos_embedtable, anddeepstack_merger_list. The deepstack mergers are numbered by rank inclip.vision.is_deepstack_layers, not by layer index, and are normalized at the 4x merged width. The text half nests its hyperparameters undertext_configand carries per-head QK-norm instead of attention bias. - MiniCPM-V 4.7 1B (
model_type: minicpmv4_7): dense Qwen3.5 text GGUF plus a separate CLIP projector GGUF taggedminicpmv4_7; matching HF config, tokenizer, and processor assets are required. Direct quantization is not supported.
Qwen2.5-VL and Qwen3-VL convert directly from a language GGUF plus mmproj,
with no HF vision weights: on the pinned Qwen2.5-VL-3B and Qwen3-VL-2B pairs
the report records vision_hf: 0. The result was checked against the original
checkpoints, not just against the GGUF: all 824 (Qwen2.5-VL-3B) and 625
(Qwen3-VL-2B) output tensors match the HF index name-for-name and shape-for-shape
once the mlx-vlm language_model./vision_tower. prefix convention is applied,
and element-wise comparison of the fused QKV, the stacked patch embedding and the
merger weights agrees to float16 resolution (max |diff| 3e-8; biases, norms and
ln_q bit-identical).
LLaVA-NeXT/1.6 converts directly from a language GGUF plus a CLIP mmproj, on
the same terms as the Qwen adapters: on the pinned llava-v1.6-vicuna-7b pair the
mmproj has 378 tensors and the plan consumes all 378, the 380 emitted names
(378 + the two identity post-norm parameters, below) all exist in the
llava-hf/llava-v1.6-vicuna-7b-hf index with the same shapes, and the 375-parameter
tower loads into mlx_vlm.models.llava_next.VisionModel strictly and runs. The
text half, however, must come from the same HF checkpoint: the LLaVA GGUFs that
llama.cpp ships (cjpais, cmp-nct, mradermacher) declare 32000 tokens and no
<image>, while llava-hf's checkpoint declares 32064 with image_token_index
32000, and mlx-vlm embeds a prompt before it splices in image features. Those pairs
are refused with both counts and the missing token named. Standalone SigLIP towers
and Pixtral remain
unsupported by the full-VLM adapters. Qwen3-VL text-only extraction is
supported separately above. Other full layouts need separate
architecture contracts, not aliases to these adapters.
LLaVA and Qwen2-VL have offline, real mlx-vlm loader/processor/generation tests
on Apple Silicon using tiny random checkpoints. MiniCPM-V 4.7 now runs its native
image-path tests in the same Vision workflow job (tests/test_vlm_minicpm.py),
but large pretrained caption quality is still unverified. Large pretrained
model visual quality and cross-runtime perplexity parity are not broadly
certified.
For the pinned Qwen2-VL-2B-Instruct Q4_K_M + F16 mmproj pair, real-checkpoint
acceptance is now complete: HF / GGUF-to-FP16 / llama.cpp score 12/12 on a
fixed image diagnostic; uniform language-only MLX 4-bit has 11/12 exact
matches, but its longer orange-spacesuit answer is semantically correct.
The source-preserving Q4_K/float mode reaches 12/12 exact and semantic checks.
It keeps Q4_K codes and retains higher-precision source layers as floats, rather
than forcing every projection to 4-bit. All 729 floating
output tensors pass the independent weight audit, with exact visual-feature
parity against HF on those inputs. See the full report and reproduction
commands, including the original exact-format failure
and the limited scope. This is not broad visual-quality certification. The
Qwen2.5-VL and Qwen3-VL weight checks above are structural and per-tensor; they
are not a caption-quality or perplexity result.
# Optional HF download support; local sources need only the base converter
pip install "gguf2mlx[vlm]"
# Path A: language + vision + projector weights from GGUF.
# HF supplies matching full VLM config/tokenizer/image processor, not weights.
gguf2mlx convert --type vlm \
--input language.gguf --mmproj mmproj.gguf \
--hf-model llava-hf/llava-1.5-7b-hf --output ./llava-direct
# Path B (recommended): language + projector from GGUF, original vision weights
# and matching config/tokenizer/processor from the full HF VLM checkpoint.
gguf2mlx convert --type vlm \
--input language.gguf --mmproj mmproj.gguf \
--hf-fallback-vision llava-hf/llava-1.5-7b-hf --output ./llava-hybrid
# Offline hybrid conversion from a matching local HF safetensors checkpoint
gguf2mlx convert --input language.gguf --mmproj mmproj.gguf \
--hf-fallback-vision /absolute/path/to/hf-llava --offline \
--output ./llava-offline
# Qwen2-VL: matching language GGUF + Qwen2-VL mmproj; HF supplies assets only
gguf2mlx convert --type vlm \
--input qwen2-vl.gguf --mmproj qwen2-vl-mmproj.gguf \
--hf-model Qwen/Qwen2-VL-2B-Instruct --output ./qwen2-vl-mlx
# Qwen2.5-VL: language GGUF + qwen2.5vl_merger mmproj; HF supplies assets only
gguf2mlx convert --type vlm \
--input qwen2.5-vl.gguf --mmproj qwen2.5-vl-mmproj.gguf \
--hf-model Qwen/Qwen2.5-VL-3B-Instruct --output ./qwen2.5-vl-mlx
# Qwen3-VL: language GGUF + qwen3vl_merger mmproj, deepstack mergers included
gguf2mlx convert --type vlm \
--input qwen3-vl.gguf --mmproj qwen3-vl-mmproj.gguf \
--hf-model Qwen/Qwen3-VL-2B-Instruct --output ./qwen3-vl-mlx
# MiniCPM-V 4.7 1B dense: Qwen3.5 language GGUF + matching MiniCPM-V mmproj
gguf2mlx convert --type vlm \
--input minicpm-v-4.7-1b.gguf --mmproj minicpm-v-4.7-1b-mmproj.gguf \
--hf-model openbmb/MiniCPM-V-4.7 --output ./minicpm-v-4.7-1b-mlx
# LLaVA/Qwen2-VL: affine 4-bit language projections, floating vision/projector
gguf2mlx convert --type vlm \
--input language.gguf --mmproj mmproj.gguf \
--hf-model /absolute/path/to/matching-hf-vlm --offline \
--quantize --direct-quant --q-bits 4 --q-group-size 64 --q-mode affine \
--output ./vlm-language-4bit
# Qwen2-VL Q4_K_M: avoid a second lossy quantization (recommended)
gguf2mlx convert --type vlm \
--input qwen2-vl-Q4_K_M.gguf --mmproj qwen2-vl-mmproj.gguf \
--hf-model /absolute/path/to/matching-hf-qwen2-vl --offline \
--quantize --direct-quant --preserve-source-quant \
--output ./qwen2-vl-source-preserved
--type auto(default) detects multimodal architecture metadata and tensors. A language-only GGUF cannot identify a separately distributed vision tower: pass--mmproj, HF source options or--type vlm.--type llmrejects detected multimodal inputs instead of silently discarding vision weights.- Omit
--mmprojfor a combined GGUF. Both paths require GGUF projector weights; hybrid conversion never replaces the language model or projector with HF weights. - MiniCPM-V currently requires its separate 4.7 1B dense
clipprojector GGUF. The 35B-A3B MoE and older 4.6 layouts are rejected; direct quantization is not enabled for MiniCPM-V. --hf-modelmust describe the matching full VLM, not a bare CLIP model.--hf-fallback-visionalso supplies that source's assets; if both options are specified, they must reference the same source. Pin--hf-revisionto a HF commit for reproducible downloads.--offlineuses local/cached assets only.- Matching
config.json,tokenizer.json,tokenizer_config.jsonandpreprocessor_config.jsonare required. The converter validates architecture, tensor shapes/completeness, image normalization metadata and tokenizer compatibility, then copies processor and tokenizer assets without executing remote code or loading pickle files. Not every GGUF/HF release pair is compatible; vocabulary/config mismatches fail rather than fabricating weights. All tokenizer IDs, including image and padding tokens, must fit the language embedding vocabulary; older unresized exports are not repaired by synthesizing embedding rows. - Direct conversion requires a complete supported vision layout. LLaVA-NeXT
reconciles the two omissions a real CLIP
mmprojhas — a tower one block shorter than the HF config declares, and an absentv.post_ln— instead of sending you to the hybrid path; see below. A short attention or MLP tensor, or a block count off by more than the pooler-only block, still fails. - Hybrid conversion reads only vision tensors from safetensors. Indexed HF checkpoints download only shards containing vision weights (a shared shard can also contain language weights); an unsharded checkpoint requires fetching that file. Direct conversion does not download HF weight files.
- Output includes
config.json, tokenizer/processor assets, safetensors shards, an index andvlm_conversion_report.json. Validation failures clean staging files and never overwrite an existing nonempty output directory. --dtype float16(default) orfloat32controls floating-point tensors.--quantize --direct-quanttogether enable affine 4-bit, group-size-64 quantization of language attention/MLP projection matrices for LLaVA-1.5, Qwen2-VL, Qwen2.5-VL and Qwen3-VL. MiniCPM-V 4.7 rejects the pair. Embeddings, LM head, norms, biases, vision tower and projector remain floating-point. Quantization is decode then re-quantize, not bit-preserving GGUF transpacking; misaligned projections fail instead of silently falling back.--quantizealone and MoE--mixed-precisionare rejected for VLMs. The report lists quantized modules, logical/stored tensor counts and layouts.- For a Qwen GGUF with Q4_K language projections, add
--preserve-source-quant(now an alias of--quant-source preserve; both are accepted here) to keep their original four-bit codes and 32-element groups. This option overrides the default 64-element re-quantization layout; leave--q-*at their defaults. Other language projections (including Q6_K) are decoded to the requested floating dtype, not reduced to four bits. Vision/projector and embeddings remain floating. This is an explicit mixed-precision alternative, not a promise of an all-four-bit model or bit-exact FP16 arithmetic: scales and biases still round to--dtype. The flag requires an architecture with a vision tensor plan, so it is accepted for Qwen2-VL, Qwen2.5-VL and Qwen3-VL and rejected for LLaVA and MiniCPM-V. The measured artifact is the Qwen2-VL-2B pair: about 2.7 GiB versus 2.4 GiB for uniform 4-bit and 4.1 GiB for FP16. Size and parity have not been measured for the Qwen2.5-VL or Qwen3-VL artifacts. Missing Q4_K projections, LLaVA, or use without--quantize --direct-quantfail explicitly. Every projection's source/output policy is recorded. - HF assets must be from the original unquantized checkpoint. The output rebuilds model configuration and quantization metadata; it does not carry stale HF quantization settings into the loader.
multi_modal_config.jsonis not a universal MLX-VLM requirement and is not fabricated. LLaVA uses nested text/vision configuration; Qwen2-VL, Qwen2.5-VL and Qwen3-VL all write a nestedtext_config, and the Qwen2 families also repeat the language hyperparameters at the top level because the tested loader reads them from there. Qwen3-VL does not: its text hyperparameters live only undertext_config, matching the original checkpoint. Processor/tokenizer semantics cannot be reliably inferred from tensor names or image mean/std alone, so matching HF assets remain required rather than guessed defaults.- Install
mlx-vlmseparately on Apple Silicon to load the output. The tested runtime is pinned tomlx-vlm==0.7.10intests/requirements-vlm.txtand thevlmextra, and should be installed in a separate environment. Older0.7.xreleases lack theminicpmv4_7model entry and 0.7.6's LLaVA processor is incompatible withtransformers==5.19.0: it forwards text-onlypaddingto the image processor, which now rejects it. Native CI uses Transformers 5.16.1; the existingdevextra retains its newer Transformers requirement. Do not combine the two environments. Thevlmextra pulls the matchingmlx-vlmpin on Apple Silicon, but the pinned test environment intests/requirements-vlm.txtremains the reference for what CI exercises.
GGUF vision quantization is not inherently unusable: quality depends on the source quantization and model. Hybrid conversion avoids that vision quantization loss but does not recover language/projector information already lost in GGUF. Prefer direct HF-to-MLX-VLM conversion when all original weights are available.
LLaVA-NeXT layout: the tower is shorter than HF says¶
LLaVA-NeXT is not a new vision encoder, so the interesting work is the places where the GGUF and the HF config describe the same tower differently. Four of them, three silent:
model.image_newlineis outside the tower. It is a top-level[hidden]parameter in mlx-vlm, not avision_tower.*one, and it is neitherv.*normm.*. A converter that routes by prefix drops it and produces a directory that loads, stiches anyres tiles, and separates them with nothing.- The stored tower is one block shorter. A real llava-v1.6
mmprojdeclaresclip.vision.block_count = 23for HF's 24 blocks, because HF'svision_feature_layer = -2with 24 blocks is block 22's output, and block 23 only feeds the pooler LLaVA discards. Emitting 24 layers asks for 32 tensors no file has; emitting 23 while keeping-2silently selects block 21 and produces finite, wrong features. The stored count is authoritative andvision_feature_layermoves with it, to-1. Both facts land invision_layoutin the report. v.post_lnis absent. mlx-vlm declarespost_layernormand loads strictly, so the parameter has to exist; it is written as the identity and recorded invision_identity_fills. That is an explicit fill, not a silent default: the only consumer ispooler_output, which everyllava_nextcode path discards, andtests/test_vlm_llava_next.pyasserts that perturbing the identity leaves the image features bit-identical.ffn_upandffn_downswap roles between exporters. llama.cpp's LLaVA-era loader readsffn_downasff_i(the GELU input) andffn_upas the output projection; the modern mtmd graph readsffn_upas the expansion. Both files exist in the wild under both names, and nothing in the metadata says which is which — so the shapes decide: a CLIP MLP iswidth -> feed_forward -> width, and the tensor shaped(feed_forward, width)is the expansion. The choice is recorded asclip_ffn_convention(llava-toolsormtmd) invision_layout. Every pre-mtmd export, llava-1.5 included, isllava-tools.
Two more GGUF fields look relevant and are not. image_crop_resolution and
image_split_resolution are LLaVA-1.5 leftovers: a real 1.6 export declares 224
next to a 336 tile. Tiles are cut by preprocessor_config.json, whose crop size and
grid are pinned to config.json by the processor check, so the pair is recorded in
vision_layout rather than enforced.
The text half is the one thing that cannot be reconciled. mlx-vlm embeds a
prompt before it splices in image features, so image_token_index has to be inside
the language embedding. llava-hf's checkpoints satisfy that by padding to 32064
rows and putting <image> at 32000; the LLaVA GGUFs that llama.cpp ships declare
32000 tokens with no <image> at all, because llama.cpp injects image embeddings
positionally. Pairing them is refused, with both counts and the missing token named:
GGUF tokenizer declares 32000 tokens but the HF text config declares vocab_size
32064: HF ID(s) 32000..32063 (32000=<image>) have no row in the GGUF embedding
table. config.image_token_index=32000 (<image>) is one of them: mlx-vlm embeds the
prompt before it splices in image features, so that row has to exist. Supply a GGUF
converted from the same HF checkpoint rather than a repack that trimmed the extra
rows.
SigLIP ffn roles in MiniCPM-V are declared, not guessed¶
MiniCPM-V's vision tower is a SigLIP variant, so the ffn_up/ffn_down naming
trap above applies to it in principle. In practice it does not: MiniCPM-V is only
ever converted by tools/mtmd, whose ffn_up is the expansion, so there is no
second convention to resolve. vision_minicpm.py therefore keeps the roles in
_VISION_BLOCKS and asserts the declared shapes against the file, and the mapping
says why in a comment — the shape resolution used for LLaVA-NeXT looks like a
strict improvement here too, but applied to a tower with one convention it would
only invent ambiguity.
What a mislabelled file does produce is the shape mismatch on mlp.fc1, which now
carries the diagnosis rather than the raw numbers:
VLM shape mismatch for vision_tower.encoder.layers.0.mlp.fc1.weight: got (4, 8),
expected (8, 4) — a CLIP/SigLIP MLP is width -> feed_forward -> width, so this
file labels the expansion and projection under the opposite ffn_up/ffn_down roles
from the one this adapter resolved