Publishing: MLX alongside GGUF, and PyPI releases¶
The reusable
publish-mlxworkflow and the release process for this package.Split out of the top-level README on 2026-10-20.
The gguf2mlx publish command¶
The workflow is a thin shell around this command; anything the workflow does can be reproduced locally with a token.
pip install "gguf2mlx[publish]"
export HF_TOKEN=... # source read, destination write/create
gguf2mlx publish \
--source your-org/your-model-GGUF \
--gguf your-model-Q4_K_M.gguf \
--dest your-org/your-model-MLX \
--source-revision <upload-commit-sha>
What it does, in order (all validation happens before the first network call):
- validates
--source/--dest(repository ids, no../--, source ≠ dest) and--gguf(repository-relative, normalised,.gguf); - resolves
--source-revisionto an immutable commit; - lists the repository and expands a split GGUF into its full shard set — a partial set is an error naming the missing indices, because split GGUFs are byte slices of one file and a missing shard would publish a truncated model;
- downloads every shard and streams it into the merged GGUF in fixed-size chunks, so peak memory is one buffer rather than one shard, then converts it;
- writes
gguf2mlx-provenance.json(source repo, resolved commit, shard list, GGUF SHA-256 and size, precision, converter version, quant source, output size, and the exact reproduce command); - rewrites the model card:
library_name: mlx,base_model: <source>,base_model_relation: quantizedwhen requantized,disk_size, themlxandgguf2mlxtags plus the artifact's dtype/precision tags, a conversion table (source GGUF, SHA-256, byte counts, weight and preserve gate results), a reproduce command, and a usage snippet — then copiesLICENSE/NOTICE/COPYINGfiles; - creates the destination repository and uploads the folder — but only after
conversion produced
config.jsonand at least one safetensors shard.
Dry runs and load verification¶
Two flags make a publish reviewable before it becomes visible:
# Everything except the destination write: resolve, download, convert, and
# build the card + provenance in the work tree, then report and exit.
gguf2mlx publish --source ... --gguf ... --dest ... --dry-run
# Load the converted directory with mlx-lm and run one forward pass before the
# destination repo exists, so a layout defect fails the run instead of
# publishing an unloadable model.
gguf2mlx publish --source ... --gguf ... --dest ... --verify-load
--dry-run never calls create_repo or upload_folder. --verify-load runs
before either, so a failed load leaves the destination untouched. Both are
recorded on the result (dry_run, verified_load), and --dry-run is the only
way to inspect the finished card without publishing.
--verify-load delegates to the same check the release gate uses
(fidelity.load_and_run): it loads the artifact, runs a single forward pass,
and requires finite logits. That is deliberately stricter than loading alone —
a tensor with the right name and shape but the wrong layout loads cleanly and
still produces garbage. A missing runtime counts as a failure rather than a
skip, because the flag was asked for explicitly. It requires MLX on Apple
Silicon.
--quantize enables lossy affine 4-bit (requires MLX, so Apple Silicon).
--quant-source preserve keeps the source GGUF blocks instead of quantizing
from scratch: codes are copied verbatim, except for Q6_K, whose 16-value
scales do not fit MLX's 32-value grid and are re-fitted at the same width (see
the quality gates); it still needs the
result to be loadable by mlx-lm, so it is limited to llama, qwen2,
qwen3, mistral, and the Qwen3.5 family (qwen35 and its aliases).
--keep-work-dir leaves the temporary tree in place for debugging.
Environment: HF_TOKEN is read when --token is absent. --work-dir moves the
temporary download/convert tree (defaults to the system temp directory).
Publish MLX alongside GGUF on Hugging Face¶
The reusable publish-mlx workflow lets model
publishers chain GGUF uploads into MLX publication, including repositories in
mlx-community when the publisher has write access to that organization.
It does not grant organization membership or monitor Hugging Face uploads.
Add a caller like this to your publishing repository's .github/workflows/:
name: Publish GGUF and MLX
on:
workflow_dispatch:
permissions:
contents: read
jobs:
upload-gguf:
runs-on: ubuntu-latest
outputs:
revision: ${{ steps.upload.outputs.revision }}
steps:
# Insert your existing GGUF upload steps here.
# The step with id: upload must write revision=<HF upload commit SHA>
# to "$GITHUB_OUTPUT", after all GGUF files and README.md are uploaded.
- id: upload
run: |
echo "Replace this step with your existing HF upload job."
exit 1
publish-mlx:
needs: upload-gguf
uses: barrontang/gguf2mlx/.github/workflows/publish-mlx.yml@main
with:
source_repo: your-org/your-model-GGUF
gguf_file: your-model-Q4_K_M.gguf
source_revision: ${{ needs.upload-gguf.outputs.revision }}
destination_repo: your-org/your-model-MLX
private: true
quantize: false
secrets:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
Replace the placeholder upload job with your existing pipeline and keep the
needs dependency; use its actual output name for the HF commit SHA. Pin the
reusable workflow to a reviewed full commit SHA instead of @main for
production. For uploads outside GitHub, have the trusted uploader trigger a
workflow_dispatch caller after the HF commit completes, passing the source
repository, file, and revision as inputs. Uploading to HF alone does not trigger
a GitHub workflow.
- Configure
HF_TOKENas a GitHub Actions secret: source read access (including any gated-model approval) and destination write/create access are required. Use a fine-grained token and pass only this secret, notsecrets: inherit. Never run publishing from untrusted pull requests or accept unreviewed runner, converter-version, or destination inputs. No caller code is checked out. - Float16 conversion needs only
numpy/gguf/safetensors, so the default runner isubuntu-latest. Onlyquantize: trueusesrunner(defaultmacos-15) and checks macOS/ARM64 before installing anything; MLX ships Apple Silicon wheels only. Size RAM and disk for the GGUF, HF download cache, converted weights, and temporary files; FP16 output may be much larger than a quantized source. Self-hosted runners must be trusted and their HF caches cleaned according to your private-model retention policy. converter_versiondefaults to2.2.0, the first release that ships thepublishsubcommand; select a reviewed published version supporting your model. The workflow installsgguf2mlx[vlm]to obtain the existing Hugging Face client extra, but converts supported text LLMs only, not VLM/projector or text-encoder models. Pass any shard of a split GGUF: the whole set is discovered, missing shards fail the run before anything is downloaded, and the shards are merged before conversion.source_revisiondefaults tomain, but a successful upload's commit SHA is recommended. Downloads, the original model card, and license/notice files are pinned to the resolved HF commit. The resulting model card retains source license metadata and includes a provenance JSON with source, version, and precision. Check redistribution rights and the generated card before making a model public; conversion does not change its license.- FP16 is the default.
quantize: trueexplicitly enables lossy affine 4-bit direct quantization with group size 64; it is not bit-exact preservation of GGUF quantization. (The CLI also exposes--quant-source preserve; the workflow does not, because mixing preserved widths into a public artifact is a deliberate per-model decision.) Failed downloads or conversions do not publish artifacts. - New HF repositories are private by default.
privatedoes not change the visibility of an existing repository. Each upload commits the converted folder together; reruns update matching files but do not delete stale remote shards. Prefer a fresh destination for a different model/layout, and never use the source GGUF repository as the destination. Publications to the same destination are serialized within the calling GitHub repository.
Keep the versions in pyproject.toml and src/gguf2mlx/__init__.py identical.
Merge the release changes, then run Actions → Publish to PyPI → Run workflow
on main with target pypi. Alternatively, push a matching vX.Y.Z tag.
The workflow checks the versions, builds and validates the distributions, publishes
to PyPI, and creates a GitHub Release (and its tag if missing) at the built commit.
The testpypi target does not create a GitHub Release.
The old v2.1.2 and v2.1.3 tags contain package metadata for 2.1.1; do not
rerun them or move them. Version 2.1.4 is the fresh release prepared to recover
from those failed uploads. PyPI cannot overwrite an existing distribution.
If publishing succeeds but GitHub Release creation fails, rerun only the failed
jobs from that same run rather than rebuilding or uploading the package again.