Eight Models on Hugging Face: MLX Builds With Measured Numbers

In May we announced a one-million-lira Mac Studio investment: an M3 Ultra with 512 GB of unified memory and an M4 Max with 36 GB. That post ended on a promise — that we would prepare, ourselves, the models we could run without sending data anywhere. This post is that promise, delivered.

Eight model builds are live on our Hugging Face profile. Four read text only, four also read images. Each family comes at four precisions: 4-bit, 6-bit, 8-bit and bf16. All were converted from the same source revision and measured on the same machine with the same prompt. The numbers are on the model cards; below are both the numbers and what we learned producing them.

Why conversion is needed at all

There are two ways to run a language model on Apple Silicon. The first is PyTorch with the Metal backend; it works, but it leaves most of the advantage of the unified-memory architecture on the table. The second is MLX, Apple’s own array framework. MLX never copies a tensor between CPU and GPU, because on Apple Silicon both already look at the same memory. On a 512 GB machine, that means the question of whether a model fits in GPU memory simply stops existing.

In return, MLX wants its own weight layout. A Hugging Face repo usually carries safetensors files prepared for PyTorch; MLX expects a rearranged structure to read them. Conversion is exactly that: reading the weights and writing them back in the layout MLX expects, at the precision you asked for.

Doing that conversion on our own machine every time made less sense than doing it once and publishing the result. Same goes for anyone else doing the same work.

Eight builds, two families

The source model is multimodal: it reads both text and images. But the conversion tools split on this point, and the split changes the outcome quietly.

mlx-lm converts the language tower only. Not one tensor of the vision tower reaches the output, and the generated config carries no vision_config. The result works, it is fast, it is even a bit smaller — but send it an image and nothing happens. It does not even error; the model simply never sees that input.

mlx-vlm keeps the vision tower. Of 2180 tensors in total, 333 belong to the vision tower, and those are included.

Because the same source yields two different things, we published both.

Text-only builds — Qwen3.8-27B-heretic-MLX-*

Build	Size	Bits/weight	Generation	Peak memory
4-bit	15.1 GB	4.501	37.9 tok/s	15.5 GB
6-bit	21.9 GB	6.501	27.9 tok/s	22.2 GB
8-bit	28.6 GB	8.501	22.2 tok/s	28.9 GB
bf16	50.0 GB	16	12.7 tok/s	54.1 GB

Vision builds — Qwen3.8-27B-heretic-VL-MLX-*

Build	Size	Bits/weight	Generation	Peak memory
4-bit	15.0 GB	4.695	38.9 tok/s	19.2 GB
6-bit	21.3 GB	6.661	29.2 tok/s	27.0 GB
8-bit	27.5 GB	8.627	23.1 tok/s	34.7 GB
bf16	51.0 GB	16	13.2 tok/s	55.8 GB

Measurement conditions: Mac Studio M3 Ultra, 512 GB unified memory, macOS 26.5.2. On the text side, mlx-lm 0.31.3, a 68-token prompt, 120 generated tokens. On the vision side, mlx-vlm 0.6.13, a 470-token multimodal prompt. Single run, one machine, one prompt — an order-of-magnitude guide, not a benchmark.

Three things the numbers say

First: there is a threefold speed gap between 4-bit and bf16. 37.9 against 12.7 tokens per second. Same model, same machine. Quantisation here is not a compression trick, it is a latency budget.

Second: the memory saving is more dramatic than the speed. The bf16 build wants 54 GB of peak memory; 4-bit gets by on 15.5 GB. On our 36 GB M4 Max the bf16 build will not run at all, while everything up to 8-bit runs comfortably. So quantisation is not a “slightly faster” setting — on some machines it is the difference between running and not running.

Third: the vision tower costs memory, not speed. Generation throughput on the vision builds is nearly identical to the text-only ones — marginally higher at 4-bit, in fact. But peak memory is 4-6 GB higher on every build. That is the vision tower itself plus the image-patch activations. Prompt processing stayed between 303 and 331 tokens per second across all four builds: the spread between builds shows up in generation, not prefill.

Token budget: thinking mode eats it quietly

The chat template in this family supports enable_thinking and reasoning_effort, and thinking is on by default. The model produces a reasoning block before it begins the answer, and that block spends your token budget.

In practice: leave max_tokens at something small like 128 and the whole budget can go to reasoning, leaving you with no answer at all. Instead of the answer to your question, you get the model thinking out loud.

Two ways around it. If the answer needs to be short, turn thinking off:

prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False
)

If you want the reasoning, budget for it — 512 and up. As a middle path, reasoning_effort tunes the depth.

This matters when measuring, too. The tokens-per-second figure is the same whether thinking is on or off, but the how-long-until-I-had-an-answer figure can double. When you size an application’s latency budget, the second number is the one to look at.

Usage

For the text builds, mlx-lm is enough:

pip install mlx-lm

mlx_lm.generate --model donedynamics/Qwen3.8-27B-heretic-MLX-4bit \
  --prompt "Introduce yourself briefly." --max-tokens 256

from mlx_lm import load, generate

model, tokenizer = load("donedynamics/Qwen3.8-27B-heretic-MLX-4bit")
messages = [{"role": "user", "content": "Introduce yourself briefly."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

The vision builds need mlx-vlm — load one with mlx-lm and the vision tower silently drops out:

pip install mlx-vlm

mlx_vlm.generate --model donedynamics/Qwen3.8-27B-heretic-VL-MLX-4bit \
  --image photo.png --prompt "What does this image show?" --max-tokens 256

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("donedynamics/Qwen3.8-27B-heretic-VL-MLX-4bit")
prompt = apply_chat_template(processor, model.config, "What does this image show?", num_images=1)
print(generate(model, processor, prompt, ["photo.png"], max_tokens=256))

We did not assume vision worked — we tested it

The sentence about the vision tower being intact could have been written by reading a config file. We did not write it that way. We generated a test image: a red square on the left, a blue circle in the centre, a green triangle on the right, the caption MAC512 VISION TEST, and the code 7391-ZQ.

We fed the same image to all four vision builds. Every one named the three shapes with the right colours and the right positions, and read both strings back exactly. 4-bit included.

That is small but real evidence that quantisation did not break the vision side. Small, because one synthetic image is no substitute for a real document or invoice set — do not put it into production without testing it on your own data first.

Conversion and verification

The conversion commands are one-liners:

mlx_lm.convert --hf-path trohrbaugh/Qwen3.8-27B-heretic-ara -q --q-bits 4 \
  --mlx-path Qwen3.8-27B-heretic-MLX-4bit

mlx_vlm.convert --hf-path trohrbaugh/Qwen3.8-27B-heretic-ara -q --q-bits 4 \
  --mlx-path Qwen3.8-27B-heretic-VL-MLX-4bit

But before running them we verified the source: 7 shards, 1199 tensors, all safetensors headers parsed, no missing files. All eight builds came from the same revision (a67ae100…), so the only difference between them is precision, not source.

Quantisation is affine, group size 64. That is why the bits-per-weight column is not a whole number: at 4-bit the real cost is 4.501 bits, because a scale and a zero point are stored per group.

The lineage, and a warning

The chain: Qwen/Qwen3.8-27B → trohrbaugh/Qwen3.8-27B-heretic-ara → our MLX builds.

The middle link matters. The source model is an abliterated (“heretic”) derivative: its refusal behaviour has been surgically removed, so it answers prompts a safety-tuned model would decline. Our repos change only format and precision — they add no alignment, and remove none either.

We write that as a warning, not a selling point: evaluate it before putting it in front of users, and apply your own filtering where your use case needs it. Running these builds unfiltered in a corporate assistant, a customer-facing chat interface or a product aimed at children is not the right call. In an internal analysis tool, a document-reading pipeline or a developer tool, the behaviour is predictable and contained.

The licence is Apache-2.0 throughout the chain. Credit for the model goes to the Qwen team at Alibaba Group’s Tongyi Lab, and for the abliteration to trohrbaugh. Our contribution is the MLX conversion and the measurement.

Why this work pays for itself

The commercial version of the question is this: can we build a pipeline that runs without sending customer data to an AI provider?

The eight builds are pieces of that answer. On the 512 GB machine we run bf16 to see the quality ceiling; we use 4-bit for real-time work because it runs three times faster on the same machine; light workloads go to the 36 GB M4 Max, where only the quantised builds fit. The vision builds keep document, invoice and screenshot pipelines on the same infrastructure — most of our computer vision systems work stands on them.

When this turns into a client project it is called enterprise AI development: the model choice, the quantisation decision, the latency budget and where the data sits are all discussed at the same table. The eight builds we published are the measured data we bring to it.

If you want to try them in your own setup, the repos are open, the model cards state the measurement conditions, and the licence allows it. Write to us — working out which build suits which job is useful to us too.