Back

Qwen 27B on 16GB

Lab notes from getting a local 27B to actually answer on an M4 Mac mini. Spoiler: the GUI lost, thinking mode ate the token budget, 3-bit + terminal won.

Reference · 2026-08-25

Machine

M4 mini

16 GB unified

Start

16.1 GB

4-bit MLX

ASCII prune

15.4 GB

still too fat

Working

11.2 GB

3-bit + no think

What we started with

Local copy of Qwen3.8-27B Uncensored MLX, 4-bit, Apple Silicon. Architecture is qwen3_5. It is a 27B hybrid-attention model with a huge tokenizer (248,320 tokens) and a default “think forever” chat template.

16 GB of RAM is the recommended floor for LM Studio. A 16 GB weight file plus OS plus KV cache is not a floor. It is a wall.

What we actually did

Leave the original alone

Worked on copies only. Source stayed at /Users/jobes/Models/Qwen3.8-27B-Uncensored-MLX/4-bit.

ASCII-condense the vocab

Script: scripts/prune_qwen38_ascii.py. Kept every byte token, every special, and any vocab row whose GPT-2 bytes were all ASCII. Dropped 120,130 non-ASCII rows. 248,320 → 128,190.

Embeddings and LM head were row-gathered in quantized space (no requant). English/code tokenization stays identical after the id remap. Non-ASCII still works via byte tokens, it just costs more tokens.

Saved about 0.7 GB. Still ~15.4 GB. Still would not fit.

Requant 4-bit → 3-bit

Script: scripts/requant_qwen_3bit.py. Lazy-load the ASCII 4-bit, dequantize each QuantizedLinear / QuantizedEmbedding, requant affine 3-bit group 64, flush every 10 modules so the 16 GB machine does not die mid-conversion.

Result: /Users/jobes/Models/Qwen3.8-27B-Uncensored-MLX-ASCII-3bit — about 11.2 GB.

LM Studio Bionic did not save us

The folder picker looks for GGUF. This model is MLX safetensors, so it says “no LLM was found.”

Bionic’s scanner wants a two-level layout, same as the 4-bit copies: ModelName/4-bit/. We symlinked the 3-bit folder to …-MLX-ASCII/3-bit and it showed up as qwen3.8-27b-uncensored-mlx-ascii (27B, 11.24 GB).

It still would not run. GUI + 11 GB weights + KV cache on 16 GB unified memory is a no. Do not fight it on this Mac.

Terminal loads it. Thinking hid the answer.

mlx_lm is lean enough to load the 3-bit. The chat template defaults to enable_thinking = true and reasoning_effort = xhigh. Generation starts inside and a small --max-tokens never reaches the reply. Looks like the model is “working” and never answering.

Fix: pass enable_thinking: false. The template then emits an empty think block and the model answers immediately. Use generate, not mlx_lm.chat — chat does not take that flag.

The command that works

python -m mlx_lm generate \
  --model /Users/jobes/Models/Qwen3.8-27B-Uncensored-MLX-ASCII-3bit \
  --prompt "What is 2+2?" \
  --chat-template-config '{"enable_thinking": false}' \
  --max-tokens 256

Want a little reasoning without a novel:

--chat-template-config '{"enable_thinking": true, "reasoning_effort": "low"}' \
--max-tokens 1024

Where the files live

What	Path	Size
Original 4-bit	…/Qwen3.8-27B-Uncensored-MLX/4-bit	16.1 GB
ASCII 4-bit	…/Qwen3.8-27B-Uncensored-MLX-ASCII/4-bit	15.4 GB
ASCII 3-bit (use this)	…/Qwen3.8-27B-Uncensored-MLX-ASCII-3bit	11.2 GB

All under /Users/jobes/Models/. Bionic sees the 3-bit via a symlink at …-MLX-ASCII/3-bit. That listing is cosmetic. Load it with mlx_lm.

Rules of thumb

→16 GB Mac: skip 27B 4-bit. 3-bit is the line.
→LM Studio import = GGUF. MLX folders need the publisher/quant layout, and the app still wants more RAM than this box has.
→Qwen3.5 thinks first. If you only see thoughts, thinking is on and max tokens is too small.
→Answers: enable_thinking: false on mlx_lm generate.

Jobes / JobeeBabyBoi · M4 Mac mini · Aug 2026