Compare Qwen3.6-35B-A3B, DeepSeek-V4-Flash and Gemma 4; understand how quantization decides what fits in 128 GB.
Coleção de modelos convertidos para MLX; usada para escolher tamanhos e quantizações que cabem no M5 Max.
Modelos de IA são enormes. Para caber no Mac, usamos quantização: representar cada peso com menos bits. Quanto menos bits, menor e mais rápido, mas com leve perda de qualidade. O segredo é escolher a quantização certa para cada tarefa.
Pense em compressão de música: MP3 de 128 kbps é pequeno e "bom o suficiente"; FLAC é enorme e fiel. Para ouvir no carro, MP3 vale a pena; para masterização, você quer FLAC.
Quantization reduces weight precision (and optionally KV-cache precision). On the M5 Max 128 GB, Q4_K_M or 4bit is the sweet spot: perplexity degradation is typically only 1–3% versus FP16. Q3 visibly drops on long reasoning and code; Q2/IQ2 are only worthwhile for giant-context workloads where window size matters more than fidelity. Q8_0/8-bit is used when maximum quality is critical and the model still fits.
For text/code/RAG, Qwen3.6-35B-A3B is the main recommendation: ~80 tok/s in 8-bit on the M5 Max 128 GB, 256K context, strong in Portuguese and a leader in code benchmarks. For extreme 1M-token context, DeepSeek-V4-Flash fits tightly in 3-bit (~103 GB). Gemma 4 also handles text, but shines in multimodal tasks.
Qwen3.6-35B-A3B is MoE (~35B total, ~3B active per token), measured at ~80–110 tok/s on the M5 Max 128 GB with 256K context. Detailed quality benchmarks (SWE-Verified, MMLU) are proxies based on the Qwen3.5-35B-A3B architecture; use it as the speed/memory balance engine. DeepSeek-V4-Flash is MoE 284B/~13B active, Arena ELO 1431, up to 1M context. Gemma-4-31B is dense, Arena ELO 1441–1452, MMLU-Pro 85.2%, 256K context, native multimodal. Qwen3.5-122B-A10B delivers ELO 1418 and 262K context; GLM-5.2 reaches ELO 1465.
| Model | Local speed* | LLMCheck | SWE-Verified | MMLU | Arena ELO | Context | Quantization | Multilingual |
|---|---|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | ~80–110 tok/s | — | ~70%* | ~86%* | ~1396* | 256K | 8bit ~22–28 GB / 4bit ~14 GB | Strong PT/EN |
| DeepSeek-V4-Flash | ~34 tok/s (IQ2/UD-IQ3) | — | — | — | 1431 | 1M | UD-IQ3_XXS ~103 GB | — |
| Gemma 4 31B | — | — | — | MMLU-Pro 85.2% | 1441–1452 | 256K | 4bit ~20 GB | MMMLU 88.4% |
| Gemma 4 26B-A4B | ~50 tok/s | — | — | — | — | 256K | 4bit ~18 GB | — |
| Qwen3.5-122B-A10B | — | — | — | — | 1418 | 262K | 4bit ~70 GB | MMMLU 86.7% |
| GLM-5.2 | — | — | — | — | 1465 | — | varies | — |
*Speeds measured on an M5 Max 128 GB with MLX; tok/s depend on prompt and system load. SWE-Verified, MMLU and Arena ELO values for Qwen3.6-35B-A3B are proxies based on Qwen3.5-35B-A3B and community reports.
Além de quantizar pesos, você pode quantizar a KV cache (memória de contexto), ativar Flash Attention e usar speculative decoding (um modelo pequeno sugere tokens que o grande confirma). Essas técnicas multiplicam a janela de contexto e a velocidade.
KV cache quantization reduz memória de estados chave/valor em 1,3–6,4×. Flash Attention diminui acessos à HBM. Speculative decoding usa um draft model para gerar candidatos. No MLX: --kv-bits 8 e --max-kv-size 65536; no Ollama: OLLAMA_KV_CACHE_TYPE=q8_0.
# KV cache quantization on MLX python -m mlx_lm.server --model mlx-community/Qwen3.6-35B-A3B-8bit --kv-bits 8 --max-kv-size 65536 # Ollama com KV cache q8_0 export OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve # Speculative decoding com draft model python -m mlx_lm.generate --model large_model --draft-model small_draft_model --num-draft-tokens 4
Test what you learned with quick exercises.