Combine many local agents, fuse their perspectives, and validate outputs before any token leaves the cluster.
report-cluster-ai.html in the workspace — the interactive document that consolidated the research and originated this course.
Instead of trusting one model, ask several local agents the same question. Each agent can use a different model or quantization. The diversity reduces blind spots and surfaces hidden assumptions.
This pattern is cheap because every call stays on the Mac. It is especially useful for risky or ambiguous decisions.
Send the same prompt to multiple local endpoints: mlx_lm.server with Qwen3.6-35B-A3B-8bit, a lower-bit variant, and DeepSeek-V4-Flash when the context demands it. Collect the answers as a list of opinion objects with model name, confidence and raw output. The differences are more valuable than the average.
{
"opinions": [
{"model": "Qwen3.6-35B-A3B-8bit", "answer": "Use option A.", "confidence": 0.8},
{"model": "Gemma-4-26B-A4B-4bit", "answer": "Prefer option B; option A ignores latency.", "confidence": 0.7}
]
}
After collecting opinions, a local model reads all of them and writes a consolidated brief. The brief states what everyone agrees on, where they disagree, what is still unknown, and what the recommended action is.
The result is a decision-ready document small enough to send to a Frontier model or to act on directly.
Feed the opinion list into a fusion prompt: "You are a senior analyst. Summarize the consensus, disagreements and open questions. Recommend one action and explain the risk." Use Qwen3.6-35B-A3B as the fusion model because it is fast and has a 256K context window. Enforce JSON schema output so the next stage can validate it.
def fuse(opinions, prompt_template): ctx = "\n\n".join( f"[{o['model']}] {o['answer']}" for o in opinions ) return local_llm.chat( messages=[{"role": "user", "content": prompt_template + ctx}] )
Before any payload goes to the cloud, run executable gates. A gate can be a JSON schema check, a unit test, a syntax check, or a small local model that says yes or no.
Validation is the safety net. It catches hallucinated citations, malformed commands and unsupported claims before they become expensive Frontier mistakes.
Example gates: jsonschema.validate for structure, python -m py_compile for generated code, subprocess to run a dry-run command, or a tiny local classifier that flags uncertainty. If any gate fails, the orchestrator routes the task back to the worker instead of forwarding it to the Frontier API.
import jsonschema schema = { "type": "object", "required": ["recommendation", "risks", "confidence"], "properties": { "confidence": {"type": "number", "minimum": 0, "maximum": 1} } } jsonschema.validate(fused_brief, schema) # raises if the brief is malformed
Scale the agent factory with three layers: the orchestrator decides the goal, team leads break it into subtasks, and workers run local models and tools.
This keeps each model's context small and focused, which is exactly what makes local quantized models perform well.
The orchestrator is usually deterministic code — LangGraph, n8n or a Python state machine. Team leads are prompt templates or lightweight local models that route work. Workers are the actual model endpoints, each running in its own process or tmux pane. The Frontier model sits above the orchestrator only for the final decision.
A headless cluster has no GUI dashboard, but a terminal multiplexer is the next best thing. With tmux or cmux you can tile one pane per agent and watch every local model stream in real time.
The visibility layer is not just eye candy. It helps you debug routing, spot stuck workers, and measure latency at a glance.
Start a named session: tmux new-session -s cluster -d. Split into panes for mlx_lm.server --port 8080, mlx_vlm.server --port 8000, the orchestrator logs, and a worker pool. Use tmux send-keys to script pane commands or cmux if your cluster is defined as a YAML layout.
# Start a session and split into a 2x2 grid of panes tmux new-session -s cluster -d # pane 0: text model tmux send-keys -t cluster:0.0 'mlx_lm.server --model mlx-community/Qwen3.6-35B-A3B-8bit --port 8080' C-m # pane 1: vision model tmux split-window -h -t cluster:0.0 tmux send-keys -t cluster:0.1 'mlx_vlm.server --model mlx-community/Gemma-4-26B-A4B-MLX-4bit --port 8000' C-m # attach to watch the cluster breathe tmux attach -t cluster
Not every agent needs the biggest model. Assign roles by speed, context and modality. The table below is a starting point for the Mac M5 Max 128 GB.
Quantization choices come from the earlier architecture lesson: 8-bit for the main text engine, 4-bit for multimodal, and UD-IQ3_XXS for the giant context model loaded on demand. Keep one model hot and swap the second as needed, or run both Qwen and Gemma concurrently if memory allows.
| Agent role | Model | Why |
|---|---|---|
| Fast text / RAG worker | Qwen3.6-35B-A3B-8bit | ~80 tok/s, 256K context, strong code. |
| Vision / audio worker | Gemma-4-26B-A4B-4bit | Native multimodal, ~18 GB. |
| Long-context analyst | DeepSeek-V4-Flash UD-IQ3_XXS | 1M context, loaded on demand. |
| Fusion / validation | Qwen3.6-35B-A3B-8bit | Fast synthesis and schema-aware output. |
| Orchestrator | Deterministic code or small local model | Routing and state machine, not reasoning. |
Test what you learned with quick exercises.