Speed
The #1 inference engine for speed and throughput on Mac.
Midium has some of the fastest speeds of any provider on OpenRouter. The full runs are below.
Throughput
The #1 inference engine on Mac.
Fastest decode in every test we ran, against LM Studio, Ollama, and oMLX. Apple M5 Max, 128 GB · October 2026.
- Short coding ask
- 133 tok/s
- 2.0× Ollama · +5% oMLX
- First token, 33k prompt
- 23×
- ~12.4s vs Ollama’s ~285s · oMLX ~12.9s
- Editing a file
- 3.7×
- 268.1 vs 71.8 tok/s · 1.5× oMLX
125B mixture of experts · tuned for Apple Silicon
A 125B model at over 130 tokens a second.
Flash Next is a brand-new architecture, and Midium runs it on an engine path tuned specifically for it, built around the model’s own multi-token prediction (MTP). Same model, a lot more speed, and long prompts start answering in seconds, not minutes.
- 133 tok/s on a short coding ask: 2× Ollama, 5% ahead of oMLX.
- 3.7× Ollama’s speed, and 1.5× oMLX’s, when the answer repeats the prompt, like returning an edited file.
- 23× sooner to a first token on a 33k-token prompt: about 12.4s against 285s.
| Test | Midium · MTP on | oMLX · MTP on | Ollama · MTP on | Midium · MTP off |
|---|---|---|---|---|
| Decode | ||||
| Short coding ask42-token prompt | 132.8tok/sbest 141.8↑ 5% oMLX · 2.0× Ollama · 1.7× MTP off | 126.6best 132.4 | 67.8best 71.7 | 78.3 |
| Write a new moduleAfter a 33k-token codebase | 96.0tok/sbest 103.6↑ 11% oMLX · ↑ 84% Ollama · 1.4× MTP off | 86.3best 93.0 | 52.2best 54.4 | 70.0 |
| Edit a file6k-token file, one change | 268.1tok/sbest 278.2↑ 52% oMLX · 3.7× Ollama · 3.7× MTP off | 176.6best 182.9 | 71.8best 73.7 | 72.6 |
| Same edit, as a tool callWhole file saved through write_file | 239.2tok/sbest 240.8↑ 39% oMLX · 3.2× Ollama · 3.6× MTP off | 171.5best 176.1 | 74.6best 75.5 | 66.0 |
| Prompt processing | ||||
| Prefill, 6kFirst token ~2.3s · ~2.5s · ~31s | 2,450tok/sbest 2,636≈ oMLX · 12× Ollama | 2,476best 2,592 | 200best 214 | 2,596 |
| Prefill, 33kFirst token ~12.4s · ~12.9s · ~285s | 2,651tok/sbest 2,697↑ 4% oMLX · 23× Ollama | 2,554best 2,650 | 115best 119 | 2,583 |
Each figure is the mean of 9 runs (3 rounds × 3 seeds), “best” the fastest run; MTP off is the mean of 3 runs.
All three runtimes draft with the model’s own MTP head. Ollama runs a different 4-bit build than Midium and oMLX because neither side can load the other’s: Ollama can’t load the build Midium runs, and Midium can’t load Ollama’s, so each is measured on the build it supports. LM Studio couldn’t load this model, so it has no column (see the methodology). Setup and measurement details are in the methodology below.
Methodology
- Machine
- The same MacBook Pro, Apple M5 Max, 128 GB with fans at full speed, the chip cooled to 40 °C or lower (held for 5 seconds) before every job. Midium on MLX 0.32.3, oMLX 0.7.1.dev1, Ollama 0.40.0. Ollama needed the GPU memory limit raised to ~120 GB to avoid running out of memory; Midium runs at the default.
- Weights
- Midium and oMLX run the same oQ4e MLX build (99 GB; oMLX needs a companion patch to load it), Ollama its official NVFP4 MLX build (104 GB). They can’t share one: Ollama can’t load the oQ4e build, and Midium can’t load Ollama’s. All include the model’s multi-token prediction (MTP) head, and all three runtimes draft with it.
- Requests
- Temp 1.0, top-p 0.95, top-k 20, thinking off, streaming, 3 seeds per job, 3 rounds per runtime. Every prompt ran cold: a unique prefix on each Midium and oMLX request, a model reload before each Ollama run, and one unmeasured warm-up after every load. MTP off is Midium on the same weights, one token per step.
- Measuring
- Decode is each runtime’s own reported rate after the first token. Prefill is prompt tokens over the time to a first token (Ollama: its own reported prompt-processing rate). Tool calls: the whole 6k-token file saved through write_file.
Measured October 13, 2026 · Qwen3.8 Flash Next · Apple M5 Max, 128 GB, cooled to 40 °C before every job · temp 1.0 · thinking off · mean of 9 runs, best run below.
Every Midium number here comes from Midium’s own inference engine, not a wrapper around llama.cpp, Ollama, LM Studio or another app’s server. We build it ourselves on Apple’s MLX and tune it model by model for three things: speed, tool calls you can rely on, and catching runaway output before it wastes your time.
Batching
Requests are batched through the runtime.
Model pooling
Several instances sit behind one API and take work round-robin.
Loop detection
A watchdog detects an endless-loop hallucination, kills the model, and restarts it so throughput keeps moving.