Apple Silicon LLMs 2026: MLX + What Runs on 8–512 GB

What changed for Mac LLMs in the second half of 2026?

If you last looked at this in spring, four things moved. This table is the short version, and every row is checked against the project's own release feed as of 5 October 2026.

What can my Mac actually run?

This is the only question most people have, and it has a precise answer, because every MLX conversion on Hugging Face has a measurable on-disk size. The figures below are actual repository sizes pulled from the Hugging Face API on 5 October 2026 — not estimates.

The rule of thumb: weights must fit in unified memory with roughly 25–30% left over for the OS, the KV cache, and your other applications. A 16.1 GB model on a 24 GB Mac works but leaves little room for long contexts; on 32 GB it is comfortable.

Two caveats that trip people up. First, macOS does not hand the GPU all of your RAM by default. When a model is close to your ceiling, Apple's own mlx-lm documentation tells you to raise the wired limit explicitly:

sudo sysctl iogpu.wired_limit_mb=N

Apple's guidance is that N should be larger than the model size in megabytes but smaller than the machine's total memory. MLX also exposes mx.set_wired_limit() and mx.set_memory_limit() from Python. Second, the KV cache grows with context length and is not included in the sizes above — a 128K-token context can add several gigabytes on a 27B model.

Why is Apple Silicon good at this in the first place?

Two properties, one of which is widely misunderstood.

Unified memory. Apple's own framework documentation puts it plainly: the CPU and GPU have direct access to the same memory pool, and an MLX array simply lives in unified memory with no device placement. There is no copy across PCIe and no separate VRAM budget. This is why a 64 GB Mac can hold a model that a 24 GB discrete GPU cannot load at all, and it is the entire reason the Mac is interesting for local inference.

Memory bandwidth, not compute. Token-by-token generation is bandwidth-bound — the machine must read the active weights for every token it emits. Prompt processing is the opposite: compute-bound, and it batches well.

Apple now draws this distinction explicitly in its own marketing, which is useful because it tells you which spec to read for which number. Announcing M6, Apple attributed the Neural Accelerators in each GPU core to "significantly faster prompt processing when interacting with on-device LLMs". In the same release, tokens-per-second is attributed to something else entirely — M5 Ultra's "up to 512GB of unified memory and 1.2TB/s of memory bandwidth" is what lets you "increase the tokens-per-second speed".

So: Neural Accelerators buy you time-to-first-token. Memory bandwidth buys you tokens-per-second. When a vendor quotes one big multiplier for "AI performance", check which of the two it refers to — it is almost always the prefill number, which is the less interesting one for interactive chat.

Which Mac should I buy for local LLMs?

The October 2026 lineup is unusually messy: M5, M5 Pro, M5 Max and M5 Ultra ship alongside an M6 that exists in exactly one machine, plus an M4 that survives only in the iMac. The Mac Pro has been discontinued. Bandwidth and memory ceilings per chip:

Three traps in that table worth more than the rest of this section:

• M6's bandwidth depends on how much memory you order. The 16 GB Mac mini runs at 153 GB/s; only the 24 GB and 32 GB configurations get the 170 GB/s that Apple markets. The cheapest M6 is not the fast M6.
• "M5 Max" is two different chips for our purposes. The 32-core-GPU version does 460 GB/s; the 40-core version does 614 GB/s. That is a 33% difference in token generation rate under one name. The 128 GB memory option also requires the 40-core part.
• There is no 192 GB Mac. M5 Ultra steps 96 → 256 → 512 GB with nothing in between, so the jump from "big" to "very big" is a price cliff, not a ramp. 128 GB means an M5 Max, not an Ultra.

Note also that the 512 GB ceiling has not moved — it was the maximum on M3 Ultra and it is still the maximum on M5 Ultra. If your model needs more than that, you are clustering or renting.

What is MLX, and what is new in 0.32?

MLX is an array framework for Apple Silicon built and open-sourced by Apple — NumPy-like arrays plus autograd plus a Metal backend, with Python, Swift, C++ and Rust bindings. mlx-lm is the LLM layer on top: loading, generation, serving, quantization and fine-tuning.

The 0.32 line is maintenance-and-performance rather than headline features: the September 0.32.3 release landed 76 changes, including a force_fused option for scaled dot-product attention (added in 0.32.2), quantized matrix-multiplication fixes, and GPU stream deadlock fixes. The more interesting development is in mlx-lm, which now installs a genuine quantization toolkit as console commands: mlx_lm.awq, mlx_lm.gptq, mlx_lm.dwq and mlx_lm.dynamic_quant, alongside mlx_lm.benchmark, mlx_lm.evaluate and mlx_lm.perplexity.

DWQ is worth calling out because it is unusual. It is distillation-based quantization: you pass a full-precision teacher with --model and an already-quantized student with --quantized-model, and it trains the student — learning rate, batch size, dataset, optional gradient checkpointing — to recover quality the naive quantizer lost. That is a meaningful answer to "4-bit degraded my model" that did not exist on the Mac a year ago.

MLX also has CUDA and NCCL backends, so the framework is not strictly Apple-only any more, though on NVIDIA hardware you would reach for vLLM or TensorRT-LLM instead. Apple's developer site now describes MLX as supporting "Metal 4 and GPU Neural Accelerators" and being able to "scale training across multiple Macs with RDMA over Thunderbolt" — the framework is officially positioned for multi-machine work, not just single-Mac experiments.

What about Apple's own frameworks — Core AI and Foundation Models?

Two things changed here in 2026 and both are easy to miss if you only follow the open-source side.

Core AI is a new Apple framework for on-device inference, described on Apple's developer site as "a modern, memory-safe Swift API to load and run AI models entirely on device with zero server dependencies and zero token costs," with models "automatically specialized for the hardware they run on" and ahead-of-time compilation for fast loads. It ships alongside Core AI Models (converted community models), PyTorch extensions for converting your own, a quantization and palettization toolkit, and a Core AI Debugger app with Xcode integration. Apple lists it first among its AI developer frameworks in the M6 announcement, ahead of Core ML.

The Foundation Models framework is no longer Apple-models-only. Apple now documents it as supporting "multimodal prompts and any language model — including Apple Foundation Models or any provider that conforms to the Language Model protocol." That turns it from a fixed on-device model into a pluggable abstraction, which is a meaningful shift if you ship a Mac or iOS app and want one API over both Apple's model and your own local weights.

Apple also stated in June 2026 that the current generation of Apple Foundation Models is "custom-built in collaboration with Google and its Gemini models," running on device and on Private Cloud Compute. Apple has not published parameter counts, so treat any specific size you see quoted as unverified.

Which to use: Core AI and Foundation Models are for shipping an app to users who should not have to install anything. MLX is for running models yourself, with control over weights, quantization and serving. Most readers of this guide want MLX.

Does Ollama actually use MLX on Apple Silicon now?

This deserves a precise answer, because the half-true version is everywhere — including in the earlier version of this guide.

Ollama's MLX engine arrived in v0.19.0 on 27 March 2026 as a preview. Ollama's own announcement scoped it tightly: the preview accelerated the Qwen3.5-35B-A3B architecture and required a Mac with more than 32 GB of unified memory. Through v0.33 (August) and v0.35.x (late September) the MLX engine was maintained and expanded but was not the default path.

MLX becomes the default only in v0.40.0, whose release notes state that on Apple Silicon, architectures supported by the MLX runtime will automatically run on MLX. As of 5 October 2026 that release is still at rc3; the newest stable build is v0.35.1.

So: if you are on stable Ollama today, you are not automatically on MLX for every model. If you want guaranteed MLX execution right now, run mlx-lm directly, use LM Studio's MLX backend, or opt into the 0.40 release candidate. Our Ollama vs LM Studio vs vLLM vs llama.cpp vs MLX comparison goes deeper on the trade-offs.

Should I use MLX-LM, Ollama, LM Studio, llama.cpp or Splash?

How fast is a Mac at local inference, really?

Throughput claims in this space are mostly unsourced. The best data available is Apple's own, because the mlx-lm repository ships a benchmarks file that reports prefill and decode separately and pairs each quantization with an MMLU Pro score — so you can see the speed gain and the quality cost in one table. These runs are Qwen3-4B-Instruct-2507 on a 64 GB M4 Max, via mlx_lm.benchmark -p 2048 -g 128:

That table is the single most useful thing in this guide, for two reasons. Decode speed rises 2.6× from bf16 to 4-bit while prefill stays flat — direct confirmation that generation is bandwidth-bound and prompt processing is not. And the quality cost of 4-bit is real but modest: 3.3 MMLU Pro points, about 5%. The 6-bit row is the quiet winner, giving up half a point for double the speed and a 53% memory cut.

Two other sourced datasets worth knowing:

• Ollama, March 2026, M5 with Qwen3.5-35B-A3B: its MLX path reported 1,810 tok/s prefill and 112 tok/s decode, versus 1,154 and 58 on the previous llama.cpp-based build.
• LM Studio, September 2026, Splash engine on a 48 GB M5 Pro with Qwen3.8-27B: 74 tok/s on short prompts, 54 tok/s at 32K context, 170 tok/s aggregate across four concurrent requests. Benchmarks are attributed to Inco AI and the quantization is not stated.
• llama.cpp's hardware thread collects maintainer-run numbers on Llama-2-7B Q4_0: roughly 120 tok/s decode on M5 Max (40-core), 179 on M5 Ultra, and 36 on M6. Treat cross-generation comparisons there with care — older rows were measured on 2023 builds, and on unchanged M5 Max hardware prefill tripled from software improvements alone.

The practical takeaways: a current Mac generates in the 50–120 tokens/sec band on a well-matched model, long contexts cost roughly 25–30% of short-prompt speed, and concurrency scales aggregate throughput well even though single-stream speed does not improve. A tokens/sec figure quoted without a chip, model, quantization and context length tells you nothing.

Which quantization should I use on a Mac — MLX or GGUF?

MLX quantization is the default choice. Conversions are one command (mlx_lm.convert --model <hf-repo> -q), they run fastest on Apple Silicon, and mlx-community has already done it for most models you would want.

On which bit-width: Apple's own benchmark table above is the best guide available, and it argues for 6-bit more than the community's 4-bit default — 6-bit cost half an MMLU Pro point versus bf16 while roughly doubling decode speed. Use 4-bit when memory is the binding constraint, 6-bit when you have headroom, 8-bit for small models where every point matters. Below 4-bit, expect real degradation rather than a rounding error.

GGUF remains genuinely useful on a Mac for two reasons: brand-new architectures land there first, and its quantization ladder is far more granular at the bottom end. The GLM-5.3-Flash GGUF release spans twelve rungs from BF16 at 641.6 GB down to IQ1_S at 93.1 GB. Nothing in the MLX ecosystem gives you that many options below 4-bit, and when you are trying to squeeze a 321B model into 128 GB, those rungs are the whole game.

One correction to advice you will still find online, including in the previous version of this article: AWQ and GPTQ are no longer off-limits on Apple Silicon. mlx-lm ships both as first-class CLI commands, along with DWQ and dynamic quantization. EXL2 remains NVIDIA-only.

Practical rule: use the mlx-community 4-bit build if one exists, 8-bit if the model is small or the task is precision-sensitive, and drop to GGUF when you need either a model that MLX has not caught up with or a quantization rung below 4-bit.

Which models should I download right now?

Download counts in mlx-community are a decent proxy for what actually works, and they point at a short list as of October 2026.

• Best all-rounder, 32 GB and up: Qwen3.8-27B. The 4-bit build is 16.1 GB and the most-downloaded recent LLM conversion in mlx-community by a wide margin — it is also the model LM Studio chose to showcase its fastest Apple Silicon engine. Walkthrough: how to run Qwen 3.8 locally; family breakdown in our Qwen 3.8 lineup guide.
• Coding, 24 GB: Devstral-Small-2 24B 4-bit (15.1 GB), or Qwen3-Coder-30B-A3B (17.2 GB) if you want MoE speed instead of dense quality.
• 16 GB: Qwen3.5-9B 4-bit (6.0 GB), or Gemma 4 12B QAT 4-bit (11.0 GB).
• 8 GB: Qwen3.5-4B 4-bit (3.1 GB) or Gemma 4 E2B 4-bit (3.6 GB). More in our roundup of the best small LLMs to run locally.
• 128 GB and up: GLM-5.3-Flash — see the next section.

For the wider picture beyond what fits on a Mac, see our open-source LLMs landscape pillar.

Can I run a 321B model like GLM-5.3-Flash on a Mac?

Yes, if you have a lot of memory and accept a real quality trade-off. This model is a good worked example because it is the largest thing people currently try to run on Apple Silicon, and the numbers are unusually clear.

GLM-5.3-Flash is a mixture-of-experts model published by Z.ai under the MIT licence in August 2026. Its configuration reports 321 billion total parameters across 45 layers with 288 routed experts, 8 of which activate per token, plus one shared expert — roughly 18B active. It ships natively in FP8 (E4M3), is multimodal, and declares a context window of 1,048,576 tokens. The source repository is 328.4 GB.

Note that "4-bit" is not one download. The mlx-community build is 204.0 GB because it keeps the vision tower and embeddings at BF16 and lifts many projection layers to 5–6 bits; pipenetwork's is 177.6 GB. A 26 GB spread under the same label is enough to decide whether a machine can load it, so check the actual repo size rather than trusting the name.

So 128 GB is the practical floor, and only via a 1–2-bit GGUF or a 2-bit MLX build. Unsloth, which publishes the GGUF ladder, states plainly that its smallest 1-bit quant needs 100 GB of combined memory and that 3-bit wants a 128 GB device. Be honest about what that buys: sub-2-bit quantization on a model this size degrades instruction-following and long-chain reasoning in ways benchmarks under-report. If you want GLM-5.3-Flash behaving like GLM-5.3-Flash, 256 GB for a 4-bit build is the real entry point.

One practical trap: GLM-5.3-Flash is multimodal, and mlx-lm cannot load it. The glm5_next architecture has no implementation in mlx-lm 0.32.0 — you need mlx-vlm (0.7.4, released 28 September 2026) or llama.cpp instead. The same applies to the Qwen 3.8 family. If a model card says image-text-to-text, reach for mlx-vlm first. Step-by-step: how to run GLM-5.3-Flash locally and the GLM-5.3-Flash complete guide.

For most people the correct conclusion is to run Qwen3.8-27B on 32 GB instead. It costs an order of magnitude less and is not an order of magnitude worse.

Can I cluster several Macs together?

Yes, and MLX supports it officially. The distributed documentation lists four backends: MPI, Ring (all-reduce and all-gather over TCP, always available), NCCL for CUDA, and JACCL, which does RDMA over Thunderbolt and achieves, in Apple's words, communication latency an order of magnitude lower than the Ring backend.

The requirements are strict and worth knowing before you buy cables. JACCL needs macOS 26.2 or later — so any machine on the current macOS 27, released 14 September 2026, qualifies — plus Thunderbolt 5 and a fully connected mesh, meaning every node directly cabled to every other node with an isolated local network per link. RDMA must also be enabled manually from macOS recovery mode. The mesh requirement is what caps practical clusters at a handful of machines: node count drives cable count quadratically.

Apple's documentation does not publish specific bandwidth or latency figures for JACCL, so treat any precise Gbps or microsecond number you see as unverified. On the community side, EXO remains the most popular wrapper for distributing a model across local machines.

Clustering makes sense when data sovereignty is a hard requirement and the hardware amortises against a real API bill. For a solo developer it does not — rent a GPU. The cloud and Kubernetes path is covered in our self-hosting LLMs guide.

Can I fine-tune an LLM on a Mac?

Yes, and this is one of the Mac's genuinely strong suits, because fine-tuning is memory-hungry and unified memory is cheap compared with VRAM. mlx-lm supports LoRA (the default), QLoRA for quantized bases, DoRA, and --fine-tune-type full.

The documented command is straightforward:

mlx_lm.lora --model <path_or_hf_repo> --train --data <data_dir> --iters 600

When you run out of memory, Apple's documentation gives five levers in rough order of usefulness: quantize the base model, reduce --batch-size from its default of 4, reduce --num-layers from its default of 16, split long examples into shorter ones, and add --grad-checkpoint to trade compute for memory. The documented 32 GB configuration is --batch-size 1 --num-layers 4, and an M1 Max with 32 GB reached roughly 250 tokens/sec of training throughput.

The honest limit: an NVIDIA box still trains faster on anything that fits in its VRAM. The Mac wins when the model does not fit at all. Recipes and hyperparameters are in our fine-tuning LLMs guide, and if you are considering pre-training from scratch, read our guide to self-training a small LLM first — the answer is usually "rent a GPU".

How do I get started in five minutes?

Path A — mlx-lm, for guaranteed MLX execution.

pip install mlx-lm
mlx_lm.generate --model mlx-community/Qwen3.8-27B-4bit --prompt "Explain unified memory"
mlx_lm.server --model mlx-community/Qwen3.8-27B-4bit

mlx_lm.chat gives you a REPL, and mlx_lm.cache_prompt pre-computes a long prompt so repeated queries against the same context skip prefill — genuinely useful for document and codebase work.

Path B — Ollama, for the simplest install. Install it, ollama pull a model, ollama run it, and point any OpenAI-compatible client at the local endpoint. Check the version if runtime matters to you, per the Ollama section above.

Path C — LM Studio, for a GUI. Install, search the model catalogue, filter to MLX builds, download, load, chat. It also exposes a local OpenAI-compatible server and lets you select alternative engines such as Splash. A broader tool survey is in our roundup of the best free local LLM tools.

So what is the actual recommendation?

A decision rule rather than a sales pitch:

• Already own a 16 GB Mac? Don't upgrade yet. Run Qwen3.5-9B at 4-bit and find out whether local inference is genuinely part of your workflow before spending anything.
• Cheapest serious entry point: a Mac mini. The M6 at 32 GB gets you 170 GB/s and comfortably runs Qwen3.8-27B; the M5 Pro at 64 GB roughly doubles bandwidth to 307 GB/s. This is the best value in the lineup for local inference by a clear margin.
• Buying a laptop for this? 32 GB is the real entry point, 64 GB is the comfortable one. Going 16 → 32 GB unlocks Qwen3.8-27B, which is a far bigger capability jump than 32 → 64.
• Want to run 100B+ models? 128 GB on an M5 Max to start, 256 GB on an M5 Ultra to do it honestly. Price that against a year of API calls first; for most people the API wins.
• Serving real users? Don't use a Mac. Concurrency is where Apple Silicon stops being the cheap option — see the cheapest way to run a local LLM.

If you remember one thing: buy unified memory first, because it decides whether a model runs at all, and bandwidth only decides how fast it runs once it fits.

FAQ

Is MLX faster than llama.cpp on Apple Silicon?

Generally yes, and Ollama's own March 2026 measurements showed its MLX path roughly 1.9× faster on decode and 1.6× faster on prefill than its previous llama.cpp-based build on M5. But llama.cpp remains the better choice when a model has no MLX conversion yet or when you need a quantization rung MLX does not offer, which in practice means anything below 4-bit.

How much RAM do I need to run an LLM on a Mac?

8 GB runs a 3–4B model usefully for narrow tasks. 16 GB runs a 9–12B general assistant. 32 GB runs Qwen3.8-27B at 4-bit, which is the current quality-per-gigabyte sweet spot. 64 GB adds full precision or long contexts. Beyond that you are buying the ability to run frontier MoE models, which starts at 128 GB.

What is the current version of MLX?

MLX is at 0.32.3, released 29 September 2026, and mlx-lm is at 0.32.0, published 1 October 2026. Both release roughly monthly. Upgrade with pip install --upgrade mlx mlx-lm.

Does Ollama use MLX on Apple Silicon?

Partly. MLX shipped as a preview in Ollama 0.19.0 on 27 March 2026 and becomes the automatic default for supported architectures in 0.40.0, which was still a release candidate as of early October 2026. On the current stable 0.35.1 it is not the universal default, so if guaranteed MLX execution matters, use mlx-lm or LM Studio's MLX backend.

What is the difference between MLX 4-bit and GGUF Q4 on a Mac?

Both store weights at roughly four bits each. MLX's format is tuned for Apple's GPU and generally runs faster; GGUF is portable and offers a much finer ladder of low-bit options. For any mainstream model, use the MLX build. Reach for GGUF when the model is too new for MLX or when you need to go below 4-bit.

Can I run GLM-5.3-Flash on a Mac?

Only with 128 GB of unified memory or more, and at that floor only via a 1–2-bit GGUF (93.1–108.7 GB) or a 2-bit MLX build (111.4 GB), all of which degrade the model noticeably. A 4-bit MLX build is 177.6–204 GB depending on publisher and wants a 256 GB machine. Note also that it is multimodal, so you need mlx-vlm or llama.cpp rather than mlx-lm.

Why won't mlx-lm load my model?

The most common cause is that it is a vision-language model. mlx-lm only implements text architectures, so multimodal models such as GLM-5.3-Flash and the Qwen 3.8 family need mlx-vlm instead. If a Hugging Face model card shows the image-text-to-text pipeline tag, use mlx-vlm or llama.cpp. The second most common cause is that the architecture is too new and no MLX implementation exists yet, in which case llama.cpp is usually weeks ahead.

Can I use AWQ or GPTQ quantization on Apple Silicon?

Yes, as of the current mlx-lm, which ships mlx_lm.awq and mlx_lm.gptq as CLI commands alongside mlx_lm.dwq and mlx_lm.dynamic_quant. Older advice that these are NVIDIA-only is out of date. EXL2 is still NVIDIA-only.

How many Macs can I cluster, and what do I need?

Practically a handful. MLX's JACCL backend needs macOS 26.2 or later, Thunderbolt 5, RDMA enabled from recovery mode, and a fully connected mesh in which every machine is directly cabled to every other one. That topology requirement is what limits cluster size, since cable count grows quadratically with nodes.

Can I run Claude or GPT locally on my Mac?

No. Neither Anthropic nor OpenAI publishes weights, so those models are API-only. The strongest open-weight alternatives you can actually run are Qwen3.8-27B for anything that fits in 32 GB and GLM-5.3-Flash if you have 128 GB or more.

Which Mac gives the most tokens per second per dollar?

A Mac mini. The M6 at 32 GB gives you 170 GB/s of bandwidth from $899, and the M5 Pro configuration roughly doubles that to 307 GB/s with up to 64 GB. Laptops cost substantially more for the same memory and bandwidth because you are also paying for the display and battery. If the machine will sit on a desk, buy the mini.

Is a Mac better than a PC with a discrete GPU for local LLMs?

It depends entirely on model size. A discrete GPU has far more bandwidth than any Mac but a hard VRAM ceiling, so it wins decisively on models that fit in its memory and cannot run the ones that do not. A Mac's unified memory means a 128 GB machine loads a model no consumer GPU can hold at all. Buy the GPU for speed on small and mid-size models; buy the Mac for capacity, portability and idle power draw.

Is a Mac good enough to serve an application?

For internal tools, prototypes and single-digit concurrent users, yes. Beyond that, no — throughput per dollar turns against you quickly, and you lose the ability to scale horizontally. Keep the Mac for development and privacy-sensitive work, and read our self-hosting guide for the production path.

Related guides

• Ollama vs LM Studio vs vLLM vs llama.cpp vs MLX — picking a runtime
• How to run Qwen 3.8 locally — the best current Mac model, step by step
• How to run GLM-5.3-Flash locally — the 321B option
• Best small LLMs to run locally — for 8–16 GB Macs
• LM Studio complete guide — the GUI path
• Self-hosting LLMs — the cloud and Kubernetes side
• Fine-tuning LLMs — LoRA, QLoRA and DoRA recipes
• Open-source LLMs landscape — the full model family picture