How to Run GLM-5.2 Locally — and When to Pick 5.3-Flash

Should you run GLM-5.2 or GLM-5.3-Flash?

This is now the first question, and for most people the answer is not GLM-5.2.

On 25 August 2026 Z.ai published GLM-5.3-Flash to Hugging Face under the MIT licence. Its config.json shows a 45-layer MoE with 288 routed experts, 1 shared expert and 8 experts active per token, shipped natively in FP8 — 320B total parameters with 18B active. GLM-5.2, by contrast, is a 78-layer MoE with 256 routed experts and 8 active, quoted by Z.ai at ~744B total / 40B active (vLLM's recipe lists 743B/39B; the BF16 checkpoint's tensors sum to 753.3B).

That difference is not incremental. It is the difference between a model that needs a rack and a model that needs a workstation.

The plain recommendation: if you are self-hosting on hardware you own, run GLM-5.3-Flash. Unsloth's own documentation for Flash notes it "outperforms GLM-5.2 across benchmarks," citing DeepSWE at 63.4 versus 46.2 and Toolathlon at 78.4 versus 59.9 (Unsloth, August 2026). It is a better model on the tasks most people self-host for, it is the same MIT licence, and it needs roughly a third of the memory. We have written the setup up separately: how to run GLM-5.3-Flash locally, with model background in the GLM-5.3-Flash complete guide.

Three cases still justify GLM-5.2 specifically:

• You already have an 8x H200 node. GLM-5.2's FP8 path has had two and a half months of production hardening, a stable vLLM release (0.23.0), and a published AMD MXFP4 checkpoint. Flash still requires a vLLM nightly and a dedicated Docker image.
• You need the full 1M-token window with documented behaviour at that length. GLM-5.2 is evaluated at 1M; Flash's card documents evaluation to 300K.
• You are fine-tuning. If you already have a GLM-5.2 training pipeline and adapters, the switching cost is real.

Note that GLM-5.3 — the full-size model, not Flash — also has weights now, but it is not MIT: Hugging Face lists it under a bespoke "GLM-5.3 licence". It shares GLM-5.2's architecture and 743B/39B shape, so every hardware number on this page transfers to it directly. See our GLM-5.3 launch guide for the model-level differences.

What is the minimum hardware to run GLM-5.2 at all?

The honest floor, stated three ways:

• Absolute minimum to get tokens out: ~217 GB of storage for the 1-bit UD-IQ1_S GGUF, plus enough combined VRAM + system RAM to hold it. In practice that means a 256 GB RAM machine, with or without a GPU. A single 24 GB GPU helps prompt processing but does not change the memory requirement.
• Minimum for a genuinely pleasant single-user setup: a 256 GB unified-memory Mac Studio, or a workstation with 256–384 GB DDR5 and one 24 GB+ GPU for attention layers.
• Minimum for GPU-resident serving: 893 GB of VRAM for the FP8 checkpoint, which in practice means 8x H200 or 8x H20 at 141 GB each (vLLM recipes, updated July 2026). On Blackwell, the NVIDIA NVFP4 checkpoint drops that to 558 GB; on AMD MI355X the Quark MXFP4 checkpoint drops it to 446 GB.

There is no configuration in which GLM-5.2 runs on a single consumer GPU. A 24 GB card holds about 3% of the smallest usable quantisation. Everything else has to live in system RAM or on other GPUs, and the memory bandwidth of that path is what sets your token rate.

How much memory does each GLM-5.2 quantisation need?

The table below lists on-disk sizes measured directly from unsloth/GLM-5.2-GGUF on 31 August 2026 by summing the actual shard byte counts, alongside the memory figures Unsloth publishes in its GLM-5.2 guide. Where the two disagree, trust the measured column for your download budget and the Unsloth column for your RAM budget — running the model needs headroom above the file size for KV cache and activations.

One correction worth flagging: earlier versions of this guide, and several posts still circulating, cite UD-IQ2_M as a 239 GB download. The current repo revision is 288.6 GB across six shards. Unsloth re-uploaded the dynamic quants on 23 June 2026 and the older figure no longer matches what you will actually pull. Budget disk accordingly.

Unsloth characterises the 1-bit quant as retaining roughly 76% top-1 accuracy at 86% smaller, and the 2-bit dynamic quant at roughly 82% accuracy while 84% smaller. The 4-bit and 5-bit dynamic variants are described as generally lossless against the BF16 baseline.

Which GLM-5.2 file should you actually download?

Three decisions cover almost everyone:

• 256 GB Mac Studio or 256 GB RAM workstation → UD-IQ1_S (216.7 GB). The 2-bit quant no longer fits comfortably at this tier now that it is 288.6 GB.
• 384 GB+ RAM → UD-IQ2_M (288.6 GB) or UD-IQ3_XXS (331.7 GB). This is the sweet spot for local quality.
• 8x H200 node → skip GGUF entirely and serve zai-org/GLM-5.2-FP8 under vLLM. GGUF on datacentre GPUs throws away MTP speculative decoding and continuous batching.

How do you serve GLM-5.2 with vLLM or SGLang?

This is the production path. GLM-5.2 requires vLLM 0.23.0 or newer, or SGLang v0.5.13.post1 or newer. The FP8 checkpoint is the practical default — it fits a single 8x H200 node, where BF16 does not.

uv venv && source .venv/bin/activate
uv pip install "vllm==0.23.0" --torch-backend=auto
uv pip install "transformers>=5.9.0"

vllm serve zai-org/GLM-5.2-FP8 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.2-fp8

Flag notes, all from the official vLLM recipe:

• --tool-call-parser glm47 and --reasoning-parser glm45 are the correct parsers for GLM-5.2 despite the version numbers looking wrong. Omitting them gives you tool calls emitted as raw text.
• --speculative-config with MTP is the headline GLM-5.2 change: Multi-Token Prediction was extended from 3 to 5 draft tokens versus GLM-5/5.1. It is opt-in, and it is where most of your throughput comes from.
• --kv-cache-dtype fp8 on H100/H200. Blackwell keeps fp8_e4m3; H20 should not use FP8 KV cache at all.
• FP8 requires DeepGEMM. Install it with vLLM's install_deepgemm.sh or you will get correct output at badly degraded speed.
• Thinking is on by default. Disable per request with "chat_template_kwargs": {"enable_thinking": false}, or dial it with {"reasoning_effort": "high"}.

If you are on 8x H20 and want maximum single-stream speed, the recipe documents a DSpark speculative-decoding configuration using an externally trained 7-token draft model, reporting 70+ tokens/sec for a single request:

vllm serve zai-org/GLM-5.2-FP8 \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 --reasoning-parser glm45 --enable-auto-tool-choice \
  --max-model-len 200000 --max-num-seqs 16 --max-num-batched-tokens 32768 \
  --speculative-config '{"model": "RedHatAI/GLM-5.2-speculator.dspark","num_speculative_tokens": 7,"method": "dspark"}' \
  --served-model-name glm-5.2-fp8

The equivalent SGLang launch is straightforward:

python3 -m sglang.launch_server \
  --model-path zai-org/GLM-5.2-FP8 \
  --tp 8 --host 0.0.0.0 --port 30000

Reaching the full 1M-token context needs 8x B200 at 180 GB each with FP8 KV cache. On 8x H200 you should plan on capping --max-model-len around 131072–200000 and treating the 1M figure as a headline rather than a default.

How do you run GLM-5.2 on a RAM-heavy box with llama.cpp?

This is the path for anyone without datacentre GPUs. The MoE architecture is what makes it viable: only ~40B of ~744B parameters are active per token, so the bottleneck is memory bandwidth rather than compute.

# 1. Build llama.cpp (CUDA; use -DGGML_METAL=ON on Apple Silicon)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j $(nproc)

# 2. Pull the 1-bit dynamic GGUF (~217 GB)
pip install "huggingface_hub[cli]"
hf download unsloth/GLM-5.2-GGUF \
  --include "UD-IQ1_S/*" \
  --local-dir ~/models/glm-5-2

# 3. Serve, keeping all MoE expert weights on the CPU
./build/bin/llama-server \
  --model ~/models/glm-5-2/UD-IQ1_S/GLM-5.2-UD-IQ1_S-00001-of-00006.gguf \
  --ctx-size 32768 \
  --n-gpu-layers 999 \
  --cpu-moe \
  --cache-type-k q4_1 --cache-type-v q4_1 \
  --temp 1.0 --top-p 0.95 --min-p 0.01 \
  --jinja \
  --threads $(nproc) \
  --host 0.0.0.0 --port 8080

Two things here have changed since most GLM-5.2 tutorials were written. First, --cpu-moe (short form -cmoe) is now a first-class llama.cpp flag meaning "keep all Mixture-of-Experts weights in the CPU" — it replaces the hand-written --ot "exps=CPU" regex, and --n-cpu-moe N (-ncmoe) lets you offload only the first N layers' experts if you have VRAM to spare. Second, --cache-type-k q4_1 --cache-type-v q4_1 quantises the KV cache, which is what buys you meaningful context length on a memory-constrained box.

Unsloth's recommended sampling settings for GLM-5.2 are temperature 1.0, top_p 0.95, min_p 0.01, with top_p 1.0 for SWE-Bench Pro-style coding evaluation. Maximum generation length is 163,840 tokens.

Verify the server:

curl -s http://localhost:8080/v1/models | jq .

What throughput should you expect from GLM-5.2?

Be realistic about this, because it is the single biggest reason people abandon a local GLM-5.2 setup after a week.

• 8x H200, FP8, vLLM with MTP: the reference regime. The vLLM recipe's 8k-in/1k-out benchmark on 8x H200 measured 526 output tokens/sec aggregate at 32 concurrent requests, with median time-to-first-token around 14.5 seconds at that load. The recipe notes synthetic benchmarks under-report real speed because MTP acceptance is low on random data.
• 8x H20 with DSpark drafting: 70+ tokens/sec single-stream, per the same recipe.
• Mac Studio or CPU/RAM box with GGUF: single-digit tokens/sec. This is usable for solo asynchronous work — fire a task, come back — and not usable for interactive pair-programming or for more than one person.
• Prompt processing on the RAM path is the real pain point. A 64k-token prompt can take minutes to ingest before the first token appears.

One caveat specific to llama.cpp right now: the DSA lightning-indexer implementation merged in PR #25407 made long-context output correct, but its author measured it as slower than the previous indexer-ignoring build — prefill 14% to 24.5% worse as sequences deepen, decode about 6.5% worse — because indexers are computed on every "full" layer even when they are no-ops below 2,048 tokens. A dedicated fused kernel is the identified fix. If you are on short prompts, you are paying for accuracy you are not using.

Is GLM-5.2 available on Ollama or LM Studio?

Partially, and the distinction matters. Ollama's library lists GLM-5.2 with exactly one tag: glm-5.2:cloud. That routes your request to a hosted endpoint — it does not download weights and does not run on your machine. There is no local Ollama tag for GLM-5.2, which is unsurprising given the smallest usable quant is 217 GB.

LM Studio can load the Unsloth GGUFs directly, since it wraps llama.cpp, and it is the easiest GUI route on a large-memory Mac. Throughput will match an equivalently configured llama.cpp build. If you specifically want a one-command ollama pull experience with local weights, GLM-5.3-Flash at ~119 GB is a far more realistic target than GLM-5.2.

Why does GLM-5.2 fail to load or run slowly?

Out of memory at load. Drop a quant level, lower --ctx-size to 16384, and quantise the KV cache with --cache-type-k q4_1 --cache-type-v q4_1. NVMe swap works as a stopgap but costs 30–50% throughput.

Generation below 2 tokens/sec. Check nvtop. If GPU utilisation sits near zero during generation, expert traffic is saturating PCIe. Confirm your offload actually applied by grepping the load log for tensors assigned to CPU, and make sure the model file is on NVMe rather than SATA.

Tool calls arriving as plain text under vLLM. You omitted --tool-call-parser glm47 --enable-auto-tool-choice. Same for reasoning traces without --reasoning-parser glm45.

FP8 serving is slow on H100/H200. DeepGEMM is not installed. Run vLLM's install_deepgemm.sh.

Poor MTP acceptance rate. A known issue fixed in vLLM upstream; update your branch or use the vllm/vllm-openai:glm52 Docker image.

Kernel panic or stall on Mac Studio. macOS pages aggressively past ~85% of unified memory. Wire more for the GPU before launching: sudo sysctl iogpu.wired_limit_mb=$((220*1024)). This resets on reboot.

Is running GLM-5.2 locally worth it versus the API?

Run the arithmetic before you buy hardware, because it has moved against self-hosting.

Z.ai charges $1.40 per million input tokens, $0.26 cached, and $4.40 per million output for GLM-5.2 (Z.ai pricing). OpenRouter currently routes it across 27 providers at $0.4875 input and $1.56 output — roughly a third of Z.ai's direct rate, at the full 1,048,576-token context.

At OpenRouter pricing, a heavy individual burning 20M input and 4M output tokens a month pays about $16. A 256 GB Mac Studio costs several thousand dollars and gives you single-digit tokens/sec. The break-even is not measured in months; for a single developer it does not arrive.

And GLM-5.3-Flash's API pricing makes the gap starker: $0.15 input and $0.50 output per million, currently halved to $0.075 and $0.25 under a promotion running to 9 September 2026. That is roughly a ninth of GLM-5.2's direct rate for a model that benchmarks better on coding.

Self-hosting GLM-5.2 is justified by data residency (prompts and code never leave your network), fine-tuning (MIT permits commercial derivatives, and no API gives you weights), or capacity you already own (an idle 8x H200 node changes every number above). It is not justified by cost alone at individual scale. For the general framing on when self-hosting pays, see our self-hosting LLMs guide.

FAQ

What hardware do you need to run GLM-5.2 locally?

At minimum, ~217 GB of storage plus ~256 GB of combined VRAM and system RAM for the 1-bit UD-IQ1_S GGUF. A 256 GB unified-memory Mac Studio or a 256–384 GB RAM workstation with one 24 GB GPU are the two realistic consumer configurations. For GPU-resident serving of the FP8 checkpoint, vLLM specifies 893 GB of VRAM, which means an 8x H200 or 8x H20 node.

How much VRAM does GLM-5.2 need?

It depends entirely on the path. Pure GPU serving needs 893 GB for FP8, 558 GB for NVIDIA's NVFP4 checkpoint on Blackwell, 446 GB for AMD's MXFP4 on MI355X, or 1,786 GB for BF16 across multiple nodes. On the llama.cpp offload path, VRAM matters far less than total RAM — a single 24 GB card holding attention layers while ~217 GB of experts sit in system memory is a working configuration.

Can I run GLM-5.2 on a single GPU?

Not in the sense most people mean. No single GPU exists with enough VRAM. What does work is a single 24 GB card handling attention and dense layers while every Mixture-of-Experts weight is offloaded to system RAM via llama.cpp's --cpu-moe flag, on a machine with 256 GB or more. Expect 1–3 tokens per second in that configuration.

Should I run GLM-5.2 or GLM-5.3-Flash?

For most self-hosters, GLM-5.3-Flash. It is 320B total with 18B active versus GLM-5.2's ~744B/40B, ships under the same MIT licence, fits a 128 GB machine at 3-bit where GLM-5.2 fits nothing under 217 GB, and Unsloth reports it outperforming GLM-5.2 across benchmarks. Stay on GLM-5.2 if you already own an 8x H200 node, need documented 1M-context behaviour, or have an existing fine-tuning pipeline.

Is GLM-5.2 on Ollama?

Only as glm-5.2:cloud, which proxies to a hosted endpoint rather than downloading weights. There is no local Ollama tag, because the smallest usable quantisation is 217 GB. LM Studio can load the Unsloth GGUFs locally since it wraps llama.cpp, and that is the practical GUI route on a large-memory Mac.

Is running GLM-5.2 locally worth it?

Rarely on cost. OpenRouter serves GLM-5.2 at $0.4875 per million input and $1.56 per million output tokens, so heavy individual use runs to tens of dollars a month against several thousand for hardware that delivers single-digit tokens per second. Self-hosting earns its keep on data residency, fine-tuning rights under the MIT licence, or GPU capacity you already own — not on token economics.

Does GLM-5.2's sparse attention work in llama.cpp?

Yes, as of PR #25407 merged on 24 July 2026, which implemented the DSA lightning indexer including the per-layer full/shared top-k distinction and GLM's interleaved RoPE rotation. Long-context output is now correct. The trade-off is speed: the PR author measured prefill 14–24.5% slower and decode ~6.5% slower than the previous implementation, pending a fused kernel.

What should you do next?

The decision rule is short. If you have an 8x H200 node sitting idle, serve zai-org/GLM-5.2-FP8 under vLLM 0.23.0 with MTP enabled and you will get genuinely good throughput. If you have a workstation, a Mac Studio, or anything smaller, run GLM-5.3-Flash instead — it is a better model on coding and agentic work, it is the same MIT licence, and it is the difference between "fits" and "does not fit". If you have neither and just want the model, OpenRouter at $0.4875 per million input tokens will cost less than the electricity for a local rig.

For deeper background: the GLM-5.2 complete guide covers architecture and benchmarks, the GLM-5.3-Flash guide covers the newer model, and the open-source LLMs landscape puts both in context against DeepSeek V4 and Kimi K2.6. If you are wiring a local model into a real agent loop, our AI coding agents guide covers which harnesses work with GLM's Anthropic-compatible endpoint.

Standing up the inference server is the easy half. The harder half is everything around it — retrieval, evaluation harnesses, and agent loops that do not run away with a 1M-token context window. If you need people who have shipped that on production systems, Codersera matches you with vetted remote engineers on the stack you are actually running.