Quick answer. To run 70B and 120B models on a Ryzen AI Max+ 395, allocate memory to the iGPU (96 GB via AMD Adrenalin on Windows, or a GTT kernel parameter plus a small BIOS framebuffer on Linux), then run LM Studio, Ollama, or llama.cpp on the Vulkan backend. Expect ~100 t/s on 30B MoE models and single-digit t/s on dense 70B.
The AMD Ryzen AI Max+ 395 (codename Strix Halo) is unusual: a single chip pairs 16 Zen 5 cores with a 40-compute-unit Radeon 8060S iGPU and up to 128 GB of LPDDR5X unified memory that the CPU and GPU share. That capacity is the whole story for local LLMs, because it lets you load models that physically do not fit on any single consumer discrete GPU. This is the hands-on setup the spec-sheet comparisons skip: how to actually hand ~100 GB to the iGPU, which backend to pick, which quants fit, and the honest speed you should expect.
What makes Strix Halo different for local LLMs?
Most inference boxes force a trade-off between capacity and speed. A 24 GB RTX 4090 is fast but can't hold a 70B model. Strix Halo flips the constraint: it has enormous capacity (128 GB unified) but modest bandwidth (a 256-bit LPDDR5X-8000 bus, ~256 GB/s theoretical, roughly 215 GB/s measured under real load). Token generation is bandwidth-bound, so the memory bus is the ceiling — not the compute units. Keep that number in your head; it explains every performance result below.
How do you allocate 96GB+ to the iGPU?
Out of the box the iGPU is only handed a few gigabytes. You have to explicitly give it the memory. The mechanism differs by OS.
Windows: AMD Adrenalin Variable Graphics Memory
Open AMD Adrenalin → Performance → Tuning and set Variable Graphics Memory (VGM) to Custom. On a 128 GB machine the maximum is 96 GB, leaving 32 GB for Windows. Reboot. The setting is stored at the firmware level, so it survives driver and OS updates. AMD's own guidance is that 96 GB dedicated lets the iGPU address up to ~112 GB (96 dedicated + ~16 shared), which covers Q4 models up to roughly 128B parameters.
Linux: small BIOS framebuffer + a GTT kernel parameter
On Linux the better pattern is the opposite of Windows: keep the BIOS dedicated allocation tiny and let GTT (the Graphics Translation Table — kernel-managed shared memory the iGPU maps on demand) do the heavy lifting. In BIOS set Integrated Graphics → UMA Frame Buffer Size → 512 MB. Then raise the GTT limit with a kernel boot parameter.
Two parameters do the job depending on your kernel. The simplest is amdgpu.gttsize, which takes megabytes:
# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdgpu.gttsize=110000"
# ~107 GB of GTT; adjust down if you have less RAMOn kernels that still expose the TTM allocator directly, use the page-count form instead (each page is 4 KiB, so 27,648,000 pages ≈ 105 GiB). On kernel 6.10+ the module was renamed, so the prefix is amdttm rather than ttm:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdttm.pages_limit=27648000 amdttm.page_pool_size=27648000"Apply and reboot, then confirm the GPU actually sees the memory:
sudo update-grub && sudo reboot
# after reboot, check total GTT handed to the GPU:
cat /sys/class/drm/card*/device/mem_info_gtt_total
rocminfo | grep -A4 "Pool 1"If mem_info_gtt_total doesn't reflect your target, your parameter name is wrong for your kernel — try the other form. Which one works is genuinely kernel-version dependent, so verify rather than assume.
ROCm or Vulkan — which backend should you use?
Both work on the 8060S (gfx1151). Community benchmarks are consistent on the practical takeaway:
- Vulkan (RADV) is the default you want. At normal context lengths it matches or beats ROCm for token generation on this iGPU, and it works out of the box in LM Studio and prebuilt llama.cpp binaries with zero driver wrangling.
- ROCm pays off at long context. With tuned builds, ROCm has shown roughly 3× better prompt-processing throughput at very large context windows (130K+ tokens), which matters if you're feeding huge prompts. It's a heavier install (the ROCm stack) and gfx1151 support is still maturing.
Start on Vulkan. Only move to ROCm if you specifically need faster prompt processing on long inputs and are willing to build for it.
LM Studio, Ollama, or llama.cpp?
All three run here; pick by how much control you want.
- LM Studio 0.3.x — easiest path on Windows and Linux. It ships selectable llama.cpp runtimes (Vulkan and ROCm); pick the Vulkan runtime, set the model's GPU-offload slider to max, and it uses the memory you allocated above.
- Ollama — great for a headless API server. Note it has historically underused GTT/unified memory versus llama.cpp on these APUs; recent versions plus a correct VGM/GTT allocation resolve most of it, but check
ollama psshows the model on GPU, not split to CPU. - llama.cpp — the most control and the fastest to get new-model and quant support. Example for the MoE 120B, which streams cleanly on Strix Halo:
# auto-downloads the GGUF; full 128k context, flash attention
llama-server -hf ggml-org/gpt-oss-120b-GGUF \
--ctx-size 0 --jinja -fa -ub 2048 -b 2048Because you allocated ~96 GB+ to the GPU, you do not need --n-cpu-moe offload flags — the whole model lives in iGPU-addressable memory, which is the entire point of the platform.
Which 70B and 120B quants actually fit?
With 96 GB dedicated (plus GTT headroom), you have room for the weights and the KV cache. Approximate footprints at common quants:
| Model | Quant | Approx. weights | Fits in ~96 GB? |
|---|---|---|---|
| Qwen3-30B-A3B (MoE) | IQ4_XS / Q4 | ~17–20 GB | Easily |
| Llama 3.3 70B (dense) | Q4_K_M | ~40 GB | Comfortably |
| gpt-oss-120b (MoE) | MXFP4 | ~60–65 GB | Yes, with context headroom |
| Qwen3-235B-A22B (MoE) | Q2–Q3 | ~80–100 GB | Tight — small context only |
The 120B MoE is the sweet spot: near-frontier quality that no single 24 GB or 32 GB consumer card can hold, running entirely on one APU. The 235B is possible only at aggressive quantization and leaves little room for a long KV cache.
What tokens/sec should you actually expect?
Community benchmarks on the 128 GB Strix Halo (Vulkan/RADV, Q4-class quants) land roughly here:
| Model | Type | Token generation |
|---|---|---|
| Qwen3-30B-A3B | MoE (3B active) | ~100 t/s |
| gpt-oss-120b | MoE (MXFP4) | ~55 t/s |
| Llama 3.3 70B | Dense | ~4–6 t/s |
The pattern is the bandwidth ceiling in action. MoE models activate only a few billion parameters per token, so little memory is read each step and they fly. A dense 70B reads ~40 GB of weights per token; at ~215 GB/s that caps you near 5 t/s no matter how you tune it. If your workload is dense 70B, Strix Halo runs it — just not fast. If it's MoE, it's genuinely pleasant.
When does a discrete GPU still win?
Here's the caveat the marketing skips: for any model that fits in a discrete GPU's VRAM (roughly ≤32B at Q4), a dGPU wins on speed, and it isn't close. An RTX 5090 has ~1,792 GB/s of bandwidth against Strix Halo's ~215 GB/s real-world — about 8×. On an 8B model that means ~186 t/s on the 5090 versus ~80–90 t/s here.
So the decision is clean:
- Buy Strix Halo for capacity. It runs 70B–120B models on one quiet, roughly $1,700–$2,200 box that sips power. Nothing else at that price and form factor can.
- Buy a discrete GPU for speed under ~32B. If your models fit in 24–32 GB, a 4090/5090 will out-run the APU several times over.
For a full cross-hardware breakdown see our local LLM hardware showdown, and if budget is the deciding factor, the cheapest way to run a local LLM. For the broader stack — quantization, serving, and hardening a private deployment — our self-hosting LLMs guide is the pillar to bookmark.
If you'd rather have vetted engineers stand up and maintain a private local-inference stack instead of doing it yourself, Codersera can help you extend your team with developers who've shipped this kind of infrastructure.
FAQ
Do I need to allocate memory in BIOS or in the OS?
On Windows, use AMD Adrenalin's Variable Graphics Memory slider (up to 96 GB) — that's the primary lever. On Linux, keep the BIOS framebuffer small (512 MB) and raise GTT with an amdgpu.gttsize or amdttm.pages_limit kernel parameter, then verify with mem_info_gtt_total.
Should I choose ROCm or Vulkan?
Start with Vulkan (RADV) — it's the default in LM Studio, needs no extra setup, and matches or beats ROCm for token generation at normal context lengths on gfx1151. Switch to a tuned ROCm build only if you need faster prompt processing on very long inputs.
Can a Ryzen AI Max+ 395 really run a 120B model?
Yes. gpt-oss-120b (an MoE model in MXFP4) is ~60–65 GB and fits comfortably in a 96 GB allocation, generating around 55 t/s on Vulkan — something no single 24–32 GB consumer GPU can do at all.
Why is a dense 70B model so slow compared to a 30B MoE?
Token generation is memory-bandwidth-bound. A dense 70B reads ~40 GB of weights per token, so at ~215 GB/s you're capped around 5 t/s. A 30B MoE activates only ~3B parameters per token, reads far less memory, and hits ~100 t/s on the same hardware.
Is Strix Halo faster than an RTX 4090 or 5090?
No — for models that fit in the discrete card's VRAM (≤~32B), the dGPU is several times faster because it has roughly 8× the memory bandwidth. Strix Halo's advantage is capacity: running 70B–120B models that don't fit on a single consumer GPU at all, in a small low-power box.
Does Ollama work, or should I use llama.cpp?
Both work. Ollama is convenient for a headless API but historically under-allocated unified memory on these APUs — confirm with ollama ps that the model is on GPU. llama.cpp (and LM Studio, which wraps it) gives the most control and the earliest support for new quants.