Best Small LLMs to Run Locally: A Comprehensive Guide
Small language models have quietly become one of the most useful tools a developer can keep on their own machine. You no longer need a rack of GPUs or a cloud bill to get a capable model answering questions, drafting code, and summarising documents entirely offline.
The picture has changed a lot since the early open-weight days of the original LLaMA, Alpaca and GPT-J. In 2026 a 4-billion-parameter model on a laptop can do work that used to need something ten times its size. This guide covers the best small LLMs to run locally today, how much hardware they need, and the simplest ways to get them running.
Quick answer. The best small LLMs to run locally in 2026 are Qwen3 (1.7B–14B), Google Gemma 3 (1B–12B), Meta Llama 3.2 (1B/3B) and Microsoft Phi-4-mini (3.8B). Qwen3 is the safest all-round default. Run them with Ollama or LM Studio; most need only 8–16 GB of RAM.
What counts as a small LLM in 2026?
A small LLM (sometimes called a small language model, or SLM) is a compact model you can run on consumer hardware rather than a cloud GPU cluster. There is no official cut-off, but in practice small models today share three traits:
- Roughly 0.5B to 14B parameters — small enough that a quantized version fits in the memory of a normal laptop or a single mid-range GPU.
- They run well after 4-bit quantization, which cuts memory use by roughly 4x with only a small quality trade-off.
- They ship as open weights, so you can download, run and often fine-tune them without an API key.
The key shift since 2023 is quality-per-parameter. A modern 3–4B model is instruction-tuned, multilingual, and often better at reasoning than a 13B model from two years ago. That is why the old go-tos — vanilla LLaMA 7B, Alpaca, Vicuna, GPT-J, GPT-Neo, Falcon 7B — have been retired in favour of the families below.
Why run a small LLM locally?
Running a model on your own machine buys you several things a hosted API cannot:
- Privacy. Prompts and data never leave your device — important for proprietary code, legal, or medical work.
- Cost. No per-token billing. Once the model is downloaded, inference is free.
- Offline use. Works on a plane, in a secure network, or anywhere without reliable internet.
- Latency and control. No network round-trip, no rate limits, and you can pin a specific version so behaviour never changes underneath you.
- Customisation. You can fine-tune or attach retrieval to adapt the model to your own domain.
If you want the full picture of running open models on your own hardware, see the Codersera guide to self-hosting LLMs.
What are the best small LLMs to run locally in 2026?
These are the model families worth your download in 2026, grouped by who makes them and what they are best at.
Qwen3 — the default all-rounder
Alibaba's Qwen3 family has quietly become the answer to "what should I run locally?" for most developers. It spans a wide range of sizes — 0.6B, 1.7B, 4B, 8B and 14B on the dense side — all under a permissive Apache 2.0 license with no commercial restrictions.
- Best for: general chat, coding help, refactoring, multilingual work.
- Sweet spot: Qwen3 8B is widely seen as the best overall choice on an 8 GB GPU; Qwen3 4B is a strong pick for CPU-only laptops.
- Why it wins: strong quality-per-parameter, a clean license, and excellent tooling support.
Qwen has its own deep-dive on Codersera — see the Qwen guide.
Google Gemma 3 — strong and multimodal
Google's Gemma 3 comes in 1B, 4B, 12B and 27B sizes. The 4B and larger variants are multimodal (they can read images) and support 140+ languages, which makes Gemma the pick when you need vision or broad language coverage on-device.
- Best for: multimodal input, multilingual tasks, general use on a 16 GB machine (Gemma 3 12B).
- Note: Gemma ships under Google's own license — check the terms for commercial use.
Google's newer generation continues this line; see the Codersera Gemma guide for the latest.
Meta Llama 3.2 — the easy starting point
Meta's small Llama models — Llama 3.2 1B and 3B — are the most frictionless way to start. They are tiny, well-documented, supported everywhere, and good at on-device tasks like summarisation and simple chat.
- Best for: a first local model, summarisation, on-device assistants.
- Trade-off: the 1B/3B sizes trail Qwen3 and Phi-4 on hard reasoning, but they are hard to beat for speed and simplicity.
Microsoft Phi-4 and Phi-4-mini — reasoning above their weight
Microsoft's Phi models are trained heavily on curated and synthetic data, which makes them punch above their size on maths, logic and structured problem-solving. Phi-4 is a 14B model that regularly competes with much larger models on reasoning tasks, while Phi-4-mini (3.8B) is arguably the best small reasoner you can run on minimal hardware — around 3 GB of VRAM at 4-bit. Both ship under a permissive MIT license.
- Best for: maths, logic, step-by-step reasoning on modest hardware.
Mistral 7B — the mature, code-first option
Mistral 7B is still a dependable workhorse with one of the largest ecosystems of fine-tunes and integrations. It is Apache 2.0, runs comfortably on an 8 GB GPU, and remains a favourite for code-first local setups. Mistral's smaller Ministral models (3B and 8B) extend the family toward edge use.
- Best for: coding, a well-supported general model, easy fine-tuning.
DeepSeek-R1 distilled models — for logic-heavy work
DeepSeek released distilled reasoning models built on Qwen and Llama bases (commonly 7B, 8B and 14B). These bring chain-of-thought style reasoning down to sizes you can run locally, and they are strong at step-by-step problem solving. They are MIT-licensed and available in Ollama.
- Best for: reasoning, maths, and problems that benefit from visible thinking steps.
For the larger DeepSeek line, see the Codersera DeepSeek guide.
SmolLM2 — when you need the absolute smallest
Hugging Face's SmolLM2 comes in 135M, 360M and 1.7B sizes. These are the models to reach for on the tightest hardware, in browsers, or on edge devices. They handle summarisation and simple instructions but are not built for complex reasoning.
- Best for: ultra-low RAM, embedded and edge use, experimentation.
Small LLM comparison table
Approximate figures for 4-bit (Q4) quantized versions. Real memory use varies with context length and runtime.
| Model | Params | Approx. RAM/VRAM (Q4) | Best for | License |
|---|---|---|---|---|
| Qwen3 1.7B | 1.7B | ~2 GB | Fast chat on any laptop | Apache 2.0 |
| Qwen3 4B | 4B | ~3–4 GB | Balanced chat + light coding | Apache 2.0 |
| Qwen3 8B | 8B | ~6–8 GB | Best all-round on an 8 GB GPU | Apache 2.0 |
| Qwen3 14B | 14B | ~10–12 GB | Coding, refactoring, reasoning | Apache 2.0 |
| Gemma 3 4B | 4B | ~3–4 GB | Multimodal / 140+ languages | Gemma |
| Gemma 3 12B | 12B | ~9–10 GB | Strong general use on 16 GB | Gemma |
| Llama 3.2 3B | 3B | ~3 GB | Easy starting point | Llama |
| Phi-4-mini | 3.8B | ~3 GB | Reasoning/math on small hardware | MIT |
| Phi-4 | 14B | ~10–12 GB | Math/logic above its weight | MIT |
| Mistral 7B | 7B | ~5–6 GB | Code-first, mature ecosystem | Apache 2.0 |
| DeepSeek-R1 distill 8B | 8B | ~6–8 GB | Logic-heavy, step-by-step reasoning | MIT |
| SmolLM2 1.7B | 1.7B | ~1.5 GB | Edge / ultra-low RAM | Apache 2.0 |
How much RAM and hardware do you need?
A useful rule of thumb: a 4-bit quantized model needs roughly 0.6–0.75 GB of memory per billion parameters, plus a little headroom for context. So a 7B model needs about 5–6 GB, and a 14B model about 10–12 GB. Your options scale with the memory you have:
| You have | Comfortable model size | Good picks |
|---|---|---|
| 8 GB RAM, no GPU | 1–4B | Qwen3 4B, Phi-4-mini, Gemma 3 1B, Llama 3.2 3B |
| 8 GB GPU (VRAM) | 7–8B | Qwen3 8B, Mistral 7B, DeepSeek-R1 distill 8B |
| 16 GB RAM / VRAM | 12–14B | Gemma 3 12B, Qwen3 14B, Phi-4 |
| 24 GB+ GPU | 14–32B | Qwen3 32B, larger DeepSeek variants |
Apple Silicon Macs are a special case: their unified memory lets the GPU use most of the system RAM, so an M-series MacBook with 16–32 GB can run models that would need a dedicated GPU on a PC. See the Codersera Apple Silicon LLM guide for the details.
Golden rule: don't start with the biggest model. Start with the best model your hardware runs comfortably. A fast 4B model beats a 14B model that swaps memory and answers at one token per second.
How do you run a small LLM locally?
You don't need to touch Python to get started. Three tools cover almost every use case:
- Ollama — the simplest path. Install it, then pull and run a model with one command. It handles downloading, quantization and an OpenAI-compatible API on
localhost:11434. - LM Studio — a polished desktop app with a chat UI, model browser and one-click downloads. Best if you prefer a graphical interface.
- llama.cpp — the underlying C/C++ engine that powers much of the ecosystem. Reach for it when you want maximum control or to run GGUF files directly on CPU or Apple Silicon.
A first run with Ollama looks like this:
# Pull and chat with Qwen3 8B (best all-rounder on an 8 GB GPU)
ollama pull qwen3:8b
ollama run qwen3:8b
# Lighter options for CPU-only laptops
ollama pull qwen3:4b
ollama pull llama3.2:3b
ollama pull phi4-mini
ollama pull gemma3:4bOnce a model is running, both Ollama and LM Studio expose an OpenAI-compatible endpoint, so you can point existing tools and scripts at your local model just by changing the base URL.
How do you pick the right small LLM?
Match the model to the job rather than chasing a leaderboard:
- General chat and everyday tasks: Qwen3 (4B or 8B) — the safest default.
- Coding and refactoring: Qwen3 14B, Mistral, or a DeepSeek-R1 distill on a 12–16 GB GPU.
- Maths, logic and reasoning on small hardware: Phi-4-mini, then Phi-4 if you have 16 GB.
- Images or many languages: Gemma 3 4B or 12B.
- Absolute minimum footprint: Llama 3.2 1B, Gemma 3 1B or SmolLM2.
For a wider map of the open-weight ecosystem beyond small models, see the Codersera open-source LLM landscape.
What are the limits of small local LLMs?
Small models are genuinely useful, but it helps to know where they stop:
- Less world knowledge. They know less than frontier cloud models and can be confidently wrong, so verify facts for anything important.
- Weaker on very hard reasoning. A 3B model won't match a large hosted model on complex, multi-step problems.
- Context and speed trade-offs. Long contexts use more memory; on CPU, big models can be slow.
- Quality varies by task. A model that's great at chat may be mediocre at code. Test a couple before you commit.
For many workflows the fix is retrieval (RAG) or a quick fine-tune to give the model your own data — both are far cheaper locally than they sound.
FAQ
What is the best small LLM to run locally in 2026?
For most people, Qwen3 8B is the best all-round choice on an 8 GB GPU, and Qwen3 4B or Llama 3.2 3B work well on CPU-only laptops. Pick the largest model your hardware runs comfortably rather than the biggest one available.
What is the smallest LLM you can run locally?
Models like SmolLM2 (135M–1.7B), Llama 3.2 1B, Gemma 3 1B and Qwen3 0.6B run on almost any modern laptop, often in under 2 GB of RAM when 4-bit quantized. They handle summarisation and simple chat but are weak at complex reasoning.
How much RAM do I need to run a small LLM locally?
A 4-bit quantized 3–4B model needs roughly 3–4 GB of free RAM or VRAM, a 7–8B model needs about 6–8 GB, and a 12–14B model needs 10–16 GB. 16 GB of system RAM is a comfortable baseline for most small models.
What is the easiest way to run a small LLM locally?
Install Ollama or LM Studio, then pull a model with one command (for example ollama pull qwen3:8b). Both handle downloading, quantization and a local chat interface, and expose an OpenAI-compatible API for your own apps.
Can I run a small LLM without a GPU?
Yes. With 4-bit quantization, models up to about 7–8B run on a modern CPU, and they run noticeably faster on Apple Silicon Macs thanks to unified memory. Expect slower token speeds than on a dedicated GPU, but they're usable for chat and drafting.
Are small local LLMs good for coding?
They can be. Qwen3 and DeepSeek-R1 distilled models are strong at code among small models, and Mistral has a mature code-first ecosystem. For heavier refactoring, a 14B model on a 12–16 GB GPU gives the best quality you can get locally.
The bottom line
Small local LLMs have gone from a hobbyist experiment to a practical daily tool. Start with Qwen3 for a balanced default, reach for Phi-4-mini when you need reasoning on light hardware, and pick Gemma 3 when you need vision or many languages — all through Ollama or LM Studio in a few minutes. If you're building a product on top of local or self-hosted models and need engineering help, Codersera can connect you with vetted remote developers who ship AI features.