Docker Model Runner: Run Local LLMs (vs Ollama, 2026)

Quick answer. Docker Model Runner (DMR) runs local LLMs as native host processes on top of llama.cpp, pulls and pushes models as OCI artifacts through Docker Hub, and serves an OpenAI-compatible API on port 12434. Choose it if you already live in Docker and CI; stay on Ollama for quick laptop prototyping.

Docker Model Runner is Docker's built-in way to run open-weight LLMs locally without wiring up a separate tool. If your team already ships everything through Docker images, registries, and Compose, DMR lets models ride the same rails: pull them like images, run them with a single command, and hit them from your app over a standard OpenAI-compatible endpoint. This guide walks the whole path — install, first run, the API, publishing your own models — and ends with an honest table on when to pick it over Ollama.

What is Docker Model Runner?

DMR is a Docker CLI plugin and background service that runs inference for you. Under the hood the default engine is llama.cpp, and the model runs as a native process on your host — not inside a container namespace — so there's no container overhead on the inference path. It also supports vLLM (higher-throughput Safetensors serving on supported NVIDIA GPUs) and a Diffusers engine for image models, exposed through the same interface.

The part that makes it feel like Docker: models are packaged as OCI artifacts, the same standard container images use. That means a model lives in a registry with a tag, a digest, and a manifest, and you version and distribute it exactly like any other image. Docker publishes ready-to-pull models under the ai/ namespace on Docker Hub (Llama, Mistral, Phi, Gemma, Qwen, SmolLM and more), and you can push your own.

How do you install and enable Docker Model Runner?

DMR ships with Docker Desktop (it first landed for Apple silicon in Docker Desktop 4.40 and has since expanded to Windows and to Docker CE on Linux, including WSL2).

Docker Desktop: open Settings → AI, toggle Enable Docker Model Runner, and optionally turn on host-side TCP support so apps running directly on your machine can reach the API. You can do the same from the CLI:

docker desktop enable model-runner --tcp 12434

Docker Engine on Linux: install the plugin package, then confirm it's live:

sudo apt-get update
sudo apt-get install docker-model-plugin

docker model version

Without the TCP flag, the API is reachable through the Docker socket and, from inside containers, at the hostname model-runner.docker.internal. With TCP enabled it also listens on localhost:12434.

Run your first model

Pull a small model and chat with it interactively. SmolLM2 is a good first pick because it's tiny and downloads fast:

docker model pull ai/smollm2

docker model run ai/smollm2

That drops you into an interactive prompt. You can also pass the message inline for a one-shot response:

docker model run ai/smollm2 "Explain OCI artifacts in two sentences."

You can pin a specific quantization by tag, and you can pull straight from Hugging Face:

docker model pull ai/smollm2:360M-Q4_K_M
docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF

Housekeeping commands mirror the container ones you already know:

docker model list        # models on disk
docker model ps          # models currently loaded in memory
docker model logs        # runner logs, useful for GPU/backend debugging
docker model rm ai/smollm2

Context size and other runtime settings are set per model:

docker model configure --context-size 8192 ai/qwen2.5-coder

How do you hit the OpenAI-compatible endpoint?

This is the reason DMR drops into existing code so cleanly: the HTTP surface speaks the OpenAI API. With host TCP enabled, the base URL is http://localhost:12434/engines/v1, and the usual routes are there: /chat/completions, /completions, /embeddings, and /models.

curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "messages": [{"role": "user", "content": "Explain containerization in one line."}],
    "stream": false
  }'

Point any OpenAI SDK at that base URL (the API key is ignored locally) and existing code works unchanged:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:12434/engines/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="ai/smollm2",
    messages=[{"role": "user", "content": "Say hi from a local model."}],
)
print(resp.choices[0].message.content)

From another container in the same Docker network, swap localhost:12434 for model-runner.docker.internal and drop the TCP requirement entirely. If you run more than one engine, you can be explicit with /engines/llama.cpp/v1/....

How do you package and push a model as an OCI artifact?

Because models are OCI artifacts, publishing one is tag + push, just like an image. Re-publish an existing model under your own namespace:

docker model tag ai/smollm2 myorg/smollm2:demo
docker model push myorg/smollm2:demo

To wrap a raw GGUF file you built or fine-tuned into a portable, versioned artifact and push it in one step, use docker model package:

docker model package \
  --gguf "$(pwd)/model.gguf" \
  --push myorg/mistral-7b-v0.1:Q4_K_M

The GGUF is stored as a layer in an OCI manifest with an AI-specific media type, so it lands in Docker Hub — or any OCI-compliant registry such as Artifactory, Harbor, or a cloud container registry — alongside your normal images. That single registry story is the whole pitch: one place for images and models, one set of access controls, one promotion pipeline.

GPU support: Apple Metal and NVIDIA CUDA

DMR uses your accelerator automatically when it can. Coverage depends on OS and hardware:

Because llama.cpp handles CPU plus Metal, CUDA, ROCm, and Vulkan, the same GGUF runs on an Apple Silicon laptop, an NVIDIA workstation, or a CPU-only Linux box without changing the model reference — you just get more or less throughput.

Using models inside Docker Compose

The strongest DMR-native workflow is Compose. Declare a model with the top-level models element and link it to a service; Compose pulls the artifact, starts the runner, and injects connection details as environment variables into your app container:

services:
  app:
    build: .
    models:
      - chat

models:
  chat:
    model: ai/smollm2
    context_size: 8192

Now docker compose up brings up your app and its model together, and the same file works on a teammate's laptop or in CI. (An older provider service syntax exists in earlier docs but is deprecated — use the models element for new projects.)

Docker Model Runner vs Ollama: the honest comparison

On raw speed they're close — both lean on the llama.cpp family, both run GGUF, so the same model on the same hardware performs comparably. The real difference is workflow and ecosystem.

So which should you use?

The decision is mostly about where the rest of your work already lives.

• Switch to Docker Model Runner if you already run everything through Docker, want models to share the registry, tagging, and CI pipeline as your images, or need a model spun up next to services in a Compose file for local dev and integration tests.
• Stay on Ollama if you want the shortest path from zero to a running model on a laptop, you're prototyping and value the bigger model library and community, or you don't want Docker as a dependency at all.

For most people the two aren't rivals so much as different defaults: Ollama for the desktop tinkering loop, DMR for the moment that loop needs to become a reproducible container workflow the whole team and CI can run. If you're weighing the wider field, our guide to self-hosting LLMs covers the production side, and these two comparisons go deeper on the runtimes: best Ollama alternatives for 2026 and Ollama vs LM Studio vs vLLM vs llama.cpp vs MLX.

Standing up local inference is one thing; building the services around it is another. If you need engineers who are fluent in Docker, LLM tooling, and shipping this to production, Codersera can help you hire vetted remote developers to extend your team.

FAQ

Is Docker Model Runner free?

Yes. DMR ships with Docker Desktop and Docker Engine, and the models in the ai/ namespace on Docker Hub are open-weight and free to pull. Your only real cost is the local compute (and any registry storage if you publish private models).

Does Docker Model Runner run the model inside a container?

No. The inference engine (llama.cpp by default) runs as a native process on your host so it can use Metal or CUDA directly, which avoids container overhead on the hot path. Only the packaging and distribution use the OCI/container format.

Can I use my existing OpenAI code with Docker Model Runner?

Yes. Point any OpenAI SDK at http://localhost:12434/engines/v1 (host TCP enabled) or http://model-runner.docker.internal/engines/v1 from inside a container. The chat, completions, and embeddings routes match the OpenAI API, so most apps work with only a base-URL change.

Can Docker Model Runner pull models from Hugging Face?

Yes. Use docker model pull hf.co/<org>/<repo> for GGUF repositories, in addition to the curated ai/ models on Docker Hub.

Where does Docker Model Runner store models?

Models are stored locally as OCI artifacts managed by DMR. Run docker model list to see what's on disk and docker model rm to reclaim space; docker model ps shows what's currently loaded in memory.

Do I need a GPU to use it?

No. llama.cpp runs on CPU alone, so DMR works on a plain Linux box or a laptop without a discrete GPU. A GPU (Apple Metal, NVIDIA CUDA, AMD ROCm) simply gives you more throughput on larger models.