Docker & Containerization for Local AI: GPUs, vLLM High-Throughput & Ollama

By Sarah Lin • Advanced • 21 min read • Updated 2026-09-12

What You Will Master in This Tutorial

Configure NVIDIA Container Toolkit to pass physical GPU acceleration into Docker containers.
Deploy vLLM in Docker for high-throughput serving with PagedAttention and continuous batching.
Orchestrate a complete local AI developer stack using Docker Compose.
Optimize shared memory (shm-size) and persistent model cache storage across deployments.

1. Configuring GPU Passthrough with NVIDIA Container Toolkit

Running local LLMs inside Docker requires exposing the host GPU and CUDA drivers to the container runtime. The modern standard is the NVIDIA Container Toolkit (nvidia-ctk).

BASH

# Configure NVIDIA repository and install toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Test GPU access inside container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Note: Crucial Configuration: Always verify that nvidia-smi runs successfully inside the test container before deploying AI stacks.

2. Production Docker Compose: vLLM + WebUI + OpenClaw

Below is a battle-tested Docker Compose manifest orchestrating high-throughput vLLM serving, persistent Hugging Face cache volumes, and an OpenAI-compatible API endpoint.

YAML

services:
  vllm-inference:
    image: vllm/vllm-openai:latest
    container_name: vllm-deepseek
    runtime: nvidia
    restart: unless-stopped
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
      --tensor-parallel-size 1
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --trust-remote-code

Note: Performance Note: ipc: host is mandatory for vLLM and PyTorch to prevent Shared Memory (shm) allocation panics during high-concurrency inference.

Knowledge Check: Test Your Understanding

1. Why is 'ipc: host' required when running vLLM inside Docker?