How to Deploy Ollama on a VPS

Updated Aug 2026

verified on Ubuntu 26.04 · Aug 2026

Self-host Ollama on your own VPS — run open-source LLMs locally with GPU acceleration, an OpenAI-compatible API, and zero per-token costs.

Before you start

A VPS with 8+ GB RAM (16 GB recommended for 7B models)
A fresh Ubuntu 26.04 server with root/sudo SSH access
Optional: NVIDIA GPU for acceleration (CPU-only works for smaller models)

What Ollama is

Ollama is a lightweight, open-source tool for running large language models locally. It's like Docker for LLMs — you pull model weights with a simple command, and Ollama serves them through an OpenAI-compatible API. No API keys, no per-token billing, no data leaving your server.

The appeal is straightforward: cost and privacy. Commercial AI APIs charge per token, and a busy application can burn through credits fast. A VPS running Ollama costs the same whether you process a hundred requests a day or ten thousand. Your prompts and completions never leave your infrastructure, which matters when you're building tools that handle sensitive data or when you simply don't want your prompts training someone else's model.

Ollama supports hundreds of models — Llama 3, Mistral, Gemma, Phi, Qwen, and more — and downloads them on demand. The API is wire-compatible with OpenAI's, so most tools that work with ChatGPT can point at your Ollama instance with minimal changes.

Server sizing — RAM is the bottleneck

LLM inference is memory-bound. The model weights must fit in RAM (or VRAM) to run at all, and once they fit, more RAM means more room for context and concurrent requests.

CPU-only sizing:

8 GB RAM — runs small models (3B parameters) comfortably, 7B models with tight context
16 GB RAM — the sweet spot for 7B models with reasonable context length
32 GB RAM — 13B models, or 7B models with large context windows
64+ GB RAM — 30B+ models, or serving multiple users concurrently

GPU acceleration: If your VPS has an NVIDIA GPU, Ollama can use CUDA for dramatically faster inference. The model still loads into GPU VRAM, so the same sizing logic applies — a 7B model needs ~6 GB VRAM in FP16, ~4 GB in quantized format. GPU inference is 5-10x faster than CPU for tokens/second.

A Hetzner CX32 (4 vCPU / 8 GB) is the minimum for CPU-only small models. For serious 7B+ usage, you need at least 16 GB RAM — the Hetzner CPX31 (4 vCPU / 8 GB) won't cut it; look at the CX42 (8 vCPU / 16 GB) or a GPU-equipped option from Kamatera or Vultr.

Prepare the server

Start from a fresh Ubuntu 24.04 or 26.04 server. Update and create a non-root user:

apt update && apt upgrade -y
adduser deploy
usermod -aG sudo deploy

Lock down the firewall:

ufw allow OpenSSH
ufw allow 80
ufw allow 443
ufw enable

Install Docker (Ollama runs in a container):

curl -fsSL https://get.docker.com | sh
usermod -aG docker deploy

Log out and back in as deploy so the docker group takes effect.

Install Ollama

The simplest path is the official Docker image. Create a working directory:

mkdir ~/ollama && cd ~/ollama

Create a compose file:

services:
  ollama:
    image: ollama/ollama:latest
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]  # comment out if no GPU

volumes:
  ollama_data:

Start it:

docker compose up -d

Ollama listens on port 11434 by default. The 127.0.0.1 bind keeps it private — a reverse proxy will handle public HTTPS access.

GPU support

If you have an NVIDIA GPU, install the NVIDIA Container Toolkit first:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt update
apt install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Without a GPU, just remove the deploy.resources section from the compose file — Ollama falls back to CPU inference automatically.

Pull your first model

Pull a model and test it:

docker exec -it ollama ollama pull llama3.2
docker exec -it ollama ollama run llama3.2

That downloads the model weights (a few GB for 3B/7B variants) and drops you into an interactive chat. Type your message, press Enter, and watch it generate. /bye exits.

To pull a larger model later:

docker exec -it ollama ollama pull llama3.1:8b

The API is available immediately — no restart needed. Test it:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "What is the capital of France?"
}'

HTTPS + domain

Ollama's API should not face the internet without TLS. Point a reverse proxy at 127.0.0.1:11434 and terminate HTTPS on 443.

The simplest path is Automatic HTTPS with Caddy. Point an A record for your hostname (say ai.example.com) at the server's public IP, then have Caddy reverse-proxy that hostname to 127.0.0.1:11434. Caddy handles certificate issuance and renewal automatically.

If you're using the Caddy container approach from that guide, put Ollama and Caddy in the same compose file and proxy to the Ollama service name:

reverse_proxy ollama:11434

Using the API

Once HTTPS is up, point any OpenAI-compatible client at your endpoint. Most tools just need two values:

Base URL: https://ai.example.com/v1
Model: any model you've pulled (e.g., llama3.2)

Examples with curl:

# Chat completion (OpenAI-compatible)
curl https://ai.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# List available models
curl https://ai.example.com/v1/models

Tools like Open WebUI, LibreChat, and AnythingLLM can all point at this endpoint. If you're building your own app, the OpenAI SDK works with Ollama by changing the base URL — no code changes needed in most cases.

Backups

Ollama's state is small — just the model weights in the /root/.ollama volume. The weights themselves are re-downloadable, so you don't strictly need to back them up. What matters is your configuration and any custom Modelfiles you've created.

If you've customized models (system prompts, parameters, Modelfiles), back up the data volume:

docker compose exec ollama tar czf - /root/.ollama > ollama-config-$(date +%F).tar.gz

For a more complete backup including all downloaded models:

docker run --rm -v ollama_data:/data -v $(pwd):/backup alpine \
  tar czf /backup/ollama-full-$(date +%F).tar.gz -C /data .

This is a one-time cost — models don't change once downloaded, so you only need to back up when you add or modify something.

Upgrades

Pull the newer image and recreate:

docker compose pull
docker compose up -d

Ollama updates are usually non-breaking — the API is stable and model formats are backward-compatible. Check the Ollama blog for any notable changes.

Scaling for production

For heavier usage, consider these additions:

Separate model storage: Mount a larger disk for models if your root volume is small:

volumes:
  - /mnt/models:/root/.ollama

Multiple instances: Run separate Ollama containers for different model sizes or use cases, each on a different port.

Load balancing: For high-concurrency setups, put multiple Ollama instances behind a load balancer. Ollama handles one request per model at a time, so multiple instances means more parallel requests.

Nginx reverse proxy: For finer control than Caddy, use Nginx with rate limiting and caching. See the Next.js deployment guide for Nginx configuration patterns.

Troubleshooting

Model download fails or hangs. Check your disk space (df -h) — 7B models are 4-8 GB. Also check that you can reach ollama.com from the server (curl -I https://ollama.com).

Out of memory during inference. The model is too large for your RAM. Use a smaller model or a more aggressive quantization (e.g., q4_0 instead of q8_0). Check docker logs ollama for memory errors.

GPU not detected. Verify the NVIDIA Container Toolkit is installed and Docker can see the GPU: docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi. If that fails, re-run the toolkit installation above.

Slow responses on CPU. This is expected — CPU inference is 5-10x slower than GPU. For better performance, use a smaller model or add a GPU. The tokens/second metric in the response tells you the actual throughput.

API returns "model not found." The model name must match exactly what you pulled. Check available models with curl http://localhost:11434/api/tags.

Verification + next steps

You're done when you can: pull a model, run it interactively, make API calls over HTTPS on your domain, and get coherent responses. The API should work with any OpenAI-compatible client.

From here, pair Ollama with Open WebUI for a ChatGPT-style browser interface, or integrate it into your applications via the API. For document-based AI workflows, AnythingLLM combines Ollama with a document management system. A Hetzner CX42 (8 vCPU / 16 GB) is the value pick for 7B models; reach for Kamatera when you want GPU instances or precise RAM sizing. See Best VPS for AI & ML Workloads for the ranked picks.

Next steps

How to Deploy Ollama on a VPS

#What Ollama is

#Server sizing — RAM is the bottleneck

#Prepare the server

#Install Ollama

#GPU support

#Pull your first model

#HTTPS + domain

#Using the API

#Backups

#Upgrades

#Scaling for production

#Troubleshooting

#Verification + next steps