How to Deploy LocalAI on a VPS

copyapt update && apt upgrade -y
adduser deploy
usermod -aG sudo deploy

copyufw allow OpenSSH
ufw allow 80
ufw allow 443
ufw enable

copycurl -fsSL https://get.docker.com | sh
usermod -aG docker deploy

#Install LocalAI

Create a working directory:

copymkdir ~/localai && cd ~/localai

Create a compose file:

copyservices:
  localai:
    image: localai/localai:latest
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - localai_models:/models
    environment:
      - THREADS=4
      - CONTEXT_SIZE=2048
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]  # comment out if no GPU

volumes:
  localai_models:

The THREADS setting controls CPU usage — set it to your vCPU count. CONTEXT_SIZE controls how much text the model can handle in a single request.

Start it:

copydocker compose up -d

LocalAI listens on port 8080. The 127.0.0.1 bind keeps it private.

#GPU support

If you have an NVIDIA GPU, install the NVIDIA Container Toolkit first (same as the Ollama guide):

copycurl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt update
apt install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Without a GPU, remove the deploy.resources section — LocalAI falls back to CPU inference.

#Download models

LocalAI needs models to serve. You can download them from Hugging Face, the LocalAI model hub, or manually place them in the models volume.

Using the LocalAI CLI:

copy# Download a text model (GGUF format)
docker exec -it localai local-ai model download --name hermes-3-llama-3.1-8b

# Download an embedding model
docker exec -it localai local-ai model download --name text-embedding-ada-002

# Download a Whisper model for audio transcription
docker exec -it localai local-ai model download --name whisper-1

Manual download:

copy# Download a model directly to the volume
docker exec -it localai wget -O /models/my-model.gguf \
  https://huggingface.co/TheBloke/Hermes-3-Llama-3.1-8B-GGUF/resolve/main/hermes-3-llama-3.1-8b.Q4_K_M.gguf

List available models:

copycurl http://localhost:8080/v1/models

#HTTPS + domain

LocalAI's API should not face the internet without TLS. Point a reverse proxy at 127.0.0.1:8080 and terminate HTTPS on 443.

The simplest path is Automatic HTTPS with Caddy. Point an A record for your hostname (say ai.example.com) at the server's public IP, then have Caddy reverse-proxy that hostname to 127.0.0.1:8080.

If you're using the Caddy container approach, put LocalAI and Caddy in the same compose file and proxy to the LocalAI service name:

reverse_proxy localai:8080

#Using the API

LocalAI's API is wire-compatible with OpenAI's. Point any OpenAI client at your endpoint:

• Base URL: https://ai.example.com/v1
• API Key: anything (LocalAI doesn't validate keys by default, but you can set one)

Examples with curl:

copy# Chat completion
curl https://ai.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hermes-3-llama-3.1-8b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Image generation
curl https://ai.example.com/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "stablediffusion",
    "prompt": "a sunset over mountains"
  }'

# Audio transcription
curl https://ai.example.com/v1/audio/transcriptions \
  -F file=@audio.mp3 \
  -F model=whisper-1

# List available models
curl https://ai.example.com/v1/models

Tools like Open WebUI, LibreChat, and AnythingLLM can all point at this endpoint. If you're building your own app, the OpenAI SDK works with LocalAI by changing the base URL.

#Backups

LocalAI's state is primarily the downloaded models in the /models volume. Models are re-downloadable, so you don't strictly need to back them up. If you've customized model configurations, back up the data volume:

copydocker run --rm -v localai_models:/data -v $(pwd):/backup alpine \
  tar czf /backup/localai-$(date +%F).tar.gz -C /data .

For a complete backup including all models:

copydocker run --rm -v localai_models:/data -v $(pwd):/backup alpine \
  tar czf /backup/localai-full-$(date +%F).tar.gz -C /data .

#Upgrades

Pull the newer image and recreate:

copydocker compose pull
docker compose up -d

LocalAI updates are usually non-breaking. Check the LocalAI changelog for any notable changes.

#Troubleshooting

Model download fails. Check disk space and network connectivity. Models can be several GB, and the download may time out on slow connections. Try downloading manually to the models volume.

Out of memory during inference. The model is too large for your RAM. Use a smaller model or a more aggressive quantization. Check docker logs localai for memory errors.

GPU not detected. Verify the NVIDIA Container Toolkit is installed and Docker can see the GPU: docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi. If that fails, re-run the toolkit installation.

Slow responses on CPU. Expected without a GPU. Image generation in particular is 10-50x slower on CPU. For better performance, add a GPU or use a smaller model.

API returns errors about missing model. The model name must match exactly what you downloaded. Check available models with curl http://localhost:8080/v1/models.

Image generation fails. Stable Diffusion models require significant RAM and ideally a GPU. If you're on CPU-only, image generation may timeout or produce poor results. Consider using a smaller image model or a remote backend for image tasks.

#Verification + next steps

You're done when you can: load the API over HTTPS, make chat completion requests, generate images, transcribe audio, and get coherent responses. The API should work with any OpenAI-compatible client.

From here, explore LocalAI's advanced features like voice cloning, function calling, and custom model configurations. For a simpler text-only setup, see Ollama. For a web interface, pair LocalAI with Open WebUI. An 8 vCPU / 16 GB box is the minimum for a full-featured setup; a GPU instance is the step up. See Best VPS for AI & ML Workloads for the ranked picks.