Skip to content

Test status: tested

CI job recipe-local-chat runs recipes/local-chat/smoke.sh on GitHub-hosted ubuntu-latest (4 vCPU, 16 GB RAM, no GPU) on every push and PR. It validates the compose file, pulls qwen2.5:0.5b, asks the gateway 17+25 and asserts the reply contains 42 with non-zero completion tokens, checks ollama ps shows the model loaded and Ollama's log shows the chat request, and checks the Open WebUI health endpoint. Pinned: Ollama 0.40.6, LiteLLM v1.106.1, Open WebUI v0.12.3. Not tested: GPUs, larger models, the browser chat flow in Open WebUI.

Local private chat: Ollama + LiteLLM + Open WebUI

Data flow: browser → Open WebUI (:3000) → LiteLLM gateway (:4000, OpenAI-compatible) → Ollama (:11434) → local model. Nothing leaves your machine after the first image/model download.

Requirements

Minimum (tested) For bigger models
CPU / RAM 4 vCPU, 8 GB RAM 16 GB+ RAM, NVIDIA GPU or Apple Silicon
Disk ~8 GB (images + 0.4 GB model) 25 GB+
Software Docker 24+ with Compose v2 –
Network Internet for first pull only –

No accounts or paid API keys are needed.

Pinned versions

Component Version
Ollama ollama/ollama:0.40.6
LiteLLM ghcr.io/berriai/litellm:v1.106.1
Open WebUI ghcr.io/open-webui/open-webui:v0.12.3
Model qwen2.5:0.5b (~0.4 GB)

Typical cold start on a 4-vCPU CI runner: about 2 minutes (image pulls dominate).

Run

git clone https://github.com/CodesbyFebin/Open-Source-Ai
cd Open-Source-Ai/recipes/local-chat
export LITELLM_MASTER_KEY=sk-$(openssl rand -hex 12)   # never commit real keys
docker compose up -d
docker compose wait model-pull

Environment variables: LITELLM_MASTER_KEY (gateway key, default is a placeholder), MODEL (Ollama model, default qwen2.5:0.5b; also change litellm-config.yaml).

Verify

./smoke.sh     # same test CI runs; asks 17+25 and checks the model answers 42,
               # token usage is reported, and Ollama's own log shows the request
# or manually
curl -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
  http://localhost:4000/v1/chat/completions \
  -d '{"model":"local-llama","messages":[{"role":"user","content":"Say hi"}]}'

Open http://localhost:3000, create the first (admin) account and pick local-llama.

Clean up

docker compose down        # keep data
docker compose down -v     # also delete models and chats

Known limitations

  • qwen2.5:0.5b is a tiny model chosen so CI can run on CPU; answers are low quality. Use a 3B–8B model for real use.
  • Ollama model tags are not immutable; the digest actually pulled is printed by ollama list in the CI log.