homefoam/04-knowledge/topics/self-hosting_AI.md

self-hosting AI on lifebalance

Related: [AI], [homelab]

Current host baseline

• Hostname: lifebalance
• Hardware: Lenovo Legion T7 34IRZ8, Intel Core i9-14900K, 62 GiB RAM, NVIDIA GeForce RTX 4090 (24 GiB VRAM), 1.9 TB NVMe (/home on a 1.6 TB logical volume)
• OS: Debian GNU/Linux forky/sid, kernel 6.18.15+deb14-amd64
• NVIDIA stack: driver 550.163.01, CUDA 12.4
• Container stack: Docker 29.3.1, containerd 2.2.2, NVIDIA Container Toolkit 1.19.0

Current AI status

• No AI model server is currently running on lifebalance.
• docker.service and containerd.service are running.
• No ollama binary was found.
• No conda install was found.
• Global python3 is 3.13.12.
• A dedicated ~/sglang virtualenv exists, but it currently only has pip and uv installed.

Evidence of previous setup

SGLang

• Virtualenv: ~/sglang
• Startup script: ~/sglang/bin/start-sglang.sh
• Intended launch command:

python -m sglang.launch_server \
  --model Qwen/Qwen3-8B-AWQ \
  --port 30000 \
  --host 127.0.0.1 \
  --mem-fraction-static 0.88 \
  --context-length 32768 \
  --dtype auto \
  --trust-remote-code

• Notes:
  • configured for localhost-only serving on port 30000
  • sized to use most of the 4090 VRAM
  • comment suggests it was intended for "oh-my-opencode utility agents"
  • the script comments still reference ~/scripts/start-sglang.sh, but the file currently lives in ~/sglang/bin/start-sglang.sh
  • the venv is currently incomplete, so this script would not run without reinstalling sglang

Unsloth (Docker)

• Shell history shows installation of the NVIDIA container toolkit followed by docker run ... --gpus all unsloth/unsloth.
• Current Docker artifacts:
  • image present: unsloth/unsloth:latest (~40.8 GB)
  • stopped container: brave_dewdney
  • published ports: 8888, 8000, 2222
  • bind mount: /home/ben/work:/workspace/work
• Notes:
  • /home/ben/work is currently empty
  • only /workspace/work was bind-mounted, so data under /workspace/.cache/huggingface would have lived inside the container layer instead of a persistent host path
  • shell history/container environment included a hard-coded Jupyter password; do not reuse that pattern without rotating the credential and moving it into a safer secret/env file flow

LocalAI Packaged / Ollama notes

• [AI] already contains notes for a local-ai-packaged stack with:
  • Open WebUI
  • n8n
  • Langfuse
  • Ollama
  • Qdrant
  • python start_services.py --profile gpu-nvidia
• I did not find:
  • a local checkout of that repo on lifebalance
  • a start_services.py file on this PC
  • an installed ollama binary
  • matching running containers or service units
• This likely refers to another machine, a deleted checkout, or a plan that was documented before the actual install happened.

Related but separate artifacts

• homefoam/local-ai-beepmill.code-workspace points at a missing ../local-ai-beepmill folder.
• code/homelab_tofu/vm_beepboop.tf defines a beepboop VM on proxmox1 with GPU passthrough, 12 GiB RAM, and a 150 GiB disk.
• beepboop may have been intended as a separate AI host, but there is not enough evidence on lifebalance to say it was actually used for this stack.

Gaps to resolve

These are the main unanswered execution details, not fundamental direction questions:

• pick the exact sglang container image/tag strategy for reproducible deployment
• decide whether model files should be stored explicitly under ${AI_DATA_ROOT}/models or primarily re-hydrated through Hugging Face cache paths
• define the exact Compose profile/default-start behavior for open-webui
• decide the local secret storage pattern outside git (.env, separate secret file, password manager, or similar)
• define the minimum backup policy for open-webui state, exported manifests, and any future adapters/checkpoints
• document the exact fresh-host prerequisite list before the first real deployment

Working plan v1

Planning assumptions

These are the current decisions for planning purposes:

• first success = a local-only inference API on lifebalance
• phase 1 should be Docker-first
• phase 1 should target Qwen3 8B AWQ first
• Open WebUI should come after the SGLang API is working
• lifebalance is now the canonical LLM host; beepboop is no longer the intended host for this project
• use a custom large-data storage root from day 1 instead of tool defaults
• plan for host recreation from the beginning
• keep reproducible stack-definition files in a separate dedicated repo/worktree, not inside this KB repo
• phase 1 should include:
  • OpenAI-style API access
  • model switching
  • a web UI
  • quantized models
• training/fine-tuning should come later, not in the initial stack

Recommended architecture

Use a phased hybrid approach:

• Phase 1 baseline: SGLang + Open WebUI
• Phase 2 convenience track: add Ollama only if easier model switching and broader casual experimentation become more important
• Phase 3 training track: keep Unsloth separate as an occasional training/fine-tuning workflow, not part of the always-on serving stack

Why this is the current recommendation

• The primary goal is a strong local inference API, not just a chat UI.
• SGLang fits that goal better than starting with an Ollama-first stack.
• There is already evidence of prior SGLang experimentation on this machine, so this path builds on existing intent.
• Open WebUI can still provide the human-facing UI without forcing Ollama to be the primary backend.
• This keeps the first phase local-only while optimizing earlier for agent/tool use.

Why not start with Ollama as the primary backend

• It is more convenient for casual local experimentation than for an API-first serving plan.
• It may still be useful later, but it is no longer the best fit for the primary phase-1 objective.
• We can add it later without redesigning the whole stack.

Why not start with SGLang as the only component

• A web UI still matters in phase 1.
• Open WebUI remains useful as the human-facing interface even if SGLang is the main backend.

Why not start with Unsloth

• The first outcome should be reliable inference, not training.
• Training adds larger storage, cache, dataset, and secret-management concerns.
• Keeping training separate reduces operational complexity.

Phased roadmap

Phase 1 — stable local workstation stack

Goal: a local-only, Docker-managed baseline for everyday use.

Planned components:

• sglang for local model serving
• open-webui as the follow-on chat/testing interface after the API baseline is working
• persistent host directories for:
  • models
  • app data
  • logs
  • exported configuration

Success criteria:

• at least one primary quantized model is served reliably
• the UI works locally on lifebalance
• the API is usable by local scripts/tools and is documented as the canonical local endpoint
• all services bind locally first, or are otherwise restricted to local use
• storage locations are explicit and documented before large downloads begin

Phase 2 — convenience and model-flexibility lane

Goal: add easier model switching and a broader casual-use path only if phase-1 usage shows it is worth the extra moving parts.

Planned changes:

• evaluate whether adding Ollama improves day-to-day experimentation enough to justify dual backends
• keep SGLang as the canonical agent/API backend unless there is a good reason to change that
• decide whether:
  • Open WebUI should talk to SGLang only
  • Open WebUI should talk to Ollama only
  • both backends should coexist for different purposes

Success criteria:

• any second backend has a clear purpose
• model/API behavior is predictable and documented
• added convenience does not make the core serving path harder to operate

Phase 3 — training and fine-tuning lane

Goal: support occasional LoRA/fine-tuning experiments without destabilizing the serving stack.

Planned components:

• a separate Unsloth workflow
• dedicated storage/caching locations for datasets, checkpoints, and Hugging Face assets
• clear separation between:
  • always-on inference services
  • temporary experiment/training workloads

Success criteria:

• training artifacts persist outside disposable containers
• secrets are not passed via shell history
• training jobs do not accidentally overwrite or disrupt inference setup

Planned implementation decisions

When work begins, prefer:

• one project directory for Compose files, env files, and documentation
• one persistent storage root for models/caches/data
• localhost-first networking before any LAN exposure
• explicit secrets handling instead of shell-history passwords
• one canonical source of truth in this note for ports, paths, and services

Recommended storage layout

Use a configurable storage root variable instead of hard-coding paths throughout the stack.

Recommended current value on lifebalance:

AI_DATA_ROOT=/home/ben/llm-data

Recommended layout under that root:

${AI_DATA_ROOT}/
  models/            # durable model files when not using opaque tool defaults
  huggingface/       # HF cache and downloads
  sglang/            # SGLang-specific state/config
  open-webui/        # Open WebUI persistent app data
  logs/              # service logs kept outside containers
  exports/           # saved manifests, model lists, config exports
  scratch/           # temporary working area

Rationale:

• lifebalance has a large /home logical volume, so /home/ben/llm-data is a practical first location.
• The path should still be treated as configurable so another host can recreate the same setup with a different mount point.
• The important invariant is the directory structure and variable name, not the literal absolute path.

Reproducibility-first approach

Treat the AI stack as something that should be rebuildable on a fresh host with minimal guesswork.

Preferred pattern:

• one small project directory for the stack definition
• one configurable data root for persistent state
• one env file for host-specific values
• one setup note as the source of truth for how the stack is intended to work

Project repo and workflow pivot

The implementation repo now exists at:

• /home/ben/code/ai-stack

Current repo state:

• git repo initialized
• default branch set to main
• OpenSpec initialized in the repo
• tool integrations generated for codex and opencode
• first OpenSpec change created: bootstrap-local-ai-stack

Repository intent:

• openspec/ becomes the spec/change-management home for implementation work
• this Foam note remains the KB-facing planning and operator-context note
• implementation artifacts should now be tracked in the repo, with this note linking back to them as needed

Current OpenSpec structure in the repo:

/home/ben/code/ai-stack/
  README.md
  openspec/
    changes/
      bootstrap-local-ai-stack/
        proposal.md
        design.md
        tasks.md
        specs/
          local-ai-stack/
            spec.md
    specs/
  .codex/
  .opencode/

Operational implication:

• future concrete stack work should start as OpenSpec changes in ai-stack
• Foam should reference the repo and summarize decisions, rather than becoming the only place where implementation structure lives
• the current baseline implementation change is openspec/changes/bootstrap-local-ai-stack/

Stack definition v1

This is the concrete definition for the separate stack repo/worktree that now exists outside this KB at /home/ben/code/ai-stack.

Scope of v1

Include in v1:

• one Docker Compose stack definition repo called ai-stack
• one always-on inference service: sglang
• one follow-on UI service: open-webui
• one shared host data root: ${AI_DATA_ROOT}
• localhost-only published ports by default
• enough docs and templates to recreate the stack on another host

Do not include in v1:

• ollama
• unsloth
• training jobs
• public LAN exposure by default
• extra observability/services unless a real need appears

Service topology

local tools / scripts
        |
        v
  127.0.0.1:30000
     sglang API
        |
        +---- model weights / HF cache / logs under ${AI_DATA_ROOT}

  127.0.0.1:3001
    Open WebUI
        |
        v
     talks to sglang over local Docker networking or local host endpoint

Operational intent:

• sglang is the canonical local inference backend
• open-webui is a convenience client on top of that backend, not the center of the stack
• if open-webui breaks, the API stack should still be usable directly

Repo/worktree layout

ai-stack/
  compose.yaml
  .env.example
  .gitignore
  README.md
  docs/
    architecture.md
    recovery-checklist.md
    model-inventory.md
  sglang/
    launch.env.example
    notes.md
  open-webui/
    notes.md
  scripts/
    preflight-checklist.md

File intent:

• compose.yaml = the canonical service definition
• .env.example = documented host-specific variables with placeholder values only
• .gitignore = protect .env, logs, caches, exports, and downloaded assets from accidental commits
• README.md = operator-first quickstart for bring-up, shutdown, update, and rebuild
• docs/architecture.md = why the stack is shaped this way
• docs/recovery-checklist.md = fresh-host rebuild order
• docs/model-inventory.md = canonical list of intended models/adapters and where they are expected to come from
• sglang/launch.env.example = model/server launch parameters that may vary by host or model choice
• sglang/notes.md = operational notes for the inference service
• open-webui/notes.md = operational notes for the UI layer
• scripts/preflight-checklist.md = manual prerequisites/validation list before first start

Compose contract

Initial Compose services:

• sglang
  • required in v1
  • NVIDIA-enabled container
  • publishes 127.0.0.1:30000:30000
  • mounts model/cache/state/log paths from ${AI_DATA_ROOT}
  • uses Qwen/Qwen3-8B-AWQ as the first default model
  • should be startable independently of any UI
• open-webui
  • optional during initial API bring-up, but part of the v1 repo definition
  • publishes 127.0.0.1:3001:8080
  • persists app state under ${AI_DATA_ROOT}/open-webui
  • points at the existing sglang OpenAI-compatible endpoint

Recommended Compose profiles:

• default / baseline: sglang
• optional UI profile: open-webui

Meaning:

• the inference API can come up first by itself
• the UI can be layered on later without redefining the stack

Host path contract

AI_DATA_ROOT remains the root for all durable local state.

${AI_DATA_ROOT}/
  models/            # model files if intentionally stored outside tool-managed caches
  huggingface/       # HF_HOME / HF hub cache
  sglang/            # sglang-specific state, configs, temp runtime data
  open-webui/        # persistent UI state
  logs/              # host-visible logs
  exports/           # inventories, backup manifests, saved config exports
  scratch/           # temporary manual work area

Path rules:

• containers should only write durable state inside ${AI_DATA_ROOT}
• container-internal default paths should be overridden where practical
• logs worth keeping should land on the host, not only inside container stdout history

Env contract

The real .env should stay outside git. .env.example should document the contract.

Expected variables for v1:

AI_DATA_ROOT=/home/ben/llm-data
SGLANG_HOST_PORT=30000
OPEN_WEBUI_HOST_PORT=3001
SGLANG_MODEL=Qwen/Qwen3-8B-AWQ
SGLANG_CONTEXT_LENGTH=32768
SGLANG_MEM_FRACTION_STATIC=0.88
SGLANG_DTYPE=auto
HF_HOME=/home/ben/llm-data/huggingface
OPEN_WEBUI_DATA_DIR=/home/ben/llm-data/open-webui

Notes:

• secrets should be added only if/when a service actually needs them
• avoid putting host-specific absolute paths directly into compose.yaml when a variable will do
• if another host replaces lifebalance, the main required change should be env values, not Compose structure

Canonical endpoints and names

Use these as the first-pass defaults unless execution reveals a conflict:

• stack repo/worktree name: ai-stack
• primary service name: sglang
• UI service name: open-webui
• canonical local inference endpoint: http://127.0.0.1:30000/v1
• canonical local UI URL: http://127.0.0.1:3001
• primary model identifier: Qwen/Qwen3-8B-AWQ

Operating rules for v1

• prefer explicit image tags over floating latest where possible
• prefer localhost binding over broad exposure
• keep sglang usable without open-webui
• treat model downloads as reproducible inputs, not irreplaceable state
• treat UI state, manifests, and future adapters/checkpoints as state worth intentional preservation
• do not add routing until there is a real second-model need

Future extension points

Planned later, but not part of the initial stack definition:

• ollama as a separate convenience lane if model switching becomes more valuable than keeping one backend
• a routing/gateway layer only after there are multiple active model backends to justify it
• unsloth as a separate workflow/repo path for training or fine-tuning

What should be reproducible vs backed up

Rebuildable from config:

• container definitions
• image tags
• port mappings
• bind mounts
• service-level configuration
• startup commands
• model identifiers to download

Back up or preserve intentionally:

• secrets outside git
• exported model inventory/manifests
• any local UI/app data worth keeping
• any future training outputs, datasets, adapters, or checkpoints

Usually re-download instead of backing up:

• public base model weights
• Hugging Face cache contents
• container images

Optional exception:

• if download time becomes painful, a local backup of model files may still be worth doing, but that should be a conscious backup policy rather than an accident of where files ended up

What belongs in git vs not in git

Commit:

• Compose files
• config templates
• .env.example
• setup scripts that do not contain secrets
• documentation, recovery notes, and model manifests

Never commit:

• .env with real values
• API tokens, auth secrets, passwords
• downloaded model weights
• Hugging Face cache directories
• logs
• databases/app state unless deliberately exporting sanitized fixtures

Implementation checklist (draft)

This is a future execution checklist only. It does not mean these steps have been performed yet.

1. Create the reproducible stack-definition repo/worktree

• create a dedicated repo/worktree for the AI stack definition
• add compose.yaml
• add .env.example
• add a README.md with recovery/rebuild instructions
• add config templates for sglang and later open-webui
• add .gitignore rules for secrets, logs, caches, and downloaded assets

2. Define host-specific variables and conventions

• define AI_DATA_ROOT
• define canonical service names, ports, and container names
• define the canonical local API endpoint for sglang
• define where logs should live on the host
• define the host-specific values that belong in .env rather than in committed files

3. Create the persistent storage layout

• create ${AI_DATA_ROOT}
• create ${AI_DATA_ROOT}/models
• create ${AI_DATA_ROOT}/huggingface
• create ${AI_DATA_ROOT}/sglang
• create ${AI_DATA_ROOT}/open-webui
• create ${AI_DATA_ROOT}/logs
• create ${AI_DATA_ROOT}/exports
• create ${AI_DATA_ROOT}/scratch
• document ownership/permissions expectations for these paths

4. Prepare secrets and env handling

• decide how local secrets will be stored outside git
• create .env from .env.example
• ensure no passwords or tokens are passed through shell history
• document which values are required for a fresh-host rebuild

5. Stand up the SGLang API baseline

• define the sglang image/tag strategy
• define bind mounts for model/cache/log locations
• define the startup command for Qwen3 8B AWQ
• keep networking local-only by default
• document how the service is started, stopped, restarted, and updated
• document how the model is fetched or made available reproducibly

6. Validate the API baseline

• confirm the service starts cleanly
• confirm the model loads successfully on the 4090
• confirm the local OpenAI-compatible endpoint works
• confirm logs are written to the expected host path
• document basic smoke tests for future rebuild validation

7. Add Open WebUI after the API baseline is stable

• add open-webui to the stack-definition repo/worktree
• bind its persistent data to ${AI_DATA_ROOT}/open-webui
• connect it to the existing sglang API
• document the intended UI URL and local-only access model
• document what configuration is UI state vs committed config

8. Capture disaster-recovery details

• document host prerequisites required before stack deployment
• document the exact rebuild order on a fresh machine
• document what must be restored from backup vs re-downloaded
• export a model/config inventory into ${AI_DATA_ROOT}/exports
• add a short recovery checklist for "lifebalance died, rebuild elsewhere"

9. Leave room for later phases

• define how a future ollama lane would coexist without replacing sglang as the primary API
• define how a future unsloth workflow would keep training artifacts separate from inference services
• document what additional storage/backup policy would be needed for adapters, datasets, and checkpoints