homefoam/04-knowledge/topics/self-hosting_AI.md

# self-hosting AI on lifebalance

Related: [[AI]], [[homelab]]

## Current host baseline

* Hostname: `lifebalance`
* Hardware: Lenovo Legion T7 34IRZ8, Intel Core i9-14900K, 62 GiB RAM, NVIDIA GeForce RTX 4090 (24 GiB VRAM), 1.9 TB NVMe (`/home` on a 1.6 TB logical volume)
* OS: Debian GNU/Linux forky/sid, kernel `6.18.15+deb14-amd64`
* NVIDIA stack: driver `550.163.01`, CUDA `12.4`
* Container stack: Docker `29.3.1`, containerd `2.2.2`, NVIDIA Container Toolkit `1.19.0`

## Current AI status

* No AI model server is currently running on `lifebalance`.
* `docker.service` and `containerd.service` are running.
* No `ollama` binary was found.
* No `conda` install was found.
* Global `python3` is `3.13.12`.
* A dedicated `~/sglang` virtualenv exists, but it currently only has `pip` and `uv` installed.

## Evidence of previous setup

### SGLang

* Virtualenv: `~/sglang`
* Startup script: `~/sglang/bin/start-sglang.sh`
* Intended launch command:

```shell
python -m sglang.launch_server \
  --model Qwen/Qwen3-8B-AWQ \
  --port 30000 \
  --host 127.0.0.1 \
  --mem-fraction-static 0.88 \
  --context-length 32768 \
  --dtype auto \
  --trust-remote-code
```

* Notes:
  * configured for localhost-only serving on port `30000`
  * sized to use most of the 4090 VRAM
  * comment suggests it was intended for "oh-my-opencode utility agents"
  * the script comments still reference `~/scripts/start-sglang.sh`, but the file currently lives in `~/sglang/bin/start-sglang.sh`
  * the venv is currently incomplete, so this script would not run without reinstalling `sglang`

### Unsloth (Docker)

* Shell history shows installation of the NVIDIA container toolkit followed by `docker run ... --gpus all unsloth/unsloth`.
* Current Docker artifacts:
  * image present: `unsloth/unsloth:latest` (~40.8 GB)
  * stopped container: `brave_dewdney`
  * published ports: `8888`, `8000`, `2222`
  * bind mount: `/home/ben/work:/workspace/work`
* Notes:
  * `/home/ben/work` is currently empty
  * only `/workspace/work` was bind-mounted, so data under `/workspace/.cache/huggingface` would have lived inside the container layer instead of a persistent host path
  * shell history/container environment included a hard-coded Jupyter password; do not reuse that pattern without rotating the credential and moving it into a safer secret/env file flow

### LocalAI Packaged / Ollama notes

* [[AI]] already contains notes for a `local-ai-packaged` stack with:
  * Open WebUI
  * n8n
  * Langfuse
  * Ollama
  * Qdrant
  * `python start_services.py --profile gpu-nvidia`
* I did not find:
  * a local checkout of that repo on `lifebalance`
  * a `start_services.py` file on this PC
  * an installed `ollama` binary
  * matching running containers or service units
* This likely refers to another machine, a deleted checkout, or a plan that was documented before the actual install happened.

### Related but separate artifacts

* `homefoam/local-ai-beepmill.code-workspace` points at a missing `../local-ai-beepmill` folder.
* `code/homelab_tofu/vm_beepboop.tf` defines a `beepboop` VM on `proxmox1` with GPU passthrough, 12 GiB RAM, and a 150 GiB disk.
* `beepboop` may have been intended as a separate AI host, but there is not enough evidence on `lifebalance` to say it was actually used for this stack.

## Gaps to resolve

These are the main unanswered execution details, not fundamental direction questions:

* pick the exact `sglang` container image/tag strategy for reproducible deployment
* decide whether model files should be stored explicitly under `${AI_DATA_ROOT}/models` or primarily re-hydrated through Hugging Face cache paths
* define the exact Compose profile/default-start behavior for `open-webui`
* decide the local secret storage pattern outside git (`.env`, separate secret file, password manager, or similar)
* define the minimum backup policy for `open-webui` state, exported manifests, and any future adapters/checkpoints
* document the exact fresh-host prerequisite list before the first real deployment

## Working plan v1

### Planning assumptions

These are the current decisions for planning purposes:

* first success = a local-only inference API on `lifebalance`
* phase 1 should be Docker-first
* phase 1 should target `Qwen3 8B AWQ` first
* `Open WebUI` should come after the `SGLang` API is working
* `lifebalance` is now the canonical LLM host; `beepboop` is no longer the intended host for this project
* use a custom large-data storage root from day 1 instead of tool defaults
* plan for host recreation from the beginning
* keep reproducible stack-definition files in a separate dedicated repo/worktree, not inside this KB repo
* phase 1 should include:
  * OpenAI-style API access
  * model switching
  * a web UI
  * quantized models
* training/fine-tuning should come later, not in the initial stack

### Recommended architecture

Use a phased hybrid approach:

* **Phase 1 baseline:** `SGLang + Open WebUI`
* **Phase 2 convenience track:** add `Ollama` only if easier model switching and broader casual experimentation become more important
* **Phase 3 training track:** keep `Unsloth` separate as an occasional training/fine-tuning workflow, not part of the always-on serving stack

### Why this is the current recommendation

* The primary goal is a strong local inference API, not just a chat UI.
* `SGLang` fits that goal better than starting with an Ollama-first stack.
* There is already evidence of prior `SGLang` experimentation on this machine, so this path builds on existing intent.
* `Open WebUI` can still provide the human-facing UI without forcing Ollama to be the primary backend.
* This keeps the first phase local-only while optimizing earlier for agent/tool use.

### Why not start with Ollama as the primary backend

* It is more convenient for casual local experimentation than for an API-first serving plan.
* It may still be useful later, but it is no longer the best fit for the primary phase-1 objective.
* We can add it later without redesigning the whole stack.

### Why not start with SGLang as the only component

* A web UI still matters in phase 1.
* `Open WebUI` remains useful as the human-facing interface even if `SGLang` is the main backend.

### Why not start with Unsloth

* The first outcome should be reliable inference, not training.
* Training adds larger storage, cache, dataset, and secret-management concerns.
* Keeping training separate reduces operational complexity.

## Phased roadmap

### Phase 1 — stable local workstation stack

Goal: a local-only, Docker-managed baseline for everyday use.

Planned components:

* `sglang` for local model serving
* `open-webui` as the follow-on chat/testing interface after the API baseline is working
* persistent host directories for:
  * models
  * app data
  * logs
  * exported configuration

Success criteria:

* at least one primary quantized model is served reliably
* the UI works locally on `lifebalance`
* the API is usable by local scripts/tools and is documented as the canonical local endpoint
* all services bind locally first, or are otherwise restricted to local use
* storage locations are explicit and documented before large downloads begin

### Phase 2 — convenience and model-flexibility lane

Goal: add easier model switching and a broader casual-use path only if phase-1 usage shows it is worth the extra moving parts.

Planned changes:

* evaluate whether adding `Ollama` improves day-to-day experimentation enough to justify dual backends
* keep `SGLang` as the canonical agent/API backend unless there is a good reason to change that
* decide whether:
  * `Open WebUI` should talk to SGLang only
  * `Open WebUI` should talk to Ollama only
  * both backends should coexist for different purposes

Success criteria:

* any second backend has a clear purpose
* model/API behavior is predictable and documented
* added convenience does not make the core serving path harder to operate

### Phase 3 — training and fine-tuning lane

Goal: support occasional LoRA/fine-tuning experiments without destabilizing the serving stack.

Planned components:

* a separate `Unsloth` workflow
* dedicated storage/caching locations for datasets, checkpoints, and Hugging Face assets
* clear separation between:
  * always-on inference services
  * temporary experiment/training workloads

Success criteria:

* training artifacts persist outside disposable containers
* secrets are not passed via shell history
* training jobs do not accidentally overwrite or disrupt inference setup

## Planned implementation decisions

When work begins, prefer:

* one project directory for Compose files, env files, and documentation
* one persistent storage root for models/caches/data
* localhost-first networking before any LAN exposure
* explicit secrets handling instead of shell-history passwords
* one canonical source of truth in this note for ports, paths, and services

### Recommended storage layout

Use a configurable storage root variable instead of hard-coding paths throughout the stack.

Recommended current value on `lifebalance`:

```shell
AI_DATA_ROOT=/home/ben/llm-data
```

Recommended layout under that root:

```text
${AI_DATA_ROOT}/
  models/            # durable model files when not using opaque tool defaults
  huggingface/       # HF cache and downloads
  sglang/            # SGLang-specific state/config
  open-webui/        # Open WebUI persistent app data
  logs/              # service logs kept outside containers
  exports/           # saved manifests, model lists, config exports
  scratch/           # temporary working area
```

Rationale:

* `lifebalance` has a large `/home` logical volume, so `/home/ben/llm-data` is a practical first location.
* The path should still be treated as configurable so another host can recreate the same setup with a different mount point.
* The important invariant is the directory structure and variable name, not the literal absolute path.

### Reproducibility-first approach

Treat the AI stack as something that should be rebuildable on a fresh host with minimal guesswork.

Preferred pattern:

* one small project directory for the stack definition
* one configurable data root for persistent state
* one env file for host-specific values
* one setup note as the source of truth for how the stack is intended to work

### Project repo and workflow pivot

The implementation repo now exists at:

* `/home/ben/code/ai-stack`

Current repo state:

* git repo initialized
* default branch set to `main`
* OpenSpec initialized in the repo
* tool integrations generated for `codex` and `opencode`
* first OpenSpec change created: `bootstrap-local-ai-stack`

Repository intent:

* `openspec/` becomes the spec/change-management home for implementation work
* this Foam note remains the KB-facing planning and operator-context note
* implementation artifacts should now be tracked in the repo, with this note linking back to them as needed

Current OpenSpec structure in the repo:

```text
/home/ben/code/ai-stack/
  README.md
  openspec/
    changes/
      bootstrap-local-ai-stack/
        proposal.md
        design.md
        tasks.md
        specs/
          local-ai-stack/
            spec.md
    specs/
  .codex/
  .opencode/
```

Operational implication:

* future concrete stack work should start as OpenSpec changes in `ai-stack`
* Foam should reference the repo and summarize decisions, rather than becoming the only place where implementation structure lives
* the current baseline implementation change is `openspec/changes/bootstrap-local-ai-stack/`

## Stack definition v1

This is the concrete definition for the separate stack repo/worktree that now exists outside this KB at `/home/ben/code/ai-stack`.

### Scope of v1

Include in v1:

* one Docker Compose stack definition repo called `ai-stack`
* one always-on inference service: `sglang`
* one follow-on UI service: `open-webui`
* one shared host data root: `${AI_DATA_ROOT}`
* localhost-only published ports by default
* enough docs and templates to recreate the stack on another host

Do **not** include in v1:

* `ollama`
* `unsloth`
* training jobs
* public LAN exposure by default
* extra observability/services unless a real need appears

### Service topology

```text
local tools / scripts
        |
        v
  127.0.0.1:30000
     sglang API
        |
        +---- model weights / HF cache / logs under ${AI_DATA_ROOT}

  127.0.0.1:3001
    Open WebUI
        |
        v
     talks to sglang over local Docker networking or local host endpoint
```

Operational intent:

* `sglang` is the canonical local inference backend
* `open-webui` is a convenience client on top of that backend, not the center of the stack
* if `open-webui` breaks, the API stack should still be usable directly

### Repo/worktree layout

```text
ai-stack/
  compose.yaml
  .env.example
  .gitignore
  README.md
  docs/
    architecture.md
    recovery-checklist.md
    model-inventory.md
  sglang/
    launch.env.example
    notes.md
  open-webui/
    notes.md
  scripts/
    preflight-checklist.md
```

File intent:

* `compose.yaml` = the canonical service definition
* `.env.example` = documented host-specific variables with placeholder values only
* `.gitignore` = protect `.env`, logs, caches, exports, and downloaded assets from accidental commits
* `README.md` = operator-first quickstart for bring-up, shutdown, update, and rebuild
* `docs/architecture.md` = why the stack is shaped this way
* `docs/recovery-checklist.md` = fresh-host rebuild order
* `docs/model-inventory.md` = canonical list of intended models/adapters and where they are expected to come from
* `sglang/launch.env.example` = model/server launch parameters that may vary by host or model choice
* `sglang/notes.md` = operational notes for the inference service
* `open-webui/notes.md` = operational notes for the UI layer
* `scripts/preflight-checklist.md` = manual prerequisites/validation list before first start

### Compose contract

Initial Compose services:

* `sglang`
  * required in v1
  * NVIDIA-enabled container
  * publishes `127.0.0.1:30000:30000`
  * mounts model/cache/state/log paths from `${AI_DATA_ROOT}`
  * uses `Qwen/Qwen3-8B-AWQ` as the first default model
  * should be startable independently of any UI
* `open-webui`
  * optional during initial API bring-up, but part of the v1 repo definition
  * publishes `127.0.0.1:3001:8080`
  * persists app state under `${AI_DATA_ROOT}/open-webui`
  * points at the existing `sglang` OpenAI-compatible endpoint

Recommended Compose profiles:

* default / baseline: `sglang`
* optional UI profile: `open-webui`

Meaning:

* the inference API can come up first by itself
* the UI can be layered on later without redefining the stack

### Host path contract

`AI_DATA_ROOT` remains the root for all durable local state.

```text
${AI_DATA_ROOT}/
  models/            # model files if intentionally stored outside tool-managed caches
  huggingface/       # HF_HOME / HF hub cache
  sglang/            # sglang-specific state, configs, temp runtime data
  open-webui/        # persistent UI state
  logs/              # host-visible logs
  exports/           # inventories, backup manifests, saved config exports
  scratch/           # temporary manual work area
```

Path rules:

* containers should only write durable state inside `${AI_DATA_ROOT}`
* container-internal default paths should be overridden where practical
* logs worth keeping should land on the host, not only inside container stdout history

### Env contract

The real `.env` should stay outside git. `.env.example` should document the contract.

Expected variables for v1:

```dotenv
AI_DATA_ROOT=/home/ben/llm-data
SGLANG_HOST_PORT=30000
OPEN_WEBUI_HOST_PORT=3001
SGLANG_MODEL=Qwen/Qwen3-8B-AWQ
SGLANG_CONTEXT_LENGTH=32768
SGLANG_MEM_FRACTION_STATIC=0.88
SGLANG_DTYPE=auto
HF_HOME=/home/ben/llm-data/huggingface
OPEN_WEBUI_DATA_DIR=/home/ben/llm-data/open-webui
```

Notes:

* secrets should be added only if/when a service actually needs them
* avoid putting host-specific absolute paths directly into `compose.yaml` when a variable will do
* if another host replaces `lifebalance`, the main required change should be env values, not Compose structure

### Canonical endpoints and names

Use these as the first-pass defaults unless execution reveals a conflict:

* stack repo/worktree name: `ai-stack`
* primary service name: `sglang`
* UI service name: `open-webui`
* canonical local inference endpoint: `http://127.0.0.1:30000/v1`
* canonical local UI URL: `http://127.0.0.1:3001`
* primary model identifier: `Qwen/Qwen3-8B-AWQ`

### Operating rules for v1

* prefer explicit image tags over floating `latest` where possible
* prefer localhost binding over broad exposure
* keep `sglang` usable without `open-webui`
* treat model downloads as reproducible inputs, not irreplaceable state
* treat UI state, manifests, and future adapters/checkpoints as state worth intentional preservation
* do not add routing until there is a real second-model need

### Future extension points

Planned later, but not part of the initial stack definition:

* `ollama` as a separate convenience lane if model switching becomes more valuable than keeping one backend
* a routing/gateway layer only after there are multiple active model backends to justify it
* `unsloth` as a separate workflow/repo path for training or fine-tuning

### What should be reproducible vs backed up

Rebuildable from config:

* container definitions
* image tags
* port mappings
* bind mounts
* service-level configuration
* startup commands
* model identifiers to download

Back up or preserve intentionally:

* secrets outside git
* exported model inventory/manifests
* any local UI/app data worth keeping
* any future training outputs, datasets, adapters, or checkpoints

Usually re-download instead of backing up:

* public base model weights
* Hugging Face cache contents
* container images

Optional exception:

* if download time becomes painful, a local backup of model files may still be worth doing, but that should be a conscious backup policy rather than an accident of where files ended up

### What belongs in git vs not in git

Commit:

* Compose files
* config templates
* `.env.example`
* setup scripts that do not contain secrets
* documentation, recovery notes, and model manifests

Never commit:

* `.env` with real values
* API tokens, auth secrets, passwords
* downloaded model weights
* Hugging Face cache directories
* logs
* databases/app state unless deliberately exporting sanitized fixtures

## Implementation checklist (draft)

This is a future execution checklist only. It does **not** mean these steps have been performed yet.

### 1. Create the reproducible stack-definition repo/worktree

- [ ] create a dedicated repo/worktree for the AI stack definition
- [ ] add `compose.yaml`
- [ ] add `.env.example`
- [ ] add a `README.md` with recovery/rebuild instructions
- [ ] add config templates for `sglang` and later `open-webui`
- [ ] add `.gitignore` rules for secrets, logs, caches, and downloaded assets

### 2. Define host-specific variables and conventions

- [ ] define `AI_DATA_ROOT`
- [ ] define canonical service names, ports, and container names
- [ ] define the canonical local API endpoint for `sglang`
- [ ] define where logs should live on the host
- [ ] define the host-specific values that belong in `.env` rather than in committed files

### 3. Create the persistent storage layout

- [ ] create `${AI_DATA_ROOT}`
- [ ] create `${AI_DATA_ROOT}/models`
- [ ] create `${AI_DATA_ROOT}/huggingface`
- [ ] create `${AI_DATA_ROOT}/sglang`
- [ ] create `${AI_DATA_ROOT}/open-webui`
- [ ] create `${AI_DATA_ROOT}/logs`
- [ ] create `${AI_DATA_ROOT}/exports`
- [ ] create `${AI_DATA_ROOT}/scratch`
- [ ] document ownership/permissions expectations for these paths

### 4. Prepare secrets and env handling

- [ ] decide how local secrets will be stored outside git
- [ ] create `.env` from `.env.example`
- [ ] ensure no passwords or tokens are passed through shell history
- [ ] document which values are required for a fresh-host rebuild

### 5. Stand up the SGLang API baseline

- [ ] define the `sglang` image/tag strategy
- [ ] define bind mounts for model/cache/log locations
- [ ] define the startup command for `Qwen3 8B AWQ`
- [ ] keep networking local-only by default
- [ ] document how the service is started, stopped, restarted, and updated
- [ ] document how the model is fetched or made available reproducibly

### 6. Validate the API baseline

- [ ] confirm the service starts cleanly
- [ ] confirm the model loads successfully on the 4090
- [ ] confirm the local OpenAI-compatible endpoint works
- [ ] confirm logs are written to the expected host path
- [ ] document basic smoke tests for future rebuild validation

### 7. Add Open WebUI after the API baseline is stable

- [ ] add `open-webui` to the stack-definition repo/worktree
- [ ] bind its persistent data to `${AI_DATA_ROOT}/open-webui`
- [ ] connect it to the existing `sglang` API
- [ ] document the intended UI URL and local-only access model
- [ ] document what configuration is UI state vs committed config

### 8. Capture disaster-recovery details

- [ ] document host prerequisites required before stack deployment
- [ ] document the exact rebuild order on a fresh machine
- [ ] document what must be restored from backup vs re-downloaded
- [ ] export a model/config inventory into `${AI_DATA_ROOT}/exports`
- [ ] add a short recovery checklist for "lifebalance died, rebuild elsewhere"

### 9. Leave room for later phases

- [ ] define how a future `ollama` lane would coexist without replacing `sglang` as the primary API
- [ ] define how a future `unsloth` workflow would keep training artifacts separate from inference services
- [ ] document what additional storage/backup policy would be needed for adapters, datasets, and checkpoints


[AI]: AI.md "AI"
[homelab]: homelab.md "homelab"