ben/homefoam
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync

Browse Source
This commit is contained in:
Ben
2026-09-07 11:52:50 -06:00
parent d554c64d00
commit d304fd0e6e
11 changed files with 975 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
self-hosting AI on lifebalance.md
+616
View File
@@ -0,0 +1,616 @@
# self-hosting AI on lifebalance
Related: [[AI]], [[homelab]]
## Current host baseline
* Hostname: `lifebalance`
* Hardware: Lenovo Legion T7 34IRZ8, Intel Core i9-14900K, 62 GiB RAM, NVIDIA GeForce RTX 4090 (24 GiB VRAM), 1.9 TB NVMe (`/home` on a 1.6 TB logical volume)
* OS: Debian GNU/Linux forky/sid, kernel `6.18.15+deb14-amd64`
* NVIDIA stack: driver `550.163.01`, CUDA `12.4`
* Container stack: Docker `29.3.1`, containerd `2.2.2`, NVIDIA Container Toolkit `1.19.0`
## Current AI status
* No AI model server is currently running on `lifebalance`.
* `docker.service` and `containerd.service` are running.
* No `ollama` binary was found.
* No `conda` install was found.
* Global `python3` is `3.13.12`.
* A dedicated `~/sglang` virtualenv exists, but it currently only has `pip` and `uv` installed.
## Evidence of previous setup
### SGLang
* Virtualenv: `~/sglang`
* Startup script: `~/sglang/bin/start-sglang.sh`
* Intended launch command:
```shell
python -m sglang.launch_server \
--model Qwen/Qwen3-8B-AWQ \
--port 30000 \
--host 127.0.0.1 \
--mem-fraction-static 0.88 \
--context-length 32768 \
--dtype auto \
--trust-remote-code
```
* Notes:
* configured for localhost-only serving on port `30000`
* sized to use most of the 4090 VRAM
* comment suggests it was intended for "oh-my-opencode utility agents"
* the script comments still reference `~/scripts/start-sglang.sh`, but the file currently lives in `~/sglang/bin/start-sglang.sh`
* the venv is currently incomplete, so this script would not run without reinstalling `sglang`
### Unsloth (Docker)
* Shell history shows installation of the NVIDIA container toolkit followed by `docker run ... --gpus all unsloth/unsloth`.
* Current Docker artifacts:
* image present: `unsloth/unsloth:latest` (~40.8 GB)
* stopped container: `brave_dewdney`
* published ports: `8888`, `8000`, `2222`
* bind mount: `/home/ben/work:/workspace/work`
* Notes:
* `/home/ben/work` is currently empty
* only `/workspace/work` was bind-mounted, so data under `/workspace/.cache/huggingface` would have lived inside the container layer instead of a persistent host path
* shell history/container environment included a hard-coded Jupyter password; do not reuse that pattern without rotating the credential and moving it into a safer secret/env file flow
### LocalAI Packaged / Ollama notes
* [[AI]] already contains notes for a `local-ai-packaged` stack with:
* Open WebUI
* n8n
* Langfuse
* Ollama
* Qdrant
* `python start_services.py --profile gpu-nvidia`
* I did not find:
* a local checkout of that repo on `lifebalance`
* a `start_services.py` file on this PC
* an installed `ollama` binary
* matching running containers or service units
* This likely refers to another machine, a deleted checkout, or a plan that was documented before the actual install happened.
### Related but separate artifacts
* `homefoam/local-ai-beepmill.code-workspace` points at a missing `../local-ai-beepmill` folder.
* `code/homelab_tofu/vm_beepboop.tf` defines a `beepboop` VM on `proxmox1` with GPU passthrough, 12 GiB RAM, and a 150 GiB disk.
* `beepboop` may have been intended as a separate AI host, but there is not enough evidence on `lifebalance` to say it was actually used for this stack.
## Gaps to resolve
These are the main unanswered execution details, not fundamental direction questions:
* pick the exact `sglang` container image/tag strategy for reproducible deployment
* decide whether model files should be stored explicitly under `${AI_DATA_ROOT}/models` or primarily re-hydrated through Hugging Face cache paths
* define the exact Compose profile/default-start behavior for `open-webui`
* decide the local secret storage pattern outside git (`.env`, separate secret file, password manager, or similar)
* define the minimum backup policy for `open-webui` state, exported manifests, and any future adapters/checkpoints
* document the exact fresh-host prerequisite list before the first real deployment
## Working plan v1
### Planning assumptions
These are the current decisions for planning purposes:
* first success = a local-only inference API on `lifebalance`
* phase 1 should be Docker-first
* phase 1 should target `Qwen3 8B AWQ` first
* `Open WebUI` should come after the `SGLang` API is working
* `lifebalance` is now the canonical LLM host; `beepboop` is no longer the intended host for this project
* use a custom large-data storage root from day 1 instead of tool defaults
* plan for host recreation from the beginning
* keep reproducible stack-definition files in a separate dedicated repo/worktree, not inside this KB repo
* phase 1 should include:
* OpenAI-style API access
* model switching
* a web UI
* quantized models
* training/fine-tuning should come later, not in the initial stack
### Recommended architecture
Use a phased hybrid approach:
* **Phase 1 baseline:** `SGLang + Open WebUI`
* **Phase 2 convenience track:** add `Ollama` only if easier model switching and broader casual experimentation become more important
* **Phase 3 training track:** keep `Unsloth` separate as an occasional training/fine-tuning workflow, not part of the always-on serving stack
### Why this is the current recommendation
* The primary goal is a strong local inference API, not just a chat UI.
* `SGLang` fits that goal better than starting with an Ollama-first stack.
* There is already evidence of prior `SGLang` experimentation on this machine, so this path builds on existing intent.
* `Open WebUI` can still provide the human-facing UI without forcing Ollama to be the primary backend.
* This keeps the first phase local-only while optimizing earlier for agent/tool use.
### Why not start with Ollama as the primary backend
* It is more convenient for casual local experimentation than for an API-first serving plan.
* It may still be useful later, but it is no longer the best fit for the primary phase-1 objective.
* We can add it later without redesigning the whole stack.
### Why not start with SGLang as the only component
* A web UI still matters in phase 1.
* `Open WebUI` remains useful as the human-facing interface even if `SGLang` is the main backend.
### Why not start with Unsloth
* The first outcome should be reliable inference, not training.
* Training adds larger storage, cache, dataset, and secret-management concerns.
* Keeping training separate reduces operational complexity.
## Phased roadmap
### Phase 1 — stable local workstation stack
Goal: a local-only, Docker-managed baseline for everyday use.
Planned components:
* `sglang` for local model serving
* `open-webui` as the follow-on chat/testing interface after the API baseline is working
* persistent host directories for:
* models
* app data
* logs
* exported configuration
Success criteria:
* at least one primary quantized model is served reliably
* the UI works locally on `lifebalance`
* the API is usable by local scripts/tools and is documented as the canonical local endpoint
* all services bind locally first, or are otherwise restricted to local use
* storage locations are explicit and documented before large downloads begin
### Phase 2 — convenience and model-flexibility lane
Goal: add easier model switching and a broader casual-use path only if phase-1 usage shows it is worth the extra moving parts.
Planned changes:
* evaluate whether adding `Ollama` improves day-to-day experimentation enough to justify dual backends
* keep `SGLang` as the canonical agent/API backend unless there is a good reason to change that
* decide whether:
* `Open WebUI` should talk to SGLang only
* `Open WebUI` should talk to Ollama only
* both backends should coexist for different purposes
Success criteria:
* any second backend has a clear purpose
* model/API behavior is predictable and documented
* added convenience does not make the core serving path harder to operate
### Phase 3 — training and fine-tuning lane
Goal: support occasional LoRA/fine-tuning experiments without destabilizing the serving stack.
Planned components:
* a separate `Unsloth` workflow
* dedicated storage/caching locations for datasets, checkpoints, and Hugging Face assets
* clear separation between:
* always-on inference services
* temporary experiment/training workloads
Success criteria:
* training artifacts persist outside disposable containers
* secrets are not passed via shell history
* training jobs do not accidentally overwrite or disrupt inference setup
## Planned implementation decisions
When work begins, prefer:
* one project directory for Compose files, env files, and documentation
* one persistent storage root for models/caches/data
* localhost-first networking before any LAN exposure
* explicit secrets handling instead of shell-history passwords
* one canonical source of truth in this note for ports, paths, and services
### Recommended storage layout
Use a configurable storage root variable instead of hard-coding paths throughout the stack.
Recommended current value on `lifebalance`:
```shell
AI_DATA_ROOT=/home/ben/llm-data
```
Recommended layout under that root:
```text
${AI_DATA_ROOT}/
models/ # durable model files when not using opaque tool defaults
huggingface/ # HF cache and downloads
sglang/ # SGLang-specific state/config
open-webui/ # Open WebUI persistent app data
logs/ # service logs kept outside containers
exports/ # saved manifests, model lists, config exports
scratch/ # temporary working area
```
Rationale:
* `lifebalance` has a large `/home` logical volume, so `/home/ben/llm-data` is a practical first location.
* The path should still be treated as configurable so another host can recreate the same setup with a different mount point.
* The important invariant is the directory structure and variable name, not the literal absolute path.
### Reproducibility-first approach
Treat the AI stack as something that should be rebuildable on a fresh host with minimal guesswork.
Preferred pattern:
* one small project directory for the stack definition
* one configurable data root for persistent state
* one env file for host-specific values
* one setup note as the source of truth for how the stack is intended to work
### Project repo and workflow pivot
The implementation repo now exists at:
* `/home/ben/code/ai-stack`
Current repo state:
* git repo initialized
* default branch set to `main`
* OpenSpec initialized in the repo
* tool integrations generated for `codex` and `opencode`
* first OpenSpec change created: `bootstrap-local-ai-stack`
Repository intent:
* `openspec/` becomes the spec/change-management home for implementation work
* this Foam note remains the KB-facing planning and operator-context note
* implementation artifacts should now be tracked in the repo, with this note linking back to them as needed
Current OpenSpec structure in the repo:
```text
/home/ben/code/ai-stack/
README.md
openspec/
changes/
bootstrap-local-ai-stack/
proposal.md
design.md
tasks.md
specs/
local-ai-stack/
spec.md
specs/
.codex/
.opencode/
```
Operational implication:
* future concrete stack work should start as OpenSpec changes in `ai-stack`
* Foam should reference the repo and summarize decisions, rather than becoming the only place where implementation structure lives
* the current baseline implementation change is `openspec/changes/bootstrap-local-ai-stack/`
## Stack definition v1
This is the concrete definition for the separate stack repo/worktree that now exists outside this KB at `/home/ben/code/ai-stack`.
### Scope of v1
Include in v1:
* one Docker Compose stack definition repo called `ai-stack`
* one always-on inference service: `sglang`
* one follow-on UI service: `open-webui`
* one shared host data root: `${AI_DATA_ROOT}`
* localhost-only published ports by default
* enough docs and templates to recreate the stack on another host
Do **not** include in v1:
* `ollama`
* `unsloth`
* training jobs
* public LAN exposure by default
* extra observability/services unless a real need appears
### Service topology
```text
local tools / scripts
|
v
127.0.0.1:30000
sglang API
|
+---- model weights / HF cache / logs under ${AI_DATA_ROOT}
127.0.0.1:3001
Open WebUI
|
v
talks to sglang over local Docker networking or local host endpoint
```
Operational intent:
* `sglang` is the canonical local inference backend
* `open-webui` is a convenience client on top of that backend, not the center of the stack
* if `open-webui` breaks, the API stack should still be usable directly
### Repo/worktree layout
```text
ai-stack/
compose.yaml
.env.example
.gitignore
README.md
docs/
architecture.md
recovery-checklist.md
model-inventory.md
sglang/
launch.env.example
notes.md
open-webui/
notes.md
scripts/
preflight-checklist.md
```
File intent:
* `compose.yaml` = the canonical service definition
* `.env.example` = documented host-specific variables with placeholder values only
* `.gitignore` = protect `.env`, logs, caches, exports, and downloaded assets from accidental commits
* `README.md` = operator-first quickstart for bring-up, shutdown, update, and rebuild
* `docs/architecture.md` = why the stack is shaped this way
* `docs/recovery-checklist.md` = fresh-host rebuild order
* `docs/model-inventory.md` = canonical list of intended models/adapters and where they are expected to come from
* `sglang/launch.env.example` = model/server launch parameters that may vary by host or model choice
* `sglang/notes.md` = operational notes for the inference service
* `open-webui/notes.md` = operational notes for the UI layer
* `scripts/preflight-checklist.md` = manual prerequisites/validation list before first start
### Compose contract
Initial Compose services:
* `sglang`
* required in v1
* NVIDIA-enabled container
* publishes `127.0.0.1:30000:30000`
* mounts model/cache/state/log paths from `${AI_DATA_ROOT}`
* uses `Qwen/Qwen3-8B-AWQ` as the first default model
* should be startable independently of any UI
* `open-webui`
* optional during initial API bring-up, but part of the v1 repo definition
* publishes `127.0.0.1:3001:8080`
* persists app state under `${AI_DATA_ROOT}/open-webui`
* points at the existing `sglang` OpenAI-compatible endpoint
Recommended Compose profiles:
* default / baseline: `sglang`
* optional UI profile: `open-webui`
Meaning:
* the inference API can come up first by itself
* the UI can be layered on later without redefining the stack
### Host path contract
`AI_DATA_ROOT` remains the root for all durable local state.
```text
${AI_DATA_ROOT}/
models/ # model files if intentionally stored outside tool-managed caches
huggingface/ # HF_HOME / HF hub cache
sglang/ # sglang-specific state, configs, temp runtime data
open-webui/ # persistent UI state
logs/ # host-visible logs
exports/ # inventories, backup manifests, saved config exports
scratch/ # temporary manual work area
```
Path rules:
* containers should only write durable state inside `${AI_DATA_ROOT}`
* container-internal default paths should be overridden where practical
* logs worth keeping should land on the host, not only inside container stdout history
### Env contract
The real `.env` should stay outside git. `.env.example` should document the contract.
Expected variables for v1:
```dotenv
AI_DATA_ROOT=/home/ben/llm-data
SGLANG_HOST_PORT=30000
OPEN_WEBUI_HOST_PORT=3001
SGLANG_MODEL=Qwen/Qwen3-8B-AWQ
SGLANG_CONTEXT_LENGTH=32768
SGLANG_MEM_FRACTION_STATIC=0.88
SGLANG_DTYPE=auto
HF_HOME=/home/ben/llm-data/huggingface
OPEN_WEBUI_DATA_DIR=/home/ben/llm-data/open-webui
```
Notes:
* secrets should be added only if/when a service actually needs them
* avoid putting host-specific absolute paths directly into `compose.yaml` when a variable will do
* if another host replaces `lifebalance`, the main required change should be env values, not Compose structure
### Canonical endpoints and names
Use these as the first-pass defaults unless execution reveals a conflict:
* stack repo/worktree name: `ai-stack`
* primary service name: `sglang`
* UI service name: `open-webui`
* canonical local inference endpoint: `http://127.0.0.1:30000/v1`
* canonical local UI URL: `http://127.0.0.1:3001`
* primary model identifier: `Qwen/Qwen3-8B-AWQ`
### Operating rules for v1
* prefer explicit image tags over floating `latest` where possible
* prefer localhost binding over broad exposure
* keep `sglang` usable without `open-webui`
* treat model downloads as reproducible inputs, not irreplaceable state
* treat UI state, manifests, and future adapters/checkpoints as state worth intentional preservation
* do not add routing until there is a real second-model need
### Future extension points
Planned later, but not part of the initial stack definition:
* `ollama` as a separate convenience lane if model switching becomes more valuable than keeping one backend
* a routing/gateway layer only after there are multiple active model backends to justify it
* `unsloth` as a separate workflow/repo path for training or fine-tuning
### What should be reproducible vs backed up
Rebuildable from config:
* container definitions
* image tags
* port mappings
* bind mounts
* service-level configuration
* startup commands
* model identifiers to download
Back up or preserve intentionally:
* secrets outside git
* exported model inventory/manifests
* any local UI/app data worth keeping
* any future training outputs, datasets, adapters, or checkpoints
Usually re-download instead of backing up:
* public base model weights
* Hugging Face cache contents
* container images
Optional exception:
* if download time becomes painful, a local backup of model files may still be worth doing, but that should be a conscious backup policy rather than an accident of where files ended up
### What belongs in git vs not in git
Commit:
* Compose files
* config templates
* `.env.example`
* setup scripts that do not contain secrets
* documentation, recovery notes, and model manifests
Never commit:
* `.env` with real values
* API tokens, auth secrets, passwords
* downloaded model weights
* Hugging Face cache directories
* logs
* databases/app state unless deliberately exporting sanitized fixtures
## Implementation checklist (draft)
This is a future execution checklist only. It does **not** mean these steps have been performed yet.
### 1. Create the reproducible stack-definition repo/worktree
- [ ] create a dedicated repo/worktree for the AI stack definition
- [ ] add `compose.yaml`
- [ ] add `.env.example`
- [ ] add a `README.md` with recovery/rebuild instructions
- [ ] add config templates for `sglang` and later `open-webui`
- [ ] add `.gitignore` rules for secrets, logs, caches, and downloaded assets
### 2. Define host-specific variables and conventions
- [ ] define `AI_DATA_ROOT`
- [ ] define canonical service names, ports, and container names
- [ ] define the canonical local API endpoint for `sglang`
- [ ] define where logs should live on the host
- [ ] define the host-specific values that belong in `.env` rather than in committed files
### 3. Create the persistent storage layout
- [ ] create `${AI_DATA_ROOT}`
- [ ] create `${AI_DATA_ROOT}/models`
- [ ] create `${AI_DATA_ROOT}/huggingface`
- [ ] create `${AI_DATA_ROOT}/sglang`
- [ ] create `${AI_DATA_ROOT}/open-webui`
- [ ] create `${AI_DATA_ROOT}/logs`
- [ ] create `${AI_DATA_ROOT}/exports`
- [ ] create `${AI_DATA_ROOT}/scratch`
- [ ] document ownership/permissions expectations for these paths
### 4. Prepare secrets and env handling
- [ ] decide how local secrets will be stored outside git
- [ ] create `.env` from `.env.example`
- [ ] ensure no passwords or tokens are passed through shell history
- [ ] document which values are required for a fresh-host rebuild
### 5. Stand up the SGLang API baseline
- [ ] define the `sglang` image/tag strategy
- [ ] define bind mounts for model/cache/log locations
- [ ] define the startup command for `Qwen3 8B AWQ`
- [ ] keep networking local-only by default
- [ ] document how the service is started, stopped, restarted, and updated
- [ ] document how the model is fetched or made available reproducibly
### 6. Validate the API baseline
- [ ] confirm the service starts cleanly
- [ ] confirm the model loads successfully on the 4090
- [ ] confirm the local OpenAI-compatible endpoint works
- [ ] confirm logs are written to the expected host path
- [ ] document basic smoke tests for future rebuild validation
### 7. Add Open WebUI after the API baseline is stable
- [ ] add `open-webui` to the stack-definition repo/worktree
- [ ] bind its persistent data to `${AI_DATA_ROOT}/open-webui`
- [ ] connect it to the existing `sglang` API
- [ ] document the intended UI URL and local-only access model
- [ ] document what configuration is UI state vs committed config
### 8. Capture disaster-recovery details
- [ ] document host prerequisites required before stack deployment
- [ ] document the exact rebuild order on a fresh machine
- [ ] document what must be restored from backup vs re-downloaded
- [ ] export a model/config inventory into `${AI_DATA_ROOT}/exports`
- [ ] add a short recovery checklist for "lifebalance died, rebuild elsewhere"
### 9. Leave room for later phases
- [ ] define how a future `ollama` lane would coexist without replacing `sglang` as the primary API
- [ ] define how a future `unsloth` workflow would keep training artifacts separate from inference services
- [ ] document what additional storage/backup policy would be needed for adapters, datasets, and checkpoints
[AI]: AI.md "AI"
[homelab]: homelab.md "homelab"