Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Ben
|
574fbfeba9 | ||
|
Ben
|
d304fd0e6e |
@@ -1,14 +1,18 @@
|
||||
# AI
|
||||
|
||||
## Learning
|
||||
## Learning
|
||||
|
||||
* [The Ultimate Guide to Local AI and AI Agents (The Future is Here) - YouTube](https://www.youtube.com/watch?v=mNcXue7X8H0)
|
||||
* Docker Compose based framework for running AI locally
|
||||
* https://github.com/coleam00/local-ai-packaged
|
||||
* https://www.reddit.com/r/LocalLLaMA/
|
||||
*
|
||||
|
||||
## LocalAI Packaged
|
||||
* https://www.reddit.com/r/LocalLLaMA/
|
||||
*
|
||||
|
||||
## Projects
|
||||
|
||||
* [[self-hosting AI on lifebalance]]
|
||||
|
||||
## LocalAI Packaged
|
||||
|
||||
### Services
|
||||
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"folders": [
|
||||
{
|
||||
"path": "../code/ai-stack"
|
||||
}
|
||||
],
|
||||
"settings": {
|
||||
"workbench.colorCustomizations": {
|
||||
"activityBar.activeBackground": "#81c3a0",
|
||||
"activityBar.background": "#81c3a0",
|
||||
"activityBar.foreground": "#15202b",
|
||||
"activityBar.inactiveForeground": "#15202b99",
|
||||
"activityBarBadge.background": "#8d68b7",
|
||||
"activityBarBadge.foreground": "#e7e7e7",
|
||||
"commandCenter.border": "#15202b99",
|
||||
"sash.hoverBorder": "#81c3a0",
|
||||
"statusBar.background": "#5eb386",
|
||||
"statusBar.foreground": "#15202b",
|
||||
"statusBarItem.hoverBackground": "#47976d",
|
||||
"statusBarItem.remoteBackground": "#efe9f5",
|
||||
"statusBarItem.remoteForeground": "#15202b",
|
||||
"titleBar.activeBackground": "#5eb386",
|
||||
"titleBar.activeForeground": "#15202b",
|
||||
"titleBar.inactiveBackground": "#5eb38699",
|
||||
"titleBar.inactiveForeground": "#15202b99",
|
||||
"activityBarTop.activeBackground": "#81c3a0",
|
||||
"activityBarTop.background": "#81c3a0",
|
||||
"activityBarTop.foreground": "#15202b",
|
||||
"activityBarTop.inactiveForeground": "#15202b99",
|
||||
"commandCenter.foreground": "#15202b",
|
||||
"statusBar.debuggingBackground": "#5eb386",
|
||||
"statusBar.debuggingForeground": "#15202b"
|
||||
},
|
||||
"peacock.color": "#5eb386"
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -22,5 +22,5 @@
|
||||
|
||||
### Tooth pendants
|
||||
|
||||
* [🦷 Real Anatomy Tooth Seth #1・ 3D File for 3D printing・Cults](https://cults3d.com/en/3d-model/art/real-anatomy-tooth-seth-1)
|
||||
* [🦷 Real Anatomy Tooth Seth #1・ 3D File for 3D printing・Cults]https://cults3d.com/en/3d-model/art/real-anatomy-tooth-seth-1
|
||||
* https://cults3d.com/en/orders/128683287
|
||||
|
||||
+81
@@ -0,0 +1,81 @@
|
||||
# chezmoi
|
||||
|
||||
[chezmoi](https://www.chezmoi.io/) is a dotfile manager that keeps configuration files synchronized across machines using a git-backed source directory.
|
||||
|
||||
## Setup
|
||||
|
||||
### Configuration
|
||||
|
||||
Config file: `~/.config/chezmoi/chezmoi.toml`
|
||||
|
||||
```toml
|
||||
[git]
|
||||
autoCommit = true
|
||||
autoPush = true
|
||||
|
||||
[merge]
|
||||
command = "bash"
|
||||
args = [
|
||||
"-c",
|
||||
"cp {{ .Target | quote }} {{ printf \"%s.base\" .Target | quote }} && code --new-window --wait --merge {{ .Destination | quote }} {{ .Target | quote }} {{ printf \"%s.base\" .Target | quote }} {{ .Source | quote }}",
|
||||
]
|
||||
```
|
||||
|
||||
- **autoCommit**: Automatically commits changes after `chezmoi apply`
|
||||
- **autoPush**: Automatically pushes to remote after commit
|
||||
- **merge**: Custom merge command using VS Code's merge editor
|
||||
|
||||
### Source Directory
|
||||
|
||||
`~/.local/share/chezmoi/` - Git repository containing managed dotfiles
|
||||
|
||||
## Managed Dotfiles
|
||||
|
||||
| File/Directory | Description |
|
||||
|----------------|-------------|
|
||||
| `.bash_aliases` | Shell aliases |
|
||||
| `.bashrc` | Bash configuration |
|
||||
| `.tmux.conf` | Tmux configuration |
|
||||
| `.config/opencode/` | OpenCode AI assistant config |
|
||||
| `.gemini/settings.json` | Gemini CLI settings |
|
||||
| `.ssh/authorized_keys` | SSH authorized keys (private) |
|
||||
|
||||
### OpenCode Config
|
||||
|
||||
The `.config/opencode/` directory includes:
|
||||
- `opencode.jsonc` - Main OpenCode configuration
|
||||
- `oh-my-opencode-slim.json` - Slim preset configuration
|
||||
- `executable_opencode-tmux-launch.sh` - Script to launch OpenCode in tmux
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
# Edit a dotfile in the source directory
|
||||
chezmoi edit ~/.bashrc
|
||||
|
||||
# Apply changes to the home directory
|
||||
chezmoi apply
|
||||
|
||||
# See what would change
|
||||
chezmoi diff
|
||||
|
||||
# Add a new file to chezmoi
|
||||
chezmoi add ~/.newconfig
|
||||
|
||||
# Edit the config file
|
||||
chezmoi edit-config
|
||||
|
||||
# Initialize chezmoi on a new machine
|
||||
chezmoi init https://github.com/username/dotfiles.git
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Edit dotfiles with `chezmoi edit` (opens source file)
|
||||
2. Apply changes with `chezmoi apply`
|
||||
3. Changes auto-commit and push to git remote
|
||||
|
||||
## Related
|
||||
|
||||
- [[homelab]]
|
||||
- [[dev environments]]
|
||||
Regular → Executable
+30
-1
@@ -6,6 +6,35 @@
|
||||
],
|
||||
"settings": {
|
||||
"foam.edit.linkReferenceDefinitions": "withExtensions",
|
||||
"gitdoc.enabled": false
|
||||
"gitdoc.enabled": false,
|
||||
"workbench.colorCustomizations": {
|
||||
"activityBar.activeBackground": "#ffb733",
|
||||
"activityBar.background": "#ffb733",
|
||||
"activityBar.foreground": "#15202b",
|
||||
"activityBar.inactiveForeground": "#15202b99",
|
||||
"activityBarBadge.background": "#008053",
|
||||
"activityBarBadge.foreground": "#e7e7e7",
|
||||
"commandCenter.border": "#15202b99",
|
||||
"sash.hoverBorder": "#ffb733",
|
||||
"statusBar.background": "#ffa500",
|
||||
"statusBar.foreground": "#15202b",
|
||||
"statusBarItem.hoverBackground": "#cc8400",
|
||||
"statusBarItem.remoteBackground": "#008053",
|
||||
"statusBarItem.remoteForeground": "#15202b",
|
||||
"titleBar.activeBackground": "#ffa500",
|
||||
"titleBar.activeForeground": "#15202b",
|
||||
"titleBar.inactiveBackground": "#ffa50099",
|
||||
"titleBar.inactiveForeground": "#15202b99",
|
||||
"activityBarTop.activeBackground": "#ffb733",
|
||||
"activityBarTop.background": "#ffb733",
|
||||
"activityBarTop.foreground": "#15202b",
|
||||
"activityBarTop.inactiveForeground": "#15202b99",
|
||||
"commandCenter.foreground": "#15202b",
|
||||
"statusBar.debuggingBackground": "#ffa500",
|
||||
"statusBar.debuggingForeground": "#15202b",
|
||||
"activityBar.activeBorder": "#15202b",
|
||||
"activityBarTop.activeBorder": "#15202b"
|
||||
},
|
||||
"peacock.color": "orange"
|
||||
}
|
||||
}
|
||||
|
||||
+17
-9
@@ -55,15 +55,23 @@
|
||||
|
||||
### Home auth
|
||||
|
||||
## Hardware
|
||||
|
||||
* unnamed switch (TODO)
|
||||
* [Mikrotik CSS318-16G-2S+IN](https://mikrotik.com/product/css318_16g_2s_in#fndtn-downloads)
|
||||
* SwOS management interface: http://192.168.86.54
|
||||
* [manual](https://help.mikrotik.com/docs/spaces/SWOS/pages/76415036/CRS3xx+and+CSS326-24G-2S+series+Manual#CRS3xxandCSS32624G2S+seriesManual-LAG)
|
||||
* proxmox1
|
||||
* Beelink Mini PC EQi12, Intel Core 1220P(Max 4.4GHz 10C/12T), 16GB DDR4 500GB PCle4.0 SSD Mini Computers,Dual LAN/Wifi6/BT5.2,Dual 4K Display,Built-in Power Supply Office PC
|
||||
* proxmox2
|
||||
## Hardware
|
||||
|
||||
* unnamed switch (TODO)
|
||||
* [Mikrotik CSS318-16G-2S+IN](https://mikrotik.com/product/css318_16g_2s_in#fndtn-downloads)
|
||||
* SwOS management interface: http://192.168.86.54
|
||||
* [manual](https://help.mikrotik.com/docs/spaces/SWOS/pages/76415036/CRS3xx+and+CSS326-24G-2S+series+Manual#CRS3xxandCSS32624G2S+seriesManual-LAG)
|
||||
* lifebalance
|
||||
* Lenovo Legion T7 34IRZ8
|
||||
* Intel Core i9-14900K
|
||||
* 62 GiB RAM
|
||||
* NVIDIA GeForce RTX 4090 (24 GiB VRAM)
|
||||
* Debian GNU/Linux forky/sid
|
||||
* [[self-hosting AI on lifebalance]]
|
||||
* [[rclone google drive]]
|
||||
* proxmox1
|
||||
* Beelink Mini PC EQi12, Intel Core 1220P(Max 4.4GHz 10C/12T), 16GB DDR4 500GB PCle4.0 SSD Mini Computers,Dual LAN/Wifi6/BT5.2,Dual 4K Display,Built-in Power Supply Office PC
|
||||
* proxmox2
|
||||
* Beelink Mini PC EQi12, Intel Core 1220P(Max 4.4GHz 10C/12T), 16GB DDR4 500GB PCle4.0 SSD Mini Computers,Dual LAN/Wifi6/BT5.2,Dual 4K Display,Built-in Power Supply Office PC
|
||||
* proxmox3
|
||||
* Beelink Mini PC EQi12, Intel Core 1220P(Max 4.4GHz 10C/12T), 16GB DDR4 500GB PCle4.0 SSD Mini Computers,Dual LAN/Wifi6/BT5.2,Dual 4K Display,Built-in Power Supply Office PC
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# making
|
||||
|
||||
# making
|
||||
|
||||
## Collections
|
||||
|
||||
* https://makerworld.com/en/@beepmill/collections
|
||||
@@ -26,9 +26,11 @@
|
||||
|
||||
## Things to print
|
||||
|
||||
* https://thangs.com/designer/ForgeCore
|
||||
* Desk plants
|
||||
* https://www.printables.com/model/613681-universal-key-cover/files
|
||||
* https://thangs.com/designer/ForgeCore
|
||||
* Desk plants
|
||||
* Resin teeth #booloumud
|
||||
* https://www.printables.com/model/543553-gridfinity-storage-box-by-pred-now-parametric
|
||||
* [Modern Gridifinity Case](https://www.printables.com/model/894202-modern-gridfinity-case)
|
||||
* [Space Efficient Socket Holders - Gridfinity](https://www.printables.com/model/1101988-space-efficient-socket-holders-gridfinity/files)
|
||||
* Printing metric [[2025-06-15]]
|
||||
@@ -71,11 +73,10 @@
|
||||
|
||||
* Layer thickness
|
||||
|
||||
[//begin]: # "Autogenerated link references for markdown compatibility"
|
||||
[2025-06-15]: 2025-06-15.md "2025-06-15"
|
||||
[2025-06-17]: 2025-06-17.md "2025-06-17"
|
||||
[2025-07-01]: 2025-07-01.md "2025-07-01"
|
||||
[2025-06-30]: 2025-06-30.md "2025-06-30"
|
||||
[2025-06-19]: 2025-06-19.md "2025-06-19"
|
||||
[2025-06-18]: 2025-06-18.md "2025-06-18"
|
||||
[//end]: # "Autogenerated link references"
|
||||
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# rclone google drive
|
||||
|
||||
Related: [[homelab]]
|
||||
|
||||
## Current setup
|
||||
|
||||
This machine (`lifebalance`) is configured to access Google Drive through `rclone`.
|
||||
|
||||
* Remote name: `gdrive`
|
||||
* Config file: `~/.config/rclone/rclone.conf`
|
||||
* Mount point: `~/gdrive`
|
||||
* Auth flow used: local browser OAuth on the machine during `rclone config create`
|
||||
|
||||
## Installation
|
||||
|
||||
Installed from Debian packages:
|
||||
|
||||
```bash
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y rclone
|
||||
```
|
||||
|
||||
## Remote configuration
|
||||
|
||||
The Google Drive remote was created with full-drive scope and verified successfully.
|
||||
|
||||
Useful commands:
|
||||
|
||||
```bash
|
||||
rclone listremotes
|
||||
rclone lsd gdrive:
|
||||
rclone ls gdrive:docs
|
||||
```
|
||||
|
||||
## Persistent mount
|
||||
|
||||
The Drive mount is managed by a user systemd service.
|
||||
|
||||
* Service file: `~/.config/systemd/user/rclone-gdrive.service`
|
||||
* Mount command:
|
||||
|
||||
```bash
|
||||
/usr/bin/rclone mount gdrive: /home/ben/gdrive --config=/home/ben/.config/rclone/rclone.conf --vfs-cache-mode=writes
|
||||
```
|
||||
|
||||
Service contents:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Rclone mount for Google Drive
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/rclone mount gdrive: /home/ben/gdrive --config=/home/ben/.config/rclone/rclone.conf --vfs-cache-mode=writes
|
||||
ExecStop=/usr/bin/fusermount3 -uz /home/ben/gdrive
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
```
|
||||
|
||||
## Service management
|
||||
|
||||
```bash
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now rclone-gdrive.service
|
||||
systemctl --user status rclone-gdrive.service
|
||||
systemctl --user restart rclone-gdrive.service
|
||||
systemctl --user stop rclone-gdrive.service
|
||||
systemctl --user start rclone-gdrive.service
|
||||
```
|
||||
|
||||
Note: this mount starts automatically when the user session starts. If it should start before login after reboot, enable lingering for the user.
|
||||
|
||||
## Verification
|
||||
|
||||
These checks succeeded:
|
||||
|
||||
```bash
|
||||
mountpoint ~/gdrive
|
||||
ls ~/gdrive
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If the mount stops working:
|
||||
|
||||
```bash
|
||||
systemctl --user status rclone-gdrive.service
|
||||
journalctl --user -u rclone-gdrive.service -n 100 --no-pager
|
||||
fusermount3 -uz ~/gdrive
|
||||
systemctl --user restart rclone-gdrive.service
|
||||
```
|
||||
@@ -0,0 +1,616 @@
|
||||
|
||||
# self-hosting AI on lifebalance
|
||||
|
||||
Related: [[AI]], [[homelab]]
|
||||
|
||||
## Current host baseline
|
||||
|
||||
* Hostname: `lifebalance`
|
||||
* Hardware: Lenovo Legion T7 34IRZ8, Intel Core i9-14900K, 62 GiB RAM, NVIDIA GeForce RTX 4090 (24 GiB VRAM), 1.9 TB NVMe (`/home` on a 1.6 TB logical volume)
|
||||
* OS: Debian GNU/Linux forky/sid, kernel `6.18.15+deb14-amd64`
|
||||
* NVIDIA stack: driver `550.163.01`, CUDA `12.4`
|
||||
* Container stack: Docker `29.3.1`, containerd `2.2.2`, NVIDIA Container Toolkit `1.19.0`
|
||||
|
||||
## Current AI status
|
||||
|
||||
* No AI model server is currently running on `lifebalance`.
|
||||
* `docker.service` and `containerd.service` are running.
|
||||
* No `ollama` binary was found.
|
||||
* No `conda` install was found.
|
||||
* Global `python3` is `3.13.12`.
|
||||
* A dedicated `~/sglang` virtualenv exists, but it currently only has `pip` and `uv` installed.
|
||||
|
||||
## Evidence of previous setup
|
||||
|
||||
### SGLang
|
||||
|
||||
* Virtualenv: `~/sglang`
|
||||
* Startup script: `~/sglang/bin/start-sglang.sh`
|
||||
* Intended launch command:
|
||||
|
||||
```shell
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-8B-AWQ \
|
||||
--port 30000 \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.88 \
|
||||
--context-length 32768 \
|
||||
--dtype auto \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
* Notes:
|
||||
* configured for localhost-only serving on port `30000`
|
||||
* sized to use most of the 4090 VRAM
|
||||
* comment suggests it was intended for "oh-my-opencode utility agents"
|
||||
* the script comments still reference `~/scripts/start-sglang.sh`, but the file currently lives in `~/sglang/bin/start-sglang.sh`
|
||||
* the venv is currently incomplete, so this script would not run without reinstalling `sglang`
|
||||
|
||||
### Unsloth (Docker)
|
||||
|
||||
* Shell history shows installation of the NVIDIA container toolkit followed by `docker run ... --gpus all unsloth/unsloth`.
|
||||
* Current Docker artifacts:
|
||||
* image present: `unsloth/unsloth:latest` (~40.8 GB)
|
||||
* stopped container: `brave_dewdney`
|
||||
* published ports: `8888`, `8000`, `2222`
|
||||
* bind mount: `/home/ben/work:/workspace/work`
|
||||
* Notes:
|
||||
* `/home/ben/work` is currently empty
|
||||
* only `/workspace/work` was bind-mounted, so data under `/workspace/.cache/huggingface` would have lived inside the container layer instead of a persistent host path
|
||||
* shell history/container environment included a hard-coded Jupyter password; do not reuse that pattern without rotating the credential and moving it into a safer secret/env file flow
|
||||
|
||||
### LocalAI Packaged / Ollama notes
|
||||
|
||||
* [[AI]] already contains notes for a `local-ai-packaged` stack with:
|
||||
* Open WebUI
|
||||
* n8n
|
||||
* Langfuse
|
||||
* Ollama
|
||||
* Qdrant
|
||||
* `python start_services.py --profile gpu-nvidia`
|
||||
* I did not find:
|
||||
* a local checkout of that repo on `lifebalance`
|
||||
* a `start_services.py` file on this PC
|
||||
* an installed `ollama` binary
|
||||
* matching running containers or service units
|
||||
* This likely refers to another machine, a deleted checkout, or a plan that was documented before the actual install happened.
|
||||
|
||||
### Related but separate artifacts
|
||||
|
||||
* `homefoam/local-ai-beepmill.code-workspace` points at a missing `../local-ai-beepmill` folder.
|
||||
* `code/homelab_tofu/vm_beepboop.tf` defines a `beepboop` VM on `proxmox1` with GPU passthrough, 12 GiB RAM, and a 150 GiB disk.
|
||||
* `beepboop` may have been intended as a separate AI host, but there is not enough evidence on `lifebalance` to say it was actually used for this stack.
|
||||
|
||||
## Gaps to resolve
|
||||
|
||||
These are the main unanswered execution details, not fundamental direction questions:
|
||||
|
||||
* pick the exact `sglang` container image/tag strategy for reproducible deployment
|
||||
* decide whether model files should be stored explicitly under `${AI_DATA_ROOT}/models` or primarily re-hydrated through Hugging Face cache paths
|
||||
* define the exact Compose profile/default-start behavior for `open-webui`
|
||||
* decide the local secret storage pattern outside git (`.env`, separate secret file, password manager, or similar)
|
||||
* define the minimum backup policy for `open-webui` state, exported manifests, and any future adapters/checkpoints
|
||||
* document the exact fresh-host prerequisite list before the first real deployment
|
||||
|
||||
## Working plan v1
|
||||
|
||||
### Planning assumptions
|
||||
|
||||
These are the current decisions for planning purposes:
|
||||
|
||||
* first success = a local-only inference API on `lifebalance`
|
||||
* phase 1 should be Docker-first
|
||||
* phase 1 should target `Qwen3 8B AWQ` first
|
||||
* `Open WebUI` should come after the `SGLang` API is working
|
||||
* `lifebalance` is now the canonical LLM host; `beepboop` is no longer the intended host for this project
|
||||
* use a custom large-data storage root from day 1 instead of tool defaults
|
||||
* plan for host recreation from the beginning
|
||||
* keep reproducible stack-definition files in a separate dedicated repo/worktree, not inside this KB repo
|
||||
* phase 1 should include:
|
||||
* OpenAI-style API access
|
||||
* model switching
|
||||
* a web UI
|
||||
* quantized models
|
||||
* training/fine-tuning should come later, not in the initial stack
|
||||
|
||||
### Recommended architecture
|
||||
|
||||
Use a phased hybrid approach:
|
||||
|
||||
* **Phase 1 baseline:** `SGLang + Open WebUI`
|
||||
* **Phase 2 convenience track:** add `Ollama` only if easier model switching and broader casual experimentation become more important
|
||||
* **Phase 3 training track:** keep `Unsloth` separate as an occasional training/fine-tuning workflow, not part of the always-on serving stack
|
||||
|
||||
### Why this is the current recommendation
|
||||
|
||||
* The primary goal is a strong local inference API, not just a chat UI.
|
||||
* `SGLang` fits that goal better than starting with an Ollama-first stack.
|
||||
* There is already evidence of prior `SGLang` experimentation on this machine, so this path builds on existing intent.
|
||||
* `Open WebUI` can still provide the human-facing UI without forcing Ollama to be the primary backend.
|
||||
* This keeps the first phase local-only while optimizing earlier for agent/tool use.
|
||||
|
||||
### Why not start with Ollama as the primary backend
|
||||
|
||||
* It is more convenient for casual local experimentation than for an API-first serving plan.
|
||||
* It may still be useful later, but it is no longer the best fit for the primary phase-1 objective.
|
||||
* We can add it later without redesigning the whole stack.
|
||||
|
||||
### Why not start with SGLang as the only component
|
||||
|
||||
* A web UI still matters in phase 1.
|
||||
* `Open WebUI` remains useful as the human-facing interface even if `SGLang` is the main backend.
|
||||
|
||||
### Why not start with Unsloth
|
||||
|
||||
* The first outcome should be reliable inference, not training.
|
||||
* Training adds larger storage, cache, dataset, and secret-management concerns.
|
||||
* Keeping training separate reduces operational complexity.
|
||||
|
||||
## Phased roadmap
|
||||
|
||||
### Phase 1 — stable local workstation stack
|
||||
|
||||
Goal: a local-only, Docker-managed baseline for everyday use.
|
||||
|
||||
Planned components:
|
||||
|
||||
* `sglang` for local model serving
|
||||
* `open-webui` as the follow-on chat/testing interface after the API baseline is working
|
||||
* persistent host directories for:
|
||||
* models
|
||||
* app data
|
||||
* logs
|
||||
* exported configuration
|
||||
|
||||
Success criteria:
|
||||
|
||||
* at least one primary quantized model is served reliably
|
||||
* the UI works locally on `lifebalance`
|
||||
* the API is usable by local scripts/tools and is documented as the canonical local endpoint
|
||||
* all services bind locally first, or are otherwise restricted to local use
|
||||
* storage locations are explicit and documented before large downloads begin
|
||||
|
||||
### Phase 2 — convenience and model-flexibility lane
|
||||
|
||||
Goal: add easier model switching and a broader casual-use path only if phase-1 usage shows it is worth the extra moving parts.
|
||||
|
||||
Planned changes:
|
||||
|
||||
* evaluate whether adding `Ollama` improves day-to-day experimentation enough to justify dual backends
|
||||
* keep `SGLang` as the canonical agent/API backend unless there is a good reason to change that
|
||||
* decide whether:
|
||||
* `Open WebUI` should talk to SGLang only
|
||||
* `Open WebUI` should talk to Ollama only
|
||||
* both backends should coexist for different purposes
|
||||
|
||||
Success criteria:
|
||||
|
||||
* any second backend has a clear purpose
|
||||
* model/API behavior is predictable and documented
|
||||
* added convenience does not make the core serving path harder to operate
|
||||
|
||||
### Phase 3 — training and fine-tuning lane
|
||||
|
||||
Goal: support occasional LoRA/fine-tuning experiments without destabilizing the serving stack.
|
||||
|
||||
Planned components:
|
||||
|
||||
* a separate `Unsloth` workflow
|
||||
* dedicated storage/caching locations for datasets, checkpoints, and Hugging Face assets
|
||||
* clear separation between:
|
||||
* always-on inference services
|
||||
* temporary experiment/training workloads
|
||||
|
||||
Success criteria:
|
||||
|
||||
* training artifacts persist outside disposable containers
|
||||
* secrets are not passed via shell history
|
||||
* training jobs do not accidentally overwrite or disrupt inference setup
|
||||
|
||||
## Planned implementation decisions
|
||||
|
||||
When work begins, prefer:
|
||||
|
||||
* one project directory for Compose files, env files, and documentation
|
||||
* one persistent storage root for models/caches/data
|
||||
* localhost-first networking before any LAN exposure
|
||||
* explicit secrets handling instead of shell-history passwords
|
||||
* one canonical source of truth in this note for ports, paths, and services
|
||||
|
||||
### Recommended storage layout
|
||||
|
||||
Use a configurable storage root variable instead of hard-coding paths throughout the stack.
|
||||
|
||||
Recommended current value on `lifebalance`:
|
||||
|
||||
```shell
|
||||
AI_DATA_ROOT=/home/ben/llm-data
|
||||
```
|
||||
|
||||
Recommended layout under that root:
|
||||
|
||||
```text
|
||||
${AI_DATA_ROOT}/
|
||||
models/ # durable model files when not using opaque tool defaults
|
||||
huggingface/ # HF cache and downloads
|
||||
sglang/ # SGLang-specific state/config
|
||||
open-webui/ # Open WebUI persistent app data
|
||||
logs/ # service logs kept outside containers
|
||||
exports/ # saved manifests, model lists, config exports
|
||||
scratch/ # temporary working area
|
||||
```
|
||||
|
||||
Rationale:
|
||||
|
||||
* `lifebalance` has a large `/home` logical volume, so `/home/ben/llm-data` is a practical first location.
|
||||
* The path should still be treated as configurable so another host can recreate the same setup with a different mount point.
|
||||
* The important invariant is the directory structure and variable name, not the literal absolute path.
|
||||
|
||||
### Reproducibility-first approach
|
||||
|
||||
Treat the AI stack as something that should be rebuildable on a fresh host with minimal guesswork.
|
||||
|
||||
Preferred pattern:
|
||||
|
||||
* one small project directory for the stack definition
|
||||
* one configurable data root for persistent state
|
||||
* one env file for host-specific values
|
||||
* one setup note as the source of truth for how the stack is intended to work
|
||||
|
||||
### Project repo and workflow pivot
|
||||
|
||||
The implementation repo now exists at:
|
||||
|
||||
* `/home/ben/code/ai-stack`
|
||||
|
||||
Current repo state:
|
||||
|
||||
* git repo initialized
|
||||
* default branch set to `main`
|
||||
* OpenSpec initialized in the repo
|
||||
* tool integrations generated for `codex` and `opencode`
|
||||
* first OpenSpec change created: `bootstrap-local-ai-stack`
|
||||
|
||||
Repository intent:
|
||||
|
||||
* `openspec/` becomes the spec/change-management home for implementation work
|
||||
* this Foam note remains the KB-facing planning and operator-context note
|
||||
* implementation artifacts should now be tracked in the repo, with this note linking back to them as needed
|
||||
|
||||
Current OpenSpec structure in the repo:
|
||||
|
||||
```text
|
||||
/home/ben/code/ai-stack/
|
||||
README.md
|
||||
openspec/
|
||||
changes/
|
||||
bootstrap-local-ai-stack/
|
||||
proposal.md
|
||||
design.md
|
||||
tasks.md
|
||||
specs/
|
||||
local-ai-stack/
|
||||
spec.md
|
||||
specs/
|
||||
.codex/
|
||||
.opencode/
|
||||
```
|
||||
|
||||
Operational implication:
|
||||
|
||||
* future concrete stack work should start as OpenSpec changes in `ai-stack`
|
||||
* Foam should reference the repo and summarize decisions, rather than becoming the only place where implementation structure lives
|
||||
* the current baseline implementation change is `openspec/changes/bootstrap-local-ai-stack/`
|
||||
|
||||
## Stack definition v1
|
||||
|
||||
This is the concrete definition for the separate stack repo/worktree that now exists outside this KB at `/home/ben/code/ai-stack`.
|
||||
|
||||
### Scope of v1
|
||||
|
||||
Include in v1:
|
||||
|
||||
* one Docker Compose stack definition repo called `ai-stack`
|
||||
* one always-on inference service: `sglang`
|
||||
* one follow-on UI service: `open-webui`
|
||||
* one shared host data root: `${AI_DATA_ROOT}`
|
||||
* localhost-only published ports by default
|
||||
* enough docs and templates to recreate the stack on another host
|
||||
|
||||
Do **not** include in v1:
|
||||
|
||||
* `ollama`
|
||||
* `unsloth`
|
||||
* training jobs
|
||||
* public LAN exposure by default
|
||||
* extra observability/services unless a real need appears
|
||||
|
||||
### Service topology
|
||||
|
||||
```text
|
||||
local tools / scripts
|
||||
|
|
||||
v
|
||||
127.0.0.1:30000
|
||||
sglang API
|
||||
|
|
||||
+---- model weights / HF cache / logs under ${AI_DATA_ROOT}
|
||||
|
||||
127.0.0.1:3001
|
||||
Open WebUI
|
||||
|
|
||||
v
|
||||
talks to sglang over local Docker networking or local host endpoint
|
||||
```
|
||||
|
||||
Operational intent:
|
||||
|
||||
* `sglang` is the canonical local inference backend
|
||||
* `open-webui` is a convenience client on top of that backend, not the center of the stack
|
||||
* if `open-webui` breaks, the API stack should still be usable directly
|
||||
|
||||
### Repo/worktree layout
|
||||
|
||||
```text
|
||||
ai-stack/
|
||||
compose.yaml
|
||||
.env.example
|
||||
.gitignore
|
||||
README.md
|
||||
docs/
|
||||
architecture.md
|
||||
recovery-checklist.md
|
||||
model-inventory.md
|
||||
sglang/
|
||||
launch.env.example
|
||||
notes.md
|
||||
open-webui/
|
||||
notes.md
|
||||
scripts/
|
||||
preflight-checklist.md
|
||||
```
|
||||
|
||||
File intent:
|
||||
|
||||
* `compose.yaml` = the canonical service definition
|
||||
* `.env.example` = documented host-specific variables with placeholder values only
|
||||
* `.gitignore` = protect `.env`, logs, caches, exports, and downloaded assets from accidental commits
|
||||
* `README.md` = operator-first quickstart for bring-up, shutdown, update, and rebuild
|
||||
* `docs/architecture.md` = why the stack is shaped this way
|
||||
* `docs/recovery-checklist.md` = fresh-host rebuild order
|
||||
* `docs/model-inventory.md` = canonical list of intended models/adapters and where they are expected to come from
|
||||
* `sglang/launch.env.example` = model/server launch parameters that may vary by host or model choice
|
||||
* `sglang/notes.md` = operational notes for the inference service
|
||||
* `open-webui/notes.md` = operational notes for the UI layer
|
||||
* `scripts/preflight-checklist.md` = manual prerequisites/validation list before first start
|
||||
|
||||
### Compose contract
|
||||
|
||||
Initial Compose services:
|
||||
|
||||
* `sglang`
|
||||
* required in v1
|
||||
* NVIDIA-enabled container
|
||||
* publishes `127.0.0.1:30000:30000`
|
||||
* mounts model/cache/state/log paths from `${AI_DATA_ROOT}`
|
||||
* uses `Qwen/Qwen3-8B-AWQ` as the first default model
|
||||
* should be startable independently of any UI
|
||||
* `open-webui`
|
||||
* optional during initial API bring-up, but part of the v1 repo definition
|
||||
* publishes `127.0.0.1:3001:8080`
|
||||
* persists app state under `${AI_DATA_ROOT}/open-webui`
|
||||
* points at the existing `sglang` OpenAI-compatible endpoint
|
||||
|
||||
Recommended Compose profiles:
|
||||
|
||||
* default / baseline: `sglang`
|
||||
* optional UI profile: `open-webui`
|
||||
|
||||
Meaning:
|
||||
|
||||
* the inference API can come up first by itself
|
||||
* the UI can be layered on later without redefining the stack
|
||||
|
||||
### Host path contract
|
||||
|
||||
`AI_DATA_ROOT` remains the root for all durable local state.
|
||||
|
||||
```text
|
||||
${AI_DATA_ROOT}/
|
||||
models/ # model files if intentionally stored outside tool-managed caches
|
||||
huggingface/ # HF_HOME / HF hub cache
|
||||
sglang/ # sglang-specific state, configs, temp runtime data
|
||||
open-webui/ # persistent UI state
|
||||
logs/ # host-visible logs
|
||||
exports/ # inventories, backup manifests, saved config exports
|
||||
scratch/ # temporary manual work area
|
||||
```
|
||||
|
||||
Path rules:
|
||||
|
||||
* containers should only write durable state inside `${AI_DATA_ROOT}`
|
||||
* container-internal default paths should be overridden where practical
|
||||
* logs worth keeping should land on the host, not only inside container stdout history
|
||||
|
||||
### Env contract
|
||||
|
||||
The real `.env` should stay outside git. `.env.example` should document the contract.
|
||||
|
||||
Expected variables for v1:
|
||||
|
||||
```dotenv
|
||||
AI_DATA_ROOT=/home/ben/llm-data
|
||||
SGLANG_HOST_PORT=30000
|
||||
OPEN_WEBUI_HOST_PORT=3001
|
||||
SGLANG_MODEL=Qwen/Qwen3-8B-AWQ
|
||||
SGLANG_CONTEXT_LENGTH=32768
|
||||
SGLANG_MEM_FRACTION_STATIC=0.88
|
||||
SGLANG_DTYPE=auto
|
||||
HF_HOME=/home/ben/llm-data/huggingface
|
||||
OPEN_WEBUI_DATA_DIR=/home/ben/llm-data/open-webui
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
* secrets should be added only if/when a service actually needs them
|
||||
* avoid putting host-specific absolute paths directly into `compose.yaml` when a variable will do
|
||||
* if another host replaces `lifebalance`, the main required change should be env values, not Compose structure
|
||||
|
||||
### Canonical endpoints and names
|
||||
|
||||
Use these as the first-pass defaults unless execution reveals a conflict:
|
||||
|
||||
* stack repo/worktree name: `ai-stack`
|
||||
* primary service name: `sglang`
|
||||
* UI service name: `open-webui`
|
||||
* canonical local inference endpoint: `http://127.0.0.1:30000/v1`
|
||||
* canonical local UI URL: `http://127.0.0.1:3001`
|
||||
* primary model identifier: `Qwen/Qwen3-8B-AWQ`
|
||||
|
||||
### Operating rules for v1
|
||||
|
||||
* prefer explicit image tags over floating `latest` where possible
|
||||
* prefer localhost binding over broad exposure
|
||||
* keep `sglang` usable without `open-webui`
|
||||
* treat model downloads as reproducible inputs, not irreplaceable state
|
||||
* treat UI state, manifests, and future adapters/checkpoints as state worth intentional preservation
|
||||
* do not add routing until there is a real second-model need
|
||||
|
||||
### Future extension points
|
||||
|
||||
Planned later, but not part of the initial stack definition:
|
||||
|
||||
* `ollama` as a separate convenience lane if model switching becomes more valuable than keeping one backend
|
||||
* a routing/gateway layer only after there are multiple active model backends to justify it
|
||||
* `unsloth` as a separate workflow/repo path for training or fine-tuning
|
||||
|
||||
### What should be reproducible vs backed up
|
||||
|
||||
Rebuildable from config:
|
||||
|
||||
* container definitions
|
||||
* image tags
|
||||
* port mappings
|
||||
* bind mounts
|
||||
* service-level configuration
|
||||
* startup commands
|
||||
* model identifiers to download
|
||||
|
||||
Back up or preserve intentionally:
|
||||
|
||||
* secrets outside git
|
||||
* exported model inventory/manifests
|
||||
* any local UI/app data worth keeping
|
||||
* any future training outputs, datasets, adapters, or checkpoints
|
||||
|
||||
Usually re-download instead of backing up:
|
||||
|
||||
* public base model weights
|
||||
* Hugging Face cache contents
|
||||
* container images
|
||||
|
||||
Optional exception:
|
||||
|
||||
* if download time becomes painful, a local backup of model files may still be worth doing, but that should be a conscious backup policy rather than an accident of where files ended up
|
||||
|
||||
### What belongs in git vs not in git
|
||||
|
||||
Commit:
|
||||
|
||||
* Compose files
|
||||
* config templates
|
||||
* `.env.example`
|
||||
* setup scripts that do not contain secrets
|
||||
* documentation, recovery notes, and model manifests
|
||||
|
||||
Never commit:
|
||||
|
||||
* `.env` with real values
|
||||
* API tokens, auth secrets, passwords
|
||||
* downloaded model weights
|
||||
* Hugging Face cache directories
|
||||
* logs
|
||||
* databases/app state unless deliberately exporting sanitized fixtures
|
||||
|
||||
## Implementation checklist (draft)
|
||||
|
||||
This is a future execution checklist only. It does **not** mean these steps have been performed yet.
|
||||
|
||||
### 1. Create the reproducible stack-definition repo/worktree
|
||||
|
||||
- [ ] create a dedicated repo/worktree for the AI stack definition
|
||||
- [ ] add `compose.yaml`
|
||||
- [ ] add `.env.example`
|
||||
- [ ] add a `README.md` with recovery/rebuild instructions
|
||||
- [ ] add config templates for `sglang` and later `open-webui`
|
||||
- [ ] add `.gitignore` rules for secrets, logs, caches, and downloaded assets
|
||||
|
||||
### 2. Define host-specific variables and conventions
|
||||
|
||||
- [ ] define `AI_DATA_ROOT`
|
||||
- [ ] define canonical service names, ports, and container names
|
||||
- [ ] define the canonical local API endpoint for `sglang`
|
||||
- [ ] define where logs should live on the host
|
||||
- [ ] define the host-specific values that belong in `.env` rather than in committed files
|
||||
|
||||
### 3. Create the persistent storage layout
|
||||
|
||||
- [ ] create `${AI_DATA_ROOT}`
|
||||
- [ ] create `${AI_DATA_ROOT}/models`
|
||||
- [ ] create `${AI_DATA_ROOT}/huggingface`
|
||||
- [ ] create `${AI_DATA_ROOT}/sglang`
|
||||
- [ ] create `${AI_DATA_ROOT}/open-webui`
|
||||
- [ ] create `${AI_DATA_ROOT}/logs`
|
||||
- [ ] create `${AI_DATA_ROOT}/exports`
|
||||
- [ ] create `${AI_DATA_ROOT}/scratch`
|
||||
- [ ] document ownership/permissions expectations for these paths
|
||||
|
||||
### 4. Prepare secrets and env handling
|
||||
|
||||
- [ ] decide how local secrets will be stored outside git
|
||||
- [ ] create `.env` from `.env.example`
|
||||
- [ ] ensure no passwords or tokens are passed through shell history
|
||||
- [ ] document which values are required for a fresh-host rebuild
|
||||
|
||||
### 5. Stand up the SGLang API baseline
|
||||
|
||||
- [ ] define the `sglang` image/tag strategy
|
||||
- [ ] define bind mounts for model/cache/log locations
|
||||
- [ ] define the startup command for `Qwen3 8B AWQ`
|
||||
- [ ] keep networking local-only by default
|
||||
- [ ] document how the service is started, stopped, restarted, and updated
|
||||
- [ ] document how the model is fetched or made available reproducibly
|
||||
|
||||
### 6. Validate the API baseline
|
||||
|
||||
- [ ] confirm the service starts cleanly
|
||||
- [ ] confirm the model loads successfully on the 4090
|
||||
- [ ] confirm the local OpenAI-compatible endpoint works
|
||||
- [ ] confirm logs are written to the expected host path
|
||||
- [ ] document basic smoke tests for future rebuild validation
|
||||
|
||||
### 7. Add Open WebUI after the API baseline is stable
|
||||
|
||||
- [ ] add `open-webui` to the stack-definition repo/worktree
|
||||
- [ ] bind its persistent data to `${AI_DATA_ROOT}/open-webui`
|
||||
- [ ] connect it to the existing `sglang` API
|
||||
- [ ] document the intended UI URL and local-only access model
|
||||
- [ ] document what configuration is UI state vs committed config
|
||||
|
||||
### 8. Capture disaster-recovery details
|
||||
|
||||
- [ ] document host prerequisites required before stack deployment
|
||||
- [ ] document the exact rebuild order on a fresh machine
|
||||
- [ ] document what must be restored from backup vs re-downloaded
|
||||
- [ ] export a model/config inventory into `${AI_DATA_ROOT}/exports`
|
||||
- [ ] add a short recovery checklist for "lifebalance died, rebuild elsewhere"
|
||||
|
||||
### 9. Leave room for later phases
|
||||
|
||||
- [ ] define how a future `ollama` lane would coexist without replacing `sglang` as the primary API
|
||||
- [ ] define how a future `unsloth` workflow would keep training artifacts separate from inference services
|
||||
- [ ] document what additional storage/backup policy would be needed for adapters, datasets, and checkpoints
|
||||
|
||||
|
||||
[AI]: AI.md "AI"
|
||||
[homelab]: homelab.md "homelab"
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"folders": [
|
||||
{
|
||||
"path": "../shopify_theme"
|
||||
"path": "../code/shopify_theme"
|
||||
}
|
||||
],
|
||||
"settings": {
|
||||
@@ -17,12 +17,21 @@
|
||||
"statusBar.background": "#96bf48",
|
||||
"statusBar.foreground": "#15202b",
|
||||
"statusBarItem.hoverBackground": "#7a9d37",
|
||||
"statusBarItem.remoteBackground": "#96bf48",
|
||||
"statusBarItem.remoteBackground": "#3a80a5",
|
||||
"statusBarItem.remoteForeground": "#15202b",
|
||||
"titleBar.activeBackground": "#96bf48",
|
||||
"titleBar.activeForeground": "#15202b",
|
||||
"titleBar.inactiveBackground": "#96bf4899",
|
||||
"titleBar.inactiveForeground": "#15202b99"
|
||||
"titleBar.inactiveForeground": "#15202b99",
|
||||
"activityBarTop.activeBackground": "#accc6e",
|
||||
"activityBarTop.background": "#accc6e",
|
||||
"activityBarTop.foreground": "#15202b",
|
||||
"activityBarTop.inactiveForeground": "#15202b99",
|
||||
"commandCenter.foreground": "#15202b",
|
||||
"statusBar.debuggingBackground": "#96bf48",
|
||||
"statusBar.debuggingForeground": "#15202b",
|
||||
"activityBar.activeBorder": "#15202b",
|
||||
"activityBarTop.activeBorder": "#15202b"
|
||||
},
|
||||
"peacock.color": "#96bf48"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# tmux
|
||||
|
||||
Terminal multiplexer configuration managed via [[chezmoi]].
|
||||
|
||||
## Configuration
|
||||
|
||||
`~/.tmux.conf` - sourced from chezmoi at `~/.local/share/chezmoi/dot_tmux.conf`
|
||||
|
||||
```conf
|
||||
# TPM - Tmux Plugin Manager
|
||||
set -g @plugin 'tmux-plugins/tpm'
|
||||
set -g @plugin 'tmux-plugins/tmux-sensible'
|
||||
|
||||
# Session persistence
|
||||
set -g @plugin 'tmux-plugins/tmux-resurrect'
|
||||
set -g @plugin 'tmux-plugins/tmux-continuum'
|
||||
|
||||
# Resurrect options
|
||||
set -g @resurrect-capture-pane-contents 'on'
|
||||
|
||||
# Continuum options
|
||||
set -g @continuum-restore 'on'
|
||||
set -g @continuum-save-interval '15'
|
||||
|
||||
# Initialize TPM (must be at the bottom)
|
||||
run '~/.tmux/plugins/tpm/tpm'
|
||||
|
||||
# Base settings
|
||||
set-option -g mouse
|
||||
set-window-option -g window-size latest
|
||||
set-window-option -g aggressive-resize on
|
||||
```
|
||||
|
||||
## Plugins
|
||||
|
||||
### TPM (Tmux Plugin Manager)
|
||||
Plugin manager for tmux. Install new plugins by adding them to `.tmux.conf` then pressing `prefix + I`.
|
||||
|
||||
### tmux-sensible
|
||||
Basic settings that most tmux users want. Included by default.
|
||||
|
||||
### tmux-resurrect
|
||||
Saves and restores tmux environment after restart.
|
||||
|
||||
- `prefix + Ctrl-s` - Save current environment
|
||||
- `prefix + Ctrl-r` - Restore last save
|
||||
|
||||
Saves: sessions, windows, panes, layouts, working directories, running programs
|
||||
|
||||
### tmux-continuum
|
||||
Automatic saving and restoring. Requires tmux-resurrect.
|
||||
|
||||
- Auto-saves every 15 minutes (configurable via `@continuum-save-interval`)
|
||||
- Auto-restores on tmux start when `@continuum-restore 'on'`
|
||||
|
||||
Saves stored in `~/.tmux/resurrect/` (~5KB per save)
|
||||
|
||||
## Base Settings
|
||||
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| `mouse` | Enable mouse support (click, scroll, resize) |
|
||||
| `window-size latest` | Use newest window size when creating new windows |
|
||||
| `aggressive-resize` | Aggressively resize windows to fit smallest client |
|
||||
|
||||
## Installation
|
||||
|
||||
After updating `.tmux.conf`:
|
||||
1. Reload config: `tmux source-file ~/.tmux.conf`
|
||||
2. Install plugins: press `prefix + I` inside tmux
|
||||
Reference in New Issue
Block a user