I Built My Own AI Stack: Two Tiers, One GPU Lease, Zero Cloud
Image: generated locally, on the stack described below.
Most "local AI" setups are one model behind one endpoint. Mine became something closer to a small platform: a single GPU on a workstation, a two-node cluster, and an arbitration layer that decides which workload gets the card. This is a write-up of the architecture, the numbers I measured, and the mistakes that cost me the most time.
Nothing in it leaves the local network. No API keys, no per-token billing, no vendor deciding which model I am allowed to run this quarter.
The constraints that shaped the design
Three realities drove every decision.
24 GB of GPU memory is not 24 GB of headroom. A 27B model with image understanding and an extra prediction head occupies roughly 13 GB on its own. Add music generation, a planner model, the image pipeline or an audio-and-image model and the card is full. Every heavy capability has to run alone, and the machine has to work that out by itself.
Nobody wants to manage modes. "Switch the GPU to music mode first" is a support ticket waiting to happen. The system had to infer intent from the request, not from a setting the caller remembers to change.
No duplicated weights. Model files are large, and copying them to every machine is waste. One shared model directory over a dedicated link was the only sane option.
Tier 1: the interactive workstation stack
One consumer workstation — a 24 GB graphics card, a desktop processor, 62 GB of memory — with models stored on a separate fast disk so the system drive never fills up.
| Capability | Model | Measured |
|---|---|---|
| Text and image understanding | Qwen3.8-27B | 112 tokens per second generated, 238 processed, 32,000-token context, ~13 GB of GPU memory |
| Speech synthesis | LFM3-Audio-2B | 24 kHz audio, live streaming, answers from idle in under a second |
| Music | ACE-Step 1.5 | a 20-second song in about 40 seconds, lyrics correct |
| Sound effects | Woosh | 5 seconds of 48 kHz audio in about 1 second |
| Audio and image model | MiniCPM-o 4.5 | 8.3 GB, fits on one card |
| Embeddings and reranking | granite, jina | always loaded, small footprint |
| Text recognition | GLM-OCR | read a test card exactly |
| Face recognition | ArcFace | same person 0.98, closest different person 0.15 |
| Voice recognition | CAM++ | same voice 0.84, closest different voice 0.29 |
| Speaker separation | pyannote | two voices separated at a 3.4-second boundary |
| Image, video, 3D and rigging | ComfyUI, TRELLIS.2, UniRig | image to textured 3D model with 882k faces; 34-bone skeleton |
All of it sits behind one OpenAI-compatible API on a single address — chat, embeddings, reranking, speech, transcription, text recognition, identity, speaker separation, music, sound effects. One endpoint, one token, nothing bespoke for each service.
That front door is the easy part. The hard part is what sits behind it.
The interesting part: a GPU lease, not a GPU
One front door, one arbiter, two tiers.
The API is fronted by an arbiter. It is not a load balancer; it is a broker that owns the graphics card and hands it out in modes: interactive work, deep reasoning, image, video, 3D, rigging, music, sound effects, audio-and-image, and gaming.
The rule that makes it usable: a request that starts real work takes the card. A music request stops everything outside that mode, waits until the memory is genuinely free, starts the music engine, forwards the request, and releases the card after 15 minutes idle. Cheap reads — status checks, progress polls — never take it, so a monitoring script cannot evict a job that is halfway through.
Design details worth stealing:
- Switching modes waits for roughly 45 seconds of quiet first, so a late poll cannot kill a job that is still running.
- A safety timer stops any heavy service that has been sitting at 0% GPU for an hour.
- A game watcher hands the card to gaming when a game starts and releases it on exit. It deliberately ignores game servers that run around the clock, which would otherwise hold the card hostage forever.
- The arbiter also kills leftover worker processes from the image pipeline. The image engine's own cleanup call does not release them; they keep holding memory and come back on the next run.
This is the difference between "I have models installed" and "the machine is usable by other people." The arbitration layer is most of the engineering effort, and it is the part almost nobody writes about.
Tier 2: the two-node cluster
For work that genuinely needs a very large model, there are two machines with 128 GB of unified memory each, joined by a dedicated 200-gigabit link that stays off the home network. Models are served from one shared directory, so the files exist once.
| Property | Value |
|---|---|
| Serving | the model is split across both machines, behind an OpenAI-compatible API |
| Model | DeepSeek V4.1 Flash |
| Context | one million tokens |
| Link | dedicated 200-gigabit point-to-point connection |
| Storage | one shared model directory, used by both machines |
| Monitoring | engine metrics collected into dashboards |
Measured on a short prompt: 55 tokens in 1.13 seconds, about 49 tokens per second including the 38 tokens it spent reasoning — one request at a time, nothing warmed up beforehand. Earlier tuning on the same hardware reached about 80 tokens per second on short prompts, and 130–170 tokens per second in total across six simultaneous requests.
The cluster also exposed the least glamorous part of self-hosting: what happens when one request carries an enormous amount of context. A request with 300,000 tokens has to be read in full before a single word comes back — at roughly 1,100 tokens per second, that is several minutes during which everything else slows down. No amount of tuning removes it. It is physics plus memory that has to be filled from scratch, and the honest answer is to plan around it rather than pretend it is not happening.
What went wrong
Every one of these cost me real time. One line each, because that is all a reader needs.
Tool calls break down at long context. Past roughly 90,000 tokens the model sometimes loses the marker that opens a tool call, and the reply leaks the raw markup as text. It is an open upstream bug, unfixed in the build I run. The workaround is to keep requests shorter and avoid maximum reasoning effort when tools are enabled.
Some model variants crash the released engine. I run a patched build instead, and I pin the version so an upgrade cannot quietly break it.
Speed-up options can cancel each other out. Four settings that each looked like a free win silently removed the acceleration entirely when combined. All four stay off now.
Shared storage is where the quiet failures live. A local folder sitting where the shared model directory should be mounted hides the shared copy, and a share limited to a single address breaks the moment a second network interface changes where traffic comes from. Allow the whole interconnect, and check the mount before debugging anything else.
Licensing is not a footnote. One sound-effects model ships non-commercial weights — fine for personal use, a consideration the moment money is involved. One 3D model's licence excludes some regions. Track this per model, not per project, or you will find out at the worst possible moment.
Monitoring is not optional. On machines that share one memory pool between processor and GPU, the vendor tool simply reports memory as unavailable, and you are blind. Collect the engine's own metrics instead, and normalise the names so dashboards survive a change of backend.
Where it stands
The stack is in daily use, not a demo. It backs an agent that transcribes meetings with speaker names, reads documents, generates music and 3D assets, recognises people in photos, and sends long-context reasoning to the cluster.
Cost per token after hardware: zero. Cost in maintenance: real. Expect to own a patched inference engine, a storage layout, an arbitration policy and a monitoring stack — and expect each of them to break in a way that only your own logs will explain.
What is next
- Calibrating the recognition thresholds on a larger set instead of trusting sensible defaults. The current values sit well clear of the closest false match, but "well clear on six people" is not a certified operating point.
- Making the first read of very long requests less painful, rather than pretending it does not exist.
- Treating the arbitration layer as the actual product: more workloads on the same single card, with no human in the loop.
If you are building something similar, the order that worked for me was: arbitration first, endpoints second, models last. The models are the easy part. Deciding who gets the card, and when, is the whole job.