The GPU can move. The bridge should not. The first slm-setup post set up a split between two models. A premium agent — the cloud model behind Cursor, GitHub Copilot, or Claude Code — does the planning and the reviewing. A small model running on your own hardware under Ollama — an open-source server for running language models locally — does the mechanical work: tests, boilerplate, renames. The local model is never wired into the IDE’s model picker (the dropdown where you choose GPT or Claude as your chat model); it is exposed as a set of tools the premium agent calls. This post is the host roadmap: an AMD “Halo”-class machine is the next place that private Ollama can live. It is not a second coding product.
The pictures below are C4 diagrams from the repo. C4 is a standard way to draw a software system at increasing zoom: context (who talks to what), containers (the running pieces), components (inside one piece), and deployment (which hardware runs what). The point of walking them here: the pieces stay the same at every stage of the roadmap. Only the machine hosting the model changes.
Level 1 — the trust boundary does not move
Context. The private lab talks to a premium model for planning. It must not publish Ollama. Cloud-hosted agents are out of scope.
I still want Cursor, Copilot, or Claude in the chair. Those requests go to the vendor’s premium model. The workstation can also reach a private GPU. The vendor’s cloud agents cannot — they run in the vendor’s data center, not on your network. That is why the local model is exposed through MCP, the Model Context Protocol: an open standard that lets a desktop IDE agent launch a program on your machine and call it as a set of named tools. Because that program runs on the workstation, it can reach hardware the cloud never sees. Halo does not change the picture. A new box on a private network is still the same lab. A tunnel so a cloud agent can see Ollama is still forbidden.
Level 2 — the host is a slot
Containers. Developer to IDE to MCP to Ollama, then fast or strong. Same machine or two machines.
The repo ships the MCP server: a small local process named local-coding-slm. Ollama is operated, not rewritten. The IDE launches the server and talks to it over stdio — plain standard input and output, no network port on the workstation. The server reads the Ollama address from a gitignored .env file and calls Ollama’s HTTP chat API. Fast model by default. Strong model when the cheap answer is not good enough. The starter pair is qwen3.5:9b for everyday work and devstral-small-2 for harder local work.
This is not a sketch — the same-machine version of this diagram is what runs today, with unit tests in the repo passing against it. Halo fills the inference-host slot later: unified memory (one pool shared by CPU and GPU) and ROCm (AMD’s GPU compute stack) instead of a discrete NVIDIA card. The workstation side does not grow a second server. You change OLLAMA_BASE_URL, or better, keep Ollama listening only on the Halo box’s own localhost and forward the port to your workstation with SSH local forwarding (ssh -L) — nothing new listens on the network.
Level 3 — Halo does not touch the tools
Components. The premium agent sends a task and a few files. The server does not scan the repo, write files, or open a listen port.
The tools stay local_status, local_code, local_refactor, local_generate_tests, local_explain, and local_review — the six names the IDE agent can call, described in the first post. Each one has a short fixed system prompt. The Ollama client maps the fast and strong aliases to concrete model tags (a tag is Ollama’s name for a downloadable model version, like qwen3.5:9b), sets the context size, and times out. The deployment checker inspects the URL, the tags, gitignore, and whether Ollama’s port (11434) is listening on every interface instead of just localhost. It is a batch job. It is not on the request path.
A Halo-shaped spec that renames those tools, or invents SLM_* aliases, would break the Cursor, Copilot, and Claude configs that already work. The public contract stays local_* and OLLAMA_*.
Deployment — one active host
Variants. Only one host is active. Prefer SSH to that host’s localhost. A private-interface bind is optional. A public bind is not.
Today the solid line is same-machine Ollama: workstation and GPU in one box. That profile is running now — the MCP server, the tools, and the repo’s test suite all exercise it. The next step on the roadmap is the middle variant: two machines, with Ollama on a separate private GPU host, kept on that machine’s localhost and reached over SSH. That two-machine testing is what happens before Halo, because it proves the only part Halo actually changes — reaching the model over the wire instead of in the same box. Halo is then the same dashed line with different silicon. Same SSH-first habit. Same starter tags. Same “do not expose Ollama’s port to the internet” — a 293-day SentinelLABS–Censys scan measured 175,108 unique exposed, unauthenticated Ollama hosts across 130 countries [5] [6].
After Halo is the inference host, the useful measurement is boring: tokens per second, time to first token, peak unified memory, and whether the output is still a reviewable diff. A large advertised context window is not a reason to send the repository. Start at 16K–32K tokens of context, the way the NVIDIA starting point already does.
What a Halo box costs
The chip behind “Halo” is AMD’s Ryzen AI Max+ 395 (Strix Halo): sixteen cores, a Radeon 8060S integrated GPU, and up to 128GB of soldered LPDDR5X, most of which can be addressed as graphics memory — Framework quotes up to 96GB graphics-addressable on the 128GB build [1]. That last part is the reason to care: the 128GB configuration runs strong-model tags that will not fit on a 16GB discrete card.
It is not cheap, and it has gotten less cheap. As of September 2026 the 128GB boxes look like this [1] [2] [3]:
| Machine | Config | Price (Sep 2026) |
|---|---|---|
| Framework Desktop | 128GB, bring your own SSD | $3,449 (pre-order) |
| GMKtec EVO-X2 | 128GB + 2TB SSD | $3,649.99 |
| Beelink GTR9 Pro | 128GB + 2TB SSD | $4,349 (pre-sale) |
Those numbers are roughly double what the same machines listed for in mid-2026, when $2,000 was the anchor for a 128GB build [3] [4]. The driver is the DRAM contract market — this is soldered LPDDR5X bought in 128GB blocks, so the memory is most of the bill and none of it is upgradeable later. The 64GB builds still sit near $2,000 and will run the starter pair, but 64GB is a strange amount to solder for this use: it costs real money without unlocking the large tags that justify the platform. Check current listings before ordering; these prices have moved every quarter.
The comparison that matters for this roadmap: the current setup already works on a 16GB NVIDIA card that costs a few hundred dollars. Halo money buys headroom — larger strong-model tags, more graphics-addressable memory, one quiet box instead of a tower — not a requirement. That is why it is a roadmap phase and not a prerequisite. If the measurements on the current host say the small models are already good enough, the honest conclusion is to keep the money.
What the headroom is for — heavier refactoring
Refactoring is the clearest case for a bigger local model. It is still mechanical work — exactly what the local_refactor and local_review tools exist for — but it is the kind that punishes small models, because the model has to reproduce long stretches of existing code without skipping, inventing, or “summarizing” sections away. There is a benchmark built to measure precisely that failure: Aider’s refactoring benchmark asks a model to extract 89 large methods from real Python codebases and verifies, by parsing the result, that no code was elided. Completion rates on it separate model tiers sharply — the top listed model scores 92.1% while models a tier or two down land between roughly 20% and 60% [7].
Capability also scales with size inside a single model family. Qwen’s report on the Qwen2.5-Coder family — six sizes trained the same way, published in part “to verify the effectiveness of scaling” — shows the 32B beating the 7B on every code benchmark listed, and scoring 73.7 on Aider’s code-repair benchmark, which Qwen describes as comparable to GPT-4o [8]. A 32B coder at 4-bit quantization is roughly a 20GB model; a dense 70B at the same quantization is about 43GB. Neither fits a 16GB card. Both fit in Halo’s 128GB with room left for context [9].
Throughput is the trade, and for this workflow it is an acceptable one. Strix Halo’s memory bandwidth caps a dense 70B near 5 tokens per second [9] [10] — painful for interactive chat, fine for a tool call that runs in the background and returns a diff. Nobody is watching this output stream; the gate is whether the refactor is correct, not how fast it types. The practical adjustments are boring: give the strong slot a longer timeout, and consider the newer mixture-of-experts coders, which read far fewer weights per token and run at 70–100 tokens per second on the same silicon when speed does matter [10]. That is what “larger strong-model tags” means on this roadmap: a measurably better local_refactor, at a slower pace, still on private hardware. The measurement phase decides whether the bigger tag earns the slot.
What is in the repo
The public spec now has the Halo host as a later profile, the C4 views above, and a planning index. The running server is the same stdio bridge from the first post, with unit tests that run today against the same-machine profile. Two-machine testing over SSH is the next roadmap step; Halo slots in after that without changing the bridge. Clone it, keep the real Ollama URL out of git, and treat every local_* result as untrusted until you apply it.
References
- Framework — Framework Desktop with AMD Ryzen AI Max: Max+ 395 specs (16 cores, Radeon 8060S, LPDDR5X-8000), “up to 96GB of graphics addressable memory,” 128GB configuration at $3,449, sold as pre-order.
- Micro Center — GMKtec EVO-X2 listing: Ryzen AI Max+ 395, 128GB LPDDR5X-8000, 2TB SSD, $3,649.99.
- ComputingForGeeks — Ryzen AI Max+ 395 mini PCs compared (Aug 2026): side-by-side of Framework ($3,449), GMKtec ($3,649.99), Beelink GTR9 Pro ($4,349); documents the June-to-August price doubling and attributes it to DRAM contract pricing on soldered LPDDR5X.
- Liliputing — 128GB Ryzen AI Max+ 395 mini PCs roundup: earlier-2026 price baseline across nine vendors and the note that most of the 128GB can be used as VRAM.
- SentinelLABS + Censys — Silent Brothers (Jan 2026): joint 293-day internet scan measuring 175,108 unique exposed Ollama hosts across 130 countries (7.23M observations); 48% advertised tool-calling capability.
- LeakIX — 12,000 Ollama instances exposed (Feb 2026): 12,269 unauthenticated public instances, ~1,000 vulnerable to CVE-2024-37032, an unauthenticated remote-code-execution chain.
- Aider — Refactoring leaderboard and the benchmark design: 89 large-method extractions from real Python repositories, verified by parsing the output, built to provoke and quantify models eliding code on long outputs. Scores on the published board run from 92.1% at the top to the 20–35% range for weaker models.
- Qwen team — Qwen2.5-Coder family report: six sizes trained identically to “verify the effectiveness of scaling”; 32B-Instruct outperforms 7B across the published code benchmarks (92.7 vs 88.4 on HumanEval) and scores 73.7 on Aider code repair, “performing comparably to GPT-4o.”
- SpecPicks — Strix Halo 128GB local-LLM review: 70B Q4_K_M is a ~43GB model that fits the 128GB unified memory with context headroom; generation is bandwidth-limited on the Ryzen AI Max+ 395.
- DataHardware — Strix Halo tokens per second: dense 70B at 4-bit generates ~5 tok/s; 30B-class mixture-of-experts models run 70–100 tok/s on the same machine; ~256GB/s theoretical, ~215GB/s measured memory bandwidth.
The routing claims about Cursor, Copilot, and Claude — why a desktop MCP bridge is the only path to a private GPU — are documented with vendor sources in the first post’s reference list.
Source: github.com/jmjava/slm-setup — spec.md, docs/c4.md, docs/roadmap.md
git clone git@github.com:jmjava/slm-setup.git
No comments:
Post a Comment