Same MCP, next host — Halo on the slm-setup roadmap

The GPU can move. The bridge should not. The first slm-setup post described a split: a premium cloud model plans and reviews, while a small language model (SLM) on private hardware handles bounded generation through MCP tools. The private model runs under Ollama, an open-source server for running language models locally. “Halo” here means a PC built around AMD’s Strix Halo design, which can share up to 128GB of unified memory between the CPU and integrated graphics. The question is whether that memory makes Halo a worthwhile later Ollama host — not whether to build a second coding system.

Two C4 diagrams make the roadmap concrete. The context view shows who talks to what; the deployment view shows which machine runs each part. The software boundary stays fixed while the machine hosting Ollama changes.

What stays fixed

Context. The private lab uses a premium model for planning while Ollama remains private. Cloud-hosted agents are outside this home-lab design.

Cursor, Copilot, or Claude remains the planner. The private model is exposed through MCP, the Model Context Protocol: an open standard that lets the desktop IDE call a local program through named tools. Default vendor-hosted agents run away from the workstation ^{[16] [17] [18]}; enterprise private-connectivity and self-hosted configurations are possible, but they are separate designs. This home-lab bridge runs on the workstation, where it can reach private hardware without making Ollama public.

Local inference is not an air gap. The premium agent can access workspace files and receives the tool results through the vendor’s service. Halo does not hide that selected context from the planner; it keeps the delegated model generation on private hardware.

The workstation keeps the same stdio MCP server, the same six bounded coding tools, and the same environment settings. Stdio means the IDE launches the server and communicates through standard input and output rather than an MCP network port. The server reads the Ollama address from the environment, commonly through a gitignored .env file. Moving Ollama does not change what the IDE calls.

Deployment — one active host

Variants. Only one host is active. Prefer SSH to that host’s localhost. A private-interface bind is optional. A public bind is not.

The inference host is simply the machine running Ollama. The roadmap changes that host in three controlled stages:

1. Now: Ollama runs on the workstation, and the desktop-to-Ollama path is operational ^[13].
2. Next: Ollama moves to a second private GPU host. SSH local forwarding — an encrypted connection that makes the remote service appear local — keeps Ollama bound to that host’s loopback address, which accepts connections only from the host itself. A robust example is ssh -N -T -o ExitOnForwardFailure=yes -L 127.0.0.1:11436:127.0.0.1:11434 user@<inference-host>; MCP then uses http://127.0.0.1:11436.
3. Later: Halo takes the same host slot. Its AMD software backend still needs validation on the actual machine before any performance claim.

The security rule is unchanged: do not make Ollama internet-reachable. A 293-day SentinelLABS–Censys scan measured 175,108 internet-reachable Ollama hosts across 130 countries ^[5]; LeakIX separately documented 12,269 unauthenticated instances in its February 2026 dataset ^[6].

Current test status

Thirty-six mocked unit tests currently pass. Automated live integration checks also reach a real loopback-only Ollama through the stdio MCP server. A separate manual Cursor check invoked the MCP tools and executed the strong model’s returned test file successfully ^[13]. This validates the same-machine integration, not a complete IDE file-application workflow, the planned SSH deployment, broad model reliability, or Halo performance.

What a Halo box costs

The top-end chip considered here is AMD’s Ryzen AI Max+ 395: sixteen CPU cores, Radeon 8060S integrated graphics, and up to 128GB of soldered LPDDR5X unified memory. Framework says as much as 96GB can be graphics-addressable on its 128GB build ^[1]. That is memory capacity for quantized model weights larger than 16GB — not proof that a particular model, Ollama release, or AMD backend will run well.

Observed September 6, 2026 listings put 128GB systems between $3,449 and $4,349: Framework at $3,449, GMKtec at $3,649.99, and Beelink at $4,349 ^{[1] [2] [3]}. Published comparisons place earlier launch-era systems near $2,000 and attribute much of the later increase to soldered-memory pricing ^[3] [4]. Because that memory cannot be upgraded, capacity is a purchase-time decision. These figures are a dated snapshot, not a quotation.

What would justify buying it

Memory headroom matters only if larger models produce more complete and correct work on representative bounded tasks. Aider’s independent benchmark illustrates the risk: when models extract large methods from real Python projects, completion rates vary sharply and weaker models often skip code ^[7]. That benchmark covers hosted models, not this Halo proposal, so it supplies a useful test shape rather than a result.

Qwen’s vendor-authored report gives a second reason to compare sizes: within the Qwen2.5-Coder family, its 32-billion-parameter model scores above the 7-billion-parameter version on the published coding benchmarks ^[8]. Ollama lists the quantized 32B download at 20GB ^[11], already larger than a 16GB GPU before runtime overhead. Halo has room to evaluate it, but memory fit does not prove a quality gain.

Quality also has to fit the time budget. A direct community benchmark reports 4.7–4.9 output tokens per second for Llama 3.1 70B at 4-bit quantization on Strix Halo ^[9]. At five tokens per second, a 1,000-token answer takes about 200 seconds before prompt processing. That approaches the bridge’s current 300-second strong-model timeout ^[14], so a dense 70B model is practical only for bounded synchronous outputs unless the request contract changes. Backend and long-context results also vary materially ^[10] [12]. Replacing Ollama may require a client change unless the replacement preserves Ollama’s HTTP API.

The purchase criterion is therefore simple: benchmark named models, quantizations, context sizes, and backends on representative work; buy the memory headroom only if the quality gain is worth both the price and the wait. If the current models are already reliable enough, keep the money.

What is in the repo

The public repo provides the runnable MCP server, SSH-first deployment guidance, and dated operational and security evidence ^[13] [15]. Start with the working same-machine profile. Move to a private host only when that need is real, and consider Halo only after representative work shows that larger models would earn their additional cost. Keep machine-specific values out of git and review every local-model result before applying it.

References

1. Framework — Framework Desktop with AMD Ryzen AI Max: Max+ 395 specs (16 cores, Radeon 8060S, LPDDR5X-8000), “up to 96GB of graphics addressable memory,” 128GB configuration at $3,449, sold as pre-order.
2. Micro Center — GMKtec EVO-X2 listing: Ryzen AI Max+ 395, 128GB LPDDR5X-8000, 2TB SSD, $3,649.99.
3. ComputingForGeeks — Ryzen AI Max+ 395 mini PCs compared (Aug 2026): side-by-side of Framework ($3,449), GMKtec ($3,649.99), Beelink GTR9 Pro ($4,349); documents the June-to-August price doubling and attributes it to DRAM contract pricing on soldered LPDDR5X.
4. Liliputing — 128GB Ryzen AI Max+ 395 mini PCs roundup: earlier-2026 price baseline across nine vendors and the note that most of the 128GB can be used as VRAM.
5. SentinelLABS + Censys — Silent Brothers (Jan 2026): joint 293-day internet scan measuring 175,108 unique exposed Ollama hosts across 130 countries (7.23M observations); 48% advertised tool-calling capability.
6. LeakIX — 12,000 Ollama instances exposed (Feb 2026): 12,269 unauthenticated public instances, ~1,000 vulnerable to CVE-2024-37032, an unauthenticated remote-code-execution chain.
7. Aider — Refactoring leaderboard and the benchmark design: 89 large-method extractions from real Python repositories, verified by parsing the output, built to provoke and quantify models eliding code on long outputs. Scores on the published board run from 92.1% at the top down to roughly 21% for the weakest listed configuration.
8. Qwen team — Qwen2.5-Coder family report: six sizes trained identically to “verify the effectiveness of scaling”; 32B-Instruct outperforms 7B across the published code benchmarks (92.7 vs 88.4 on HumanEval) and scores 73.7 on Aider code repair, “performing comparably to GPT-4o.”
9. ignasivt — Strix Halo Guide: community measurements for Llama 3.1 70B Q4_K_M report 4.7–4.9 output tokens per second across short prompt lengths on a Ryzen AI Max+ 395. This is a direct practitioner benchmark, not vendor certification.
10. Strix Halo Wiki — llama.cpp performance: links reproducible backend comparisons and shows that prompt processing, token generation, driver choice, and long-context behavior can differ materially.
11. Ollama — qwen2.5-coder library page: the 32b tag is a 20GB download in the default quantization.
12. Digital Architects — Ryzen AI Max+ 395 local-LLM field notes: a practitioner report describing material stability and performance differences among Vulkan, ROCm, llama.cpp, and Ollama on Strix Halo. These observations motivate validation on the actual machine.
13. slm-setup — Local acceptance results — 2026-09-06: reproducible commands, unit results, live same-machine MCP-to-Ollama checks, direct Cursor invocation, retries, and stated limitations.
14. slm-setup — Ollama client implementation: current synchronous request behavior, 300-second strong-model timeout, and 4,096-token output cap.
15. slm-setup — Repository security scan — 2026-09-06: refreshed Gitleaks history and committed-tree results at repository commit 277f6bd, GitHub secret-scanning alert state, deployment-safety checks, exact counts, and stated limitations.
16. Cursor Docs — Cloud Agent security: each Cursor Cloud Agent runs in an isolated cloud virtual machine rather than on the developer’s laptop.
17. GitHub Docs — About Copilot cloud agent: the hosted agent runs in an ephemeral GitHub Actions environment, while IDE agent mode edits in the local development environment.
18. Anthropic Docs — Claude Code on the web: Anthropic-hosted sessions run in isolated Anthropic-managed virtual machines; organizations can also configure self-hosted cloud environments.

Web references and prices were checked on September 6, 2026. Retail prices and practitioner performance reports are time-sensitive.

Source: github.com/jmjava/slm-setup — spec.md, docs/c4.md, docs/roadmap.md
git clone https://github.com/jmjava/slm-setup.git

Local coding SLM — premium agents, private GPU, one MCP bridge

I still want the expensive model in the chair. I do not want it typing a hundred pytest functions. That work is often bounded and cheap to reject if it is wrong, making it a useful candidate for a small language model (SLM) running on my own hardware. The trick is getting Cursor, Copilot, or Claude to use that model without publishing its Ollama endpoint to the internet.

slm-setup is the setup I am using for that split. The premium agent stays the planner and the reviewer. A local process on the workstation talks to Ollama — an open-source server that runs language models on your own hardware. The local model never becomes “the model in the picker” — the IDE dropdown where you choose GPT or Claude as your chat model. It is a handful of tools the premium agent calls.

The thing that does not work

Cursor can override an OpenAI-compatible base URL, but those requests are assembled on Cursor’s servers. A workstation localhost or home-LAN Ollama URL is therefore outside the default request path; Cursor staff recommend a publicly reachable HTTPS endpoint for that configuration ^{[1] [2] [3]}. I do not want to expose Ollama publicly. The override is also shared by the OpenAI-model slot rather than configured per model, so it conflicts with keeping premium OpenAI-family models available for planning ^[4].

The same boundary applies to default vendor-hosted coding agents: Copilot’s cloud agent runs in GitHub Actions, while Anthropic-hosted Claude Code sessions run in Anthropic-managed virtual machines ^{[5] [7] [8]}. Enterprise self-hosted runners and organization-managed Claude environments are exceptions ^[6] [7], but they are separate designs. This home-lab bridge targets the machine in front of you: desktop Cursor, Copilot agent mode in the IDE, and local Claude Code.

A process on the workstation can reach a private GPU. The default vendor-hosted path does not share that network. That is the whole reason this is MCP — the Model Context Protocol, an open standard that lets a desktop IDE agent launch a local program and call it as a set of named tools — and not a model-provider hack.

Who does what

If the shape of the answer is obvious — tests, a rename, a bounded refactor, a summary, a first-pass review — I ask the local tools. If the problem is architectural, cross-system, or security-sensitive, I keep it on the premium model. If the local answer is thin or wrong, the premium model edits it. I expect to spend some premium tokens deciding and reviewing. I am trying not to spend them emitting the artifact.

you + premium agent
        │  plan, pick files, review
        ▼
local-coding-slm  (stdio, on the workstation)
        │  HTTP to localhost or an SSH local forward
        ▼
Ollama
   fast model              stronger model
 (everyday coding)      (harder local work)

The IDE and the MCP server live on the workstation. Ollama can run on that same machine or, after the planned two-machine test, on a second private host reached through SSH local forwarding — an encrypted connection that makes the remote Ollama service appear local. Machine-specific connection details never go in git.

What the local model is for

One server name: local-coding-slm. The MCP process does not run shell commands, write files, or open a network listener; it makes outbound HTTP calls to the configured Ollama API. It returns text — code, a diff, or markdown — and I decide whether to apply it. I send a few files, not the whole repository.

The tools have deliberately narrow responsibilities: write code, refactor, generate tests, explain, review, and check that Ollama is up. Each one has a short fixed system prompt. The test tool is told to write tests only. If the request is ambiguous, it should ask instead of inventing production changes.

The starter pair is a fast everyday model (qwen3.5:9b) and a stronger coding model (devstral-small-2). Use fast by default and escalate only when its answer is not good enough. The documented starting configuration uses 16K context — a working window of roughly 16,000 tokens; the stronger model may split work between CPU and GPU when GPU memory is tight ^[12]. Increase context only after measuring memory use on your own hardware.

What is tested now

Thirty-six mocked unit tests currently pass. Automated live integration checks also reach a real loopback-only Ollama — bound to 127.0.0.1 for access from this machine only — through the stdio MCP server. In a separate manual Cursor check, Cursor invoked the MCP tools and the returned strong-model test file passed both tests when executed ^[12]. This validates the current same-machine integration, not a complete IDE file-application workflow or broad model reliability.

What is in the repo

The public repo is the spec plus a running server, not a sketch. The server talks to the IDE over stdio — standard input and output, so it opens no network port of its own — and the same-machine path has been exercised against a real local Ollama runtime ^[12]. You also get a wrapper that starts the server, reads settings from the environment (commonly loaded from a gitignored .env), and templates for Cursor, VS Code Copilot, and Claude Code. Machine-specific values are never committed.

Treat local output as untrusted. Apply it, trim it, or throw it away. The premium agent can access workspace files and sends selected context through the vendor’s service; the SLM sees only the snippets passed in the tool call. There is no automatic classifier deciding which model to call. The project instructions keep the rule explicit: bounded mechanical work can go to the local tools; ambiguous, architectural, or security-sensitive work stays with the premium model.

Do not expose Ollama publicly. No ngrok, Cloudflare Tunnel, or router port-forward that makes the API internet-reachable. Ollama binds to localhost by default, and its local API requires no authentication ^[9]. A 293-day SentinelLABS–Censys joint scan measured 175,108 unique internet-reachable Ollama hosts across 130 countries ^[10], and LeakIX found roughly a thousand of the instances it counted still vulnerable to a known unauthenticated remote-code-execution chain ^[11]. For a second host, keep remote Ollama on 127.0.0.1:11434 and create a workstation-only forward: ssh -N -T -o ExitOnForwardFailure=yes -L 127.0.0.1:11436:127.0.0.1:11434 user@<inference-host>. Then point MCP at http://127.0.0.1:11436. A private-interface bind is only a firewall-restricted fallback. The current dated repository scan found no detected leaks or GitHub secret-scanning alerts, while noting that a clean scan is not proof of absence ^[13].

If you already pay for a premium coding agent and have a private GPU, this is the shape to start from. Clone the repo, set OLLAMA_BASE_URL in the gitignored .env file, and measure whether the small model is good enough for the bounded work you actually do.

The host side of this has a roadmap now: an AMD Halo-class box as a later private Ollama host, with the C4 views that show what changes and what does not. That is the follow-up post, Same MCP, next host — Halo on the slm-setup roadmap.

References

1. Cursor Docs — Custom API keys: “Your API key … is sent to our backend with every request because all requests are routed through Cursor’s servers for final prompt building.” Also notes custom keys apply to chat models only; Tab completion stays on Cursor’s models.
2. Cursor Forum (staff reply, Feb 2026) — Connecting local AI server to Cursor does not work: “All BYOK requests go through Cursor’s servers to build prompts, so localhost or local network addresses won’t work because the server can’t reach them. You’ll need to expose your Ollama instance as a public HTTPS endpoint using something like ngrok or Cloudflare Tunnel.”
3. Cursor Forum (staff reply, Mar 2026) — How to use Cursor directly with only my model API?: “There is currently no option to have Cursor communicate directly with your own server without going through Cursor’s backend.” See also staff confirmation that a full bypass is an architectural limitation: “Prompt building, context retrieval, and Cursor Tab and Agent run on our side.”
4. Cursor Forum (staff reply) — routing explanation: with a custom key plus override, “requests for OpenAI-family models (anything that’s not claude-* and not gemini-*) go to your custom endpoint”; the base URL is a single global setting, not per-model.
5. GitHub Docs — About Copilot cloud agent: the coding agent “has access to its own ephemeral development environment, powered by GitHub Actions,” and is “distinct from the ‘agent mode’ feature available in your IDE,” which “makes autonomous edits directly in your local development environment.”
6. GitHub Docs — Customize the agent environment: self-hosted runners can give Copilot access to internal network resources. This is an enterprise exception to GitHub’s default hosted environment. See also the agent firewall documentation: the cloud agent’s internet access is restricted inside the GitHub Actions environment.
7. Anthropic — Claude Code on the web: Anthropic-hosted sessions run in isolated Anthropic-managed VMs; organizations can also configure self-hosted cloud environments.
8. Anthropic — Configure cloud environments: all outbound traffic from cloud sessions passes through Anthropic’s network proxy with allowlist levels (None / Trusted / Custom / Full).
9. Ollama — FAQ and API authentication: Ollama binds to 127.0.0.1:11434 by default, and no authentication is required for its local API.
10. SentinelLABS + Censys — Silent Brothers (Jan 2026): joint 293-day internet scan; 175,108 unique exposed Ollama hosts across 130 countries, 7.23M observations, 48% advertising tool-calling. Censys’s earlier single-day snapshot (Ollama Drama) found 10.6K instances, 1.5K directly promptable.
11. LeakIX — 12,000 Ollama instances exposed (Feb 2026): 12,269 unauthenticated instances, roughly 1,000 running versions vulnerable to CVE-2024-37032 (“Probllama”), an unauthenticated path-traversal-to-RCE chain.
12. slm-setup — Local acceptance results — 2026-09-06: reproducible commands, hardware details, unit results, live same-machine MCP-to-Ollama checks, and a direct Cursor invocation with executed generated tests.
13. slm-setup — Repository security scan — 2026-09-06: the refreshed Gitleaks scan covered repository commit 277f6bd and requested all refs; GitHub returned no secret-scanning alerts; committed-tree and loopback deployment checks passed. The report gives exact scan counts and limits.

Web references were checked on September 6, 2026.

Source: github.com/jmjava/slm-setup — start with spec.md
git clone https://github.com/jmjava/slm-setup.git