The DIF test engine — prove the three layers without collapsing them

A plan that cannot fail in an interesting way is just more markdown. The last post said DIF, the orchestrator, and Embabel answer different questions. This post is the test engine that keeps them from collapsing into one runtime.

Follow-up to Three layers, one day. Source: the working test flow in jmjava/embabel-dif and the integration ladder in docs/ORCH_INTEGRATION_ROADMAP.md.

What “test engine” means here

Not a new product. Not a second daily driver. A stacked set of checks where each rung is allowed to fail before we spend complexity on the next:

./mvnw test                 # unit + CLI + FoldContractTest
                            # EmbabelLivePlatformTest skipped unless DIF_LIVE_EMBABEL=1
./scripts/dif-orch-smoke.sh # FEAT-001 ready + T03; FEAT-099 exit 1
./scripts/dif-orch-day.sh   # fold twice / architect / review / plan --projection
                            # skip when CLI or snapshots missing
./scripts/dif-live-e2e.sh   # orch Guide+Neo4j + JSONL quote + live GOAP

Default CI is the first two boxes. Live Guide and Embabel are opt-in. They reuse the orchestrator’s existing tests/test-guide-stack-live.sh. They do not put Embabel or Guide inside sdlc.sh next.

The engine’s job. Prove the three systems can talk. Prove a missing DIF checkout is skip, not a broken day. Prove a contradictory canvas cannot earn Ready For Coding. Never start a JVM to run next.

Rung A — five named checks, not prose

FoldContractTest is step 1 of the fold iteration plan. The five success criteria are tests:

Check	What fails if we are wrong
Same accepted canvas → same model	Nothing downstream is trustworthy
Review fails without “looks correct”	login-auth-broken still prints RESULT: PASS
Syntax variance does not flip invariants	A DTO rename (FEAT-070) changes what must stay true
Open T## is a MissingObligation	T03 on FEAT-001 disappears into a checklist
Requirement vs non-goal blocks Ready For Coding	FEAT-099 pagination clash still looks green

Harvested canvases under examples/canvases/ are the corpus. The fold learns from real REASONS files, not imagined IR. Adding a backend must not change the CLI or the canvas schema.

Rungs C–D — a script can trust the gate

dif-fold.sh writes a projection and a stable .gate.json. Smoke does not parse stdout for meaning. It reads JSON and exit codes:

{
  "workId": "FEAT-001-order-status-api",
  "readyForImplementation": true,
  "blockingConflicts": [],
  "missingObligations": ["T03"]
}

dif-orch-smoke.sh folds FEAT-001 (exit 0, ready, T03 missing) and FEAT-099 (exit 1, blocking pagination clash). If a sibling orch checkout is present, it folds the live examples/spring-boot-order-api canvas too. Then it hits the silent attach:

DIF_DISABLED=1 check-canvas.sh …   →  dif=skipped   (exit 0)
check-canvas.sh FEAT-001           →  dif=ready     (exit 0)
check-canvas.sh FEAT-099           →  dif=blocked   (exit 1)

One line. Agents do not get a fold dump. Missing DIF is skip, not a new ritual. That is the same opt-in shape as Guide.

Rung G — a scripted day, no Embabel

dif-orch-day.sh is the cheap “full day.” It does not start Embabel, Guide MCP, or replace sdlc.sh next.

1. Fold the same FEAT-001 canvas twice. The two .gate.json files must cmp equal.
2. Architect FEAT-001 → dif=ready, T03 still a missing obligation.
3. Architect FEAT-099 → exit 1, dif=blocked.
4. Review the orch order-status snapshots: dropping auth fails. A DTO rename still passes.
5. plan --projection builds a VerificationPlan from the folded model. No markdown re-parse.
6. Guide JSONL is an optional quote (Decision / Pitfall), not a gate.
7. Missing CLI or missing snapshots → dif=skipped. Present snapshots that drop a safeguard → dif=blocked.

Review uses examples/snapshots/order-status-*.json and the canvas safeguard paths — not the old login fixtures. Syntax-ok vs auth-broken is the whole point: a rename is legal; a dropped safeguard is not.

Rungs H–J — live, still not inside next

dif-live-e2e.sh is the three-way path that already passed here. First it asserts sdlc-engine is not a JVM — help must not mention Spring or Embabel. Then it reuses the orch Guide+Neo4j harness, runs the scripted day, quotes DIF JSONL through GuideClient under a unique Work ID (FEAT-DIF-LIVE-… so it does not collide with orch’s already-projected FEAT-001), and boots the Embabel Spring platform.

Live Embabel (EmbabelLivePlatformTest, DIF_LIVE_EMBABEL=1) runs the fixture GOAP path:

UserInput
  → captureRequest → interpretIntent → foldIntent
  → analyzeRepository → planVerification
  → VerificationPlan (readyForImplementation, missing rotation IT)

The refresh-token wording uses FixtureIntentInterpreter — no LLM. A second test plans an already-folded orch canvas without re-parsing markdown. Conflicts stay on the VerificationPlan (readyForImplementation=false). They are not a GOAP precondition Embabel 1.5 cannot treat as an action post.

Orch CI does not need Maven. It uses tests/fixtures/dif-fold-stub.sh so detect-and-skip / fail-closed can be proven with a fake CLI: skipped, ready, or blocked.

What would fail the engine

The ladder is the falsification list from the last post, turned into commands:

• Two folds of the same canvas disagree → day step 1 fails.
• A requirement vs non-goal pair still looks ready → smoke / architect on FEAT-099 fails.
• An open T03 does not show up → gate assertion fails.
• A DTO rename flips an invariant → SyntaxVarianceTest / review syntax-ok fails.
• Review cannot fail a dropped safeguard without “looks correct” → auth-broken still passes.
• sdlc-engine --help mentions Embabel → live E2E step 0 fails.
• Missing DIF breaks the orch day → skip tests fail.

If people stop reading the canvas because they treat the JSON as source of truth, we failed even if every script is green. The projection stays regenerable and disposable.

Source: github.com/jmjava/embabel-dif
Previous: Three layers, one day — DIF, the orchestrator, and Embabel
Related: sdlc-spdd-orchestrator · Embabel

Three layers, one day — DIF, the orchestrator, and Embabel

Reliable AI engineering does not require every component to be deterministic. It requires determinism at the boundaries where repeatability, traceability, and correctness matter. That is the sentence embabel-dif is testing — not a Merly reimplementation, and not a second daily driver.

Use stochastic reasoning to discover knowledge. Use deterministic representations to operationalize it once it is understood.

The hole the runbook cannot close

Coding agents are good at reading a repository and sounding like they understand it. The understanding is usually implicit and disposable:

prompt + files + luck  →  a one-off theory of the system  →  a patch

The next session starts from zero. It may decide that sessionToken was incidental, that Google login can move, or that an existing test is optional. Nothing in the process remembers which of those beliefs were load-bearing.

sdlc-spdd-orchestrator already attacks the process half: one Work ID, one REASONS Canvas, one phase at a time. Assistants are not allowed to invent a parallel workflow. That is necessary and not sufficient. The canvas is still prose. Architect, review, and sync still ask an LLM to compare the canvas to a diff. Comparison is where implicit intent creeps back in.

Process gates ask “do the prerequisite files exist?” They do not ask “did this canvas contradict itself?” or “did this diff drop a safeguard?”

The remaining hole is checkability. You can follow the runbook perfectly and still ship a contradictory canvas, mark Ready For Coding in prose, or pass review because the change “looks right.”

Three questions, three systems

Planning / requirements     why are we doing this?
REASONS Canvas              what must ship (human contract)
DIF SemanticModel           what must remain true (machine contract)
Embabel GOAP                what action to take on typed facts (optional)
DICE / Guide graph          what did we learn before (retrieval)
SDLC phases                 who is allowed to act

Layer	Owns this question	Must not own
Orchestrator	Who acts when? One Work ID, one canvas, one phase.	Folding facts. Starting a JVM. Being a planner.
DIF	What must stay true? Same accepted canvas → same model. Conflicts fail closed.	Daily orientation. Picking the Work ID. Replacing the canvas.
Embabel	What action to take on already folded facts (optional JVM path).	The fold itself. sdlc.sh next. The human contract.

Git stores what changed. A DIF-style layer stores why it had to, and what must still be true. Embabel, when present, decides what to do next. The orchestrator decides who is allowed to act.

They stay three repos on purpose. Merging Embabel or DIF into the orchestrator would fight its design: it is an installable operating model, not a compiled agent runtime. The contract between them is a file:

spdd/canvas/<WORK-ID>.md            human source of truth
        │
        ▼  fold (deterministic after accept)
.dif/projections/<WORK-ID>.json     machine projection (disposable)
.dif/projections/<WORK-ID>.gate.json
        │
        └─ orch may read the exit code
           it does not start the JVM to run next

A canvas is already a candidate intent. We do not need a new human artifact. We need a projection.

DICE is not DIF

The orchestrator already has Guide DICE as an optional working store. The acronyms smash together. The jobs do not.

DICE  = retrieve what we already believe
        (lessons, decisions, pitfalls, area subgraphs)

DIF   = freeze what must remain true, then verify it
        (intents, invariants, conflicts, obligations)

DICE answers “what did previous work in this area learn?” DIF answers “may this change proceed, and did it preserve the contract?” Both can project from the same committed files. Neither replaces the canvas or the lessons ledger. The ledger stays the system of record; SQLite, Guide, and .dif/projections/ are regenerable.

What landed today

Yesterday’s prototype proved a typed fold on a refresh-token fixture. Today the fold attaches to real REASONS canvases and fails closed in a way a script can trust.

./mvnw test
./scripts/dif-orch-smoke.sh
./scripts/dif-fold.sh --canvas examples/canvases/FEAT-001-order-status-api.md
./scripts/dif-fold.sh architect --projection .dif/projections/FEAT-099-pagination-conflict.json
./scripts/dif-fold.sh review --before examples/snapshots/login-before.json \
                            --after examples/snapshots/login-auth-broken.json

dif-fold does not start Embabel. fold writes a projection and a stable .gate.json (readyForImplementation, blockingConflicts, missingObligations) that a script can read without parsing stdout. architect and review fail closed: exit 1 means not Ready For Coding, or invariants were not preserved.

After a fold, “ready” is allowed to mean this:

• A mutually exclusive pair (“must paginate” vs “non-goal: pagination”) blocks Ready For Coding. The next command is clarification, not code. That is FEAT-099.
• An open operation (T03) shows up as a MissingObligation, not a forgotten checklist box.
• Two folds of the same accepted canvas produce the same model.
• Syntax may change (DTO names, test style). Preservation of auth and unrelated endpoints must not. That is FEAT-070.
• Review can fail a required safeguard without asking an LLM whether the change looks correct.

The ten fold-iteration steps from the steal list are implemented: fold contract tests, harvested canvases, heading classification, quoted conflicts, open-T## obligations, syntax-out-of-invariants, an optional Alloy sketch, architect/review attach, and a plan path that builds a VerificationPlan without making Guide required.

Take the idea only as far as it makes sense

The knowledge that actually hurts is not “which slash command is next.” The orchestrator already answers that. The tax is shipping a contradictory canvas or a dropped safeguard while the runbook stays green.

The filter for every attach:

Does this make the existing orchestrator commands harder to get wrong, without adding a new ritual?

Do	Do not
Keep claim → next → architect → one T## → review as the only user surface	Add dif-fold.sh next as a second daily driver
When DIF is installed, architect cannot earn Ready For Coding on a requirement vs non-goal clash	Teach users fold / projection / .gate.json as a parallel workflow
When DIF is missing, the day is unchanged	Require Embabel, Java, or OpenAI to run next
Review can fail a dropped safeguard without “looks correct”	Replace sdlc.sh gate process checks with the fold

A new orchestrator user who never heard of DIF should have a better day if it is installed, and the same day if it is not. Silent fail-closed on existing architect / code is DIF doing DIF’s job: the readiness string becomes earned. The runbook stays the orchestrator’s. Embabel stays later and optional. Wiring it into next would be the other collapse.

Path, and what would falsify it

1. DIF      canvas → SemanticModel CLI      no Embabel          (working)
2. Orch     architect / code attach         if CLI present      (silent, opt-in)
3. DIF      Embabel GOAP for JVM targets    orch still picks Work ID / T##
4. Optional project invariants into Guide   shared vocabulary, still not required

Step 1 first: if the same canvas does not fold the same way twice, nothing downstream is trustworthy. Step 2 next: attaching an exit code is cheaper than inventing a new phase. Embabel later. Guide last — retrieval already works.

The idea is wrong if two folds disagree, if review still cannot fail a safeguard without “looks correct,” if a DTO rename flips a required invariant, if sdlc.sh next starts a JVM, or if developers need a second next to have a correct day. The projection must stay regenerable. If people stop reading the canvas, we failed even if the JSON is pretty.

Source: github.com/jmjava/embabel-dif
Publication plan: BLOG_DIF_ORCH_EMBABEL.md
Related: sdlc-spdd-orchestrator · Embabel · embabel-v1-learning

Unreal Playground — Python designs a hole, Unreal and Blender build it

A golf hole should be a typed object before it is a mesh. courseforge/unreal-playground is a Python-first learning environment for AI-assisted course design. Agents write a validated CourseDesign. Unreal evaluates it in a game-engine world. Blender realizes it as a portable hole.glb. Then Python scores, critiques, and revises.

Python owns orchestration, schemas, scoring, storage, and the learning loop. Unreal and Blender are interchangeable backends — including a pure-Python fake so the whole loop runs without Docker or an editor.

The loop

prompt
  │
  ▼
DesignerAgent ──► CourseDesign JSON
  │                 (tee, green, fairway spline, hazards)
  ├──────────────► Unreal  evaluate  ──► metrics + camera PNGs
  └──────────────► Blender realize   ──► hole.glb + preview
  │
  ▼
PlayabilityAgent ──► score / critique ──► revise ──► next iteration

They are not two copies of the same renderer. Unreal answers “is this hole measurable as built?” Blender answers “what does this hole look like as geometry?” Both consume the same design object.

# Offline — no Unreal, no Docker
python python_env/orchestration/run_iteration.py \
  --prompt "short risk-reward par 4 in a Pine Barrens style" --fake

python python_env/orchestration/run_experiment.py \
  --prompt "links par 5 for a scratch golfer" --iterations 3 --fake

Each run lands under python_env/datasets/generated/<job_id>/: prompt, design JSON, job JSON, metrics, score, critique, screenshots, logs.

Where the engines actually are

Phase 1 is done. Pydantic schemas, deterministic designer/routing/critic agents, fake Unreal, run_iteration / run_experiment / run_batch, and dataset archival. The MVP hole is a short Pine Barrens par 4 you can generate on a laptop.

Phase 2 wired the CourseForge worker standard. A persistent unreal-worker runs the versioned unreal-golf-build job package. Backends swap without changing the agent graph: FakeUnrealRunner, WorkerUnrealRunner, one-shot Docker, or a host LocalUnrealRunner gated by COURSEFORGE_REAL_UNREAL=1.

Dual-engine graphics (FEAT-003) is the recent ship. Host Blender 4.x exports a real hole.glb plus Cycles preview. Host Unreal now goes past placeholder boxes:

• fairway SplineComponent, rough and tree-line proxies
• heightmap .r16 + real ALandscape import through a thin GolfCourse C++ bridge
• PCG graphs + volume ISM fill when the commandlet cannot tick a StaticMeshSpawner
• InstancedFoliage tree line from a saved foliage type

End-to-end Embabel iterations have run with both host binaries in the same job (Landscape + PCG on Unreal, glb/preview on Blender). Kind blender-worker already runs the same package over HTTP. Full in-pod Unreal golf jobs are still the open edge — the 35–49 GB worker image is a local/Kind concern, not CI.

Embabel on top of the same contracts

A Python mirror of Embabel’s nested-agent (Matryoshka) pattern lives in python_env/embabel_explore/. CourseDesignAgent nests design → evaluate → realize_blender → critique. Placement (fake / host binary / Kind worker) swaps underneath without rewriting the graph.

The gated designer studio (golf-embabel-web on port 8765) is chat + before/after, validation gates, and inline revise/redo. The same GUI can deploy into the CourseForge Kind suite. Cookbook recipes — type chaining, conditions, stuck recovery, RepeatUntil, thinking, streaming — are mapped onto this golf loop, not a second travel-agent demo.

Process sits on SDLC-SPDD canvases (FEAT-001 MVP hole, FEAT-002 worker integration, FEAT-003 dual-engine). The interesting remaining work is richer artist-authored PCG assets and a clean Kind Unreal worker run — not another schema rewrite.

Source: github.com/courseforge/unreal-playground
Related: embabel-v1-learning · SDLC-SPDD

CDK Cost Killer — tear down the stacks that keep billing

Forgotten CDK labs keep billing after you stop caring about them. cdk-cost-killer is a small CDK app that finds those stacks and tears them down — nightly for hygiene, and hourly only while you are already over budget.

It does not create a budget. It reads the account budget named Monthly budget (default limit $30; the live AWS Budgets number wins). The whole design is one Lambda, two schedules, and a few hard skip rules.

The loop

EventBridge Scheduler
   │
   ├─ 9:00 PM America/New_York  → action=kill, reason=nightly
   └─ every hour                → action=kill, reason=hourly
                                      │
                                      ▼
                              cdk-cost-killer Lambda
                                      │
                    ┌─────────────────┴─────────────────┐
                    │ read Monthly budget (actual vs cap)│
                    └─────────────────┬─────────────────┘
                                      │
              hourly + under budget? ─┤── yes → return (no-op)
                                      │
                                      ▼
                         scan enabled regions
                         group nested stacks by root
                         stop EC2 first, then DeleteStack

A third path is optional: subscribe the stack output BudgetAlertTopicArn to the budget’s 100% actual alert. SNS then invokes the same Lambda with reason=budget-notification instead of waiting for the next hourly check.

What is expensive enough to kill

A stack family is torn down only when it holds a resource that keeps costing money while it exists: EC2 instances, Elastic IPs, NAT gateways, VPC endpoints, load balancers, Auto Scaling groups, RDS, Redshift, ElastiCache, OpenSearch, EKS, ECS services.

Default scope is CDK only (AWS::CDK::Metadata or a CDK description). Set targetScope to all-cfn if you want any CloudFormation stack with those resources.

Always skipped:

• this CdkCostKiller stack
• CDKToolkit bootstrap stacks
• anything tagged CostKillerProtect=true

EC2 in a doomed family is stopped first so compute charges drop while CloudFormation delete finishes. Elastic IPs and load balancers only stop costing money after the stack is gone.

Safety switches

Knob	Default	Why it exists
dryRun	true	Log would-stop / would-delete only. Arm after CloudWatch review.
enabled	true	Redeploy false to DISABLE both schedules and make the Lambda a no-op.
protectTagKey	CostKillerProtect	Tag keepers. Nested stacks inherit the root family’s decision.
regions	all enabled	Optional allowlist, e.g. us-east-1,us-east-2.

Hourly is not a second nightly. The hourly pass reads the live budget and bails if spend is still under the cap. After the billing period resets, those invocations become no-ops again. Nightly still runs regardless — that is the “I forgot to destroy the lab” cleanup.

npx cdk deploy                    # dry-run on by default
npx cdk deploy -c dryRun=false    # arm after reviewing logs
npx cdk deploy -c enabled=false   # off switch, stack stays

Tags.of(stack).add("CostKillerProtect", "true");

Source: github.com/jmjava/cdk-cost-killer

Embabel V1 Learning — one branch, two versions, filmed cheat sheet

Memorize the rules, then debug the agents. embabel-v1-learning is a study repo for the Embabel Agent Framework: Java and Kotlin side by side, guided unit tests, and a cheat sheet extracted from the official User Guide and Cookbook — not a second copy of the cookbook travel recipes.

The important constraint is one branch. Default Maven pins Embabel 1.0 (lessons 01–15). -Pembabel-15 compiles the 1.5 extras (thinking traces, streaming objects, message lists, tool-call inspectors) without turning main into a 1.5-only fork.

./mvnw test                 # Embabel 1.0 — lessons 01–15
./mvnw test -Pembabel-15    # plus thinking / streaming extras

The mental model in 30 seconds

input → blackboard types
      → planner picks next @Action
      → action may call LLM / code / tools / subagents
      → return value posts new types → REPLAN
      → @AchievesGoal return type → DONE

GOAP does not run your methods top-to-bottom. Types are the wiring. After every action Embabel reassesses the world (OODA). Returning a new object, null, or flipping a condition changes the next step. That is the first thing the top 10 asks you to internalize.

What the curriculum now includes

The study path is built so you can spend 60–90 minutes and actually remember something:

• Cheat sheet + printable PDF — extracted API and planner rules. Keep it next to the debugger.
• Lessons 01–15 — injected Ai, annotation agents, DICE tools, HITL, conditions/bindings, subagents, RepeatUntil, planner choice, @State loops, guardrails, stuck recovery, AgentInvocation, Kotlin DSL.
• 1.5 extras — action cost, createObjectIfPossible, thinking, streaming, fromMessages, tool-call inspectors. Same @Action style; extra PromptRunner surface.
• Review circuit — checkbox path: orient → debug Write/Review → tools/conditions/guardrails → planners.
• Templates + snippets — copy-paste Java/Kotlin skeletons under templates/; type emb- in the editor (emb-agent, emb-hitl, emb-subagent, emb-dsl…).

Unit tests assert structure, not model poetry. Guided tests use FakeOperationContext: prompt contents, temperature, tool attachment, condition predicates. Live LLM judgment stays in the optional shells.

New: a filmed cheat sheet

The latest work is a Memory OS palace for the cheat sheet itself — not a narrated walkthrough of cookbook recipes. Two floors, twelve loci, one bronze type-ingot creature walking guessable body slots. Each organ is absurd and is the concept.

Floor	What you walk
1 — 1.0 mental model	Types as wiring, replan/OODA, mix code + LLM, tools attach per call, named bindings, four planner hats
2 — PromptRunner + 1.5	createObject, soft-fail null, message envelopes, thinking traces, streaming JSON bricks, tool inspectors

Published lengths: full film about 12 minutes; each floor about 5.5–5.7. The player is on GitHub Pages. Overlay labels and Q/A are stamped by the engine — stills are not allowed to paint lettering — so a rebuild on a newer Memory OS contract stays honest.

How to study it: skim the cheat sheet, watch a floor, then debug the matching *GuidedTest. The film is the retrieval hook; the test is the proof you understood the rule.

Source: github.com/jmjava/embabel-v1-learning
Player: jmjava.github.io/embabel-v1-learning
Related: Embabel Guide · memory-os

Memory OS — method-of-loci study films from Markdown

A study script should become a walk you can replay. That is the bet behind memory-os: write a palace in Markdown, compile it to a spec, and build a narrated, AI-illustrated method-of-loci film — one locus, one concept, one memorable image.

It is built on the pipeline we already proved in docgen (Markdown → TTS → declarative specs → ffmpeg), with the visual layer swapped to the OpenAI Images API so every room gets an exaggerated mnemonic instead of a Manim box diagram.

How the engine fits together

Humans write spec.md. The engine owns the rest.

your-palace.md
      │  memoryos compile
      ▼
your-palace.palace.yaml     # canonical spec
      ├─ enrich   → ~30s explanation per locus (LLM, reviewable in YAML)
      ├─ images   → images/<floor>/NN-locus.png
      ├─ narrate  → audio/<floor>/NN-locus.mp3
      └─ render   → build/video/<palace>.mp4

Sync is the core guarantee. Rendering is audio-first: one continuous narration track, every cut recorded in build/timeline.json, clips cut to those markers. A locus stays on screen until its explanation (plus a recall pause) finishes — never less than video.min_locus_sec (default 30 seconds). Image changes cannot drift ahead of the voice.

Recent engine work that mattered in practice:

• Preview clips now have sound. Early builds encoded build/clips/*.mp4 video-only (-an). The MP3s were fine; the previews were silent. Each clip now muxes its own narration so a single scene is watchable, while the assembled film still uses one continuous AAC track.
• Enrich is reviewable. Thin study-script lines expand to ~30s explanations inside the YAML. You edit the prose, not a generated MP4.
• LAN + Pages viewers. memoryos serve binds on the LAN for iPad/Safari; memoryos pages emits a static tree with chapter / section / scene navigation and viewed/unviewed state.
• Cheap, incremental images. Default is gpt-image-1-mini at medium quality (~$0.015/locus). Existing assets are reused unless you pass --force.

Dogfood: Spring Authorization Server palace

The first full palace is the Spring Authorization Server lab — the same design documented in examples/spring-auth-server/docs/c4-players.md. Vue and mobile are public clients. They talk only to Spring Authorization Server with PKCE. They never call Google or Apple token endpoints.

Token A — Google or Apple id_token. SAS is the IdP’s client.
Token B — Spring Auth id_token. The app is SAS’s client.
AT / RT — Spring Auth access and refresh tokens. These are what resource servers accept.

That three-token split is the whole point of filming it. Floor 1 already had stove = ID token (identity for the client, not an API credential) and refrigerator = access token (presented to APIs). The newer wings add the federation story: Apple/Google mint Token A for the auth server; the auth server mints Token B + AT/RT for the apps. You do not send Google’s id_token to your APIs.

The palace is specified as seven floors:

1. OAuth / OIDC protocol — what happens
2. Spring Authorization Server internals — which components make it happen
3. Authorization code + PKCE
4. Token and session lifecycle
5. Debugging — where the request broke
6. System flow — Vue, mobile, Cortex, Redis, ThreadLocal
7. Config paths — the long AuthorizationServerConfig / SecurityConfig methods

Each floor also has a federation wing (Sign in with Google / Apple). Loci stay incremental: existing images and audio are reused unless you pass --force.

memoryos compile examples/spring-auth-server/spec.md \
  --id spring-auth-server \
  -o examples/spring-auth-server/spring-auth-server.palace.yaml

memoryos build examples/spring-auth-server/spring-auth-server.palace.yaml --floor floor-1
memoryos concat … --floors floor-1,floor-2,floor-3
memoryos serve … --concat

What is next

The engine is already a consumer library, not a one-off renderer. embabel-v1-learning pins it and publishes cheat-sheet films to GitHub Pages — two floors, twelve loci, about twelve minutes. Next work on this repo is more palace coverage (federation / refresh / Cortex session-token cache) and keeping the CLI contract stable so study repos can rebuild without vendoring the engine.

Source: github.com/jmjava/memory-os
Related: docgen · embabel-v1-learning · google-oauth-poc

Obsidian MCP — engineering memory that stays Markdown

Assistant memory should be files you can open. obsidian-mcp (0.1.0) is a local MCP server that gives Cursor and GitHub Copilot persistent engineering memory — stored as ordinary Markdown in an Obsidian vault. Obsidian does not need to be running. There is no community plugin and no hosted memory API.

The vault is the source of truth. Notes stay readable in Obsidian, git, or any text editor. The same stdio process works for both Cursor (.cursor/mcp.json) and Copilot / VS Code (.vscode/mcp.json).

The loop

Cursor / Copilot
        │  MCP stdio
        ▼
obsidian-dev-memory
        │
        ▼
Obsidian Markdown vault
  AI Memory/Projects/<slug>/
    Project State.md
    Sessions/YYYY-MM-DD.md
    Decisions/YYYY-MM-DD-<slug>.md

1. Before substantial work, the assistant calls get_project_context.
2. After a real implementation, it calls capture_work_session.
3. When an architecture choice lands, it calls record_decision.
4. When overall status changes, it calls update_project_state.

Typo fixes and one-line mechanical edits are not memory. The Cursor rule and Copilot instructions say so explicitly, so the vault does not fill with noise.

The seven tools

Tool	What it does
get_project_context	Project State plus newest sessions and decisions. Empty sections if the project is new — never the whole vault.
capture_work_session	Append a timestamped section to today’s session note. Optional Git snapshot: repo, branch, short SHA, dirty flag, short file list. No full diffs.
record_decision	Write YYYY-MM-DD-<slug>.md. Collision adds -2, -3 — never overwrite.
update_project_state	Replace the concise current-state note. Not a session log.
search_memory	Local filename and text search over that project’s memory.
read_note	One vault-relative Markdown file.
append_daily_note	Append to Daily/YYYY-MM-DD.md. Never overwrite existing contents.

Safety is the product

This is not a general filesystem API. Every note path must resolve inside OBSIDIAN_VAULT_PATH. Absolute paths, ../ traversal, and detectable symlink escapes are rejected. Writes are atomic (tempfile + os.replace).

Secrets never land in the vault on purpose. Values that look like keys, tokens, JWTs, private keys, or password= assignments are replaced with [redacted-secret] before write. The assistant instructions also forbid persisting .env contents, database credentials, and customer data.

Wire it into a project

export OBSIDIAN_VAULT_PATH="$HOME/Documents/ObsidianVault"
uv sync
uv run python -m obsidian_dev_memory

./scripts/install-project.sh \
  --project /path/to/your-app \
  --vault "$OBSIDIAN_VAULT_PATH"

The installer merges MCP JSON (it does not wipe unrelated servers) and drops the Cursor rule plus Copilot instructions so both assistants use the same memory habits. Point OBSIDIAN_VAULT_PATH at a real vault directory; the server does not auto-load .env files.

0.1.0 is the first cut that is worth installing: seven tools, vault confinement, Git context on sessions, and an installer that does not destroy existing MCP config. Next work is whatever the vault teaches us we forgot to remember.

Source: github.com/jmjava/obsidian-mcp

SDLC-SPDD storage v3 — one folder, one ledger, query don't bulk-read

Agent memory only works if humans can audit it and models don't have to read the whole repo every session. That was the gap in early SDLC-SPDD dogfood: progress mirrors beside the ledger, feature folders under agent-context/, and “just grep the lessons file” retrieval. Storage v3 — now on main in sdlc-spdd-orchestrator — collapses the layout, commits to one JSONL ledger, and treats everything else as query or projection.

One folder to install

Framework-owned paths live under a single home: <repo>/sdlc-spdd/. Requirements, canvases, harness skills, installed scripts, and committed memory sit together; runtime state stays gitignored under .sdlc/.

09-install-layout — what setup-agent-prompts.sh / upgrade-project.sh lay down.

Legacy sprawled installs (framework dirs at repo root) still resolve until you run sdlc-engine storage migrate --consolidate. New projects get the single-folder layout from day one.

Ledger-first memory

The committed system of record is one append-only file: spdd/memory/lessons.jsonl. Work claim/release events go to spdd/memory/registry.jsonl. Neither file is hand-edited — agents stage via sdlc.sh capture and promote via sdlc.sh accept at retro/sync.

05-storage-model — one ledger; sqlite and Guide are downstream.

Stage quietly, accept at the gate

Captures land in gitignored .sdlc/staged/lessons.jsonl during coding and review. Nothing hits the committed ledger until retro or sync runs accept — one batched promotion instead of capture noise in git history.

06-stage-then-accept — the quiet capture / loud accept split.

Projections you can rebuild

Optional backends — local SQLite (.sdlc/index.sqlite) and Guide DICE (Neo4j via orch-guide) — are regenerable projections of the same ledger. One write path; sdlc-engine context parity checks they still match.

08-projection-parity — if Guide is down, files still win.

Lifecycle with storage gates

The SPDD hybrid lifecycle (Initialize → Analysis → … → Retro → Sync) now has explicit storage checkpoints: analysis indexes land in staged memory; retro/sync accept promotes keepers before the next Work ID.

04-lifecycle-flow — when memory moves from hot session to durable ledger.

What sits inside the home folder

Adapters talk to Cursor/Copilot/Claude; the Python engine owns workflow, persistence config, and optional Guide ops; shell scripts remain the supported install path for consumers.

02-container — the moving parts after v3.

Mental model in one line: canvases and requirements are read directly; the lessons ledger is the committed record; Guide and SQLite are working stores you query on demand; captures stage quietly and accept at gates. Full spec: docs/storage-v3.md. All diagrams: docs/diagrams/.

What we shipped to get here

• PR #141 — storage v3 on main: single-folder install, JSONL ledger + registry, staged captures, test-suite restructure (unit / integration / e2e), migration tooling.
• Consumer matrix — live install scenarios green; two intentional skips where dogfood canvas is empty (documented, not failures).
• PR #148 (in flight) — ADF template library + Vue3 ops console rebased onto v3 paths; closes stale PR #115.

Try it

1. Read storage v3 and skim the PlantUML sources (./scripts/render-diagrams.sh regenerates SVG).
2. On an existing install: sdlc-engine storage status then migrate when ready.
3. Opt into Guide only when you want cross-work graph retrieval — file indexes remain the baseline.

— John · github.com/jmjava/sdlc-spdd-orchestrator

Ongoing — dual-camera golf swing capture and MediaPipe analysis

ONGOING WORK

Practice improves when you can see what actually happened in the swing — not only feel. I’m building menkelabs/camera_recorder: a Python app that records synchronized dual USB cameras and runs MediaPipe pose analysis so a session can go from capture to metrics without a separate desktop tool chain.

Two cameras: face-on in front (golfer faces this lens) + down-the-line along the target line.

What it is

A Flask web GUI (browser at localhost:5000) for configure → record → analyze → compare → archive. No heavy desktop UI stack: live MJPEG previews, property sliders, and tabbed workflow on Windows and Linux.

• Dual capture — two USB cameras, threaded per-camera streams, high frame-rate recording target (up to ~120 FPS when the hardware allows).
• Camera roles — Face-On vs Down-the-Line; labels and scoring follow the role you assign.
• MediaPipe biomechanics — pose detection with metrics across rotation (shoulder/hip turn, X-factor, tempo), position (sway, spine), and body (lead arm, knee flex, weight shift).
• Swing phases — Address through Follow-through, with frame navigation and phase overlay on the charts.

The practice loop

Armed → record → analyze → review → next swing.

The interesting part is not a single recording — it’s the loop that keeps you in the bay:

• Auto swing detection — optional hands-free start/stop from real-time shoulder-turn monitoring (lightweight MediaPipe path while armed).
• Analysis playback — side-by-side annotated face-on + DTL panels, pose skeleton, speed control, low-memory JPEG frame store.
• Score + drills — 0–100 / A–F from the same good/ok ranges as the metric cards; strengths, focus areas, suggested drills.
• Compare — any two swings, delta cards, normalized timeline overlay; pin a reference “golden” swing.
• Progress — trends across sessions; favorites and practice notes on recordings.
• Archive — push sessions to an external disk with space/status in Settings.

Session mode is meant for continuous practice: stay armed, capture, review, go again — without restarting the app.

Where it stands

This is ongoing. The repo is public and usable for dual-cam practice and analysis today; the feature surface is already wide (checklist, USB bandwidth warnings, metronome, report/clip export). Expect rough edges, hardware-specific camera quirks, and iteration on scoring and detection thresholds.

Quick start from the repo:

pip install -r requirements.txt
python scripts/flask_gui.py
# open http://localhost:5000

Lower-end machines can drop MediaPipe complexity (--model-complexity 0) for faster analysis.

Why I’m writing this now

Same theme as other recent posts: keep the larger concept moving, and close the gap to working code. Here the concept is a local practice instrument — dual views, pose metrics, comparison over time — owned in a small Flask + OpenCV + MediaPipe stack rather than waiting on a full product surface.

Follow along at github.com/menkelabs/camera_recorder. More as the loop hardens.

— John · jmenke.blogspot.com

Thinking in code, validated — LLMs, not Neuralink

In 2019 I wrote that I think more than I code. The main point was not that coding was beneath me. It was that I could progress on larger concepts without implementing everything — keep moving the architecture, the paths of thought, the systems picture, even when leaf code could not keep up. That was a real compromise. The punchline, years later: we no longer have to make that compromise. With LLMs, thinking can turn into code at a pace that matches the concepts. In 2020 I looked to Neuralink as a possible path; what we got first was models that compile structured thought into candidate implementation.

The old claim: thought ↔ code. The new tool: models that turn structured thought into running code.

What I argued then

In why don’t I code as much as I think? — the year ahead (Dec 2019), the point was not laziness. It was how to keep moving when you cannot implement everything as fast as you can think it. The valuable part is the path of thought — channel it into repeatable patterns, and you can think in code even when the leaf work is unfinished. Implementation can change. The thought pattern is what matters.

“As long as thought patterns can be channeled into standard repeatable patterns, it should be possible to in-effect ‘think in code.’”

So “I think more than I code” meant: stay on the larger concepts; do not let unfinished leaf work freeze progress. That was empowering — and it was also a compromise. You accepted a gap between how far the thinking had gone and how much was actually built.

In Revisiting the idea of thinking in code — Neuralink (Aug 2020) I restated the thesis — “Code is thought and thought is code — it’s bi-directional” — and took Neuralink seriously as a candidate path: if thought could be read as signal, then thought is code in a much more direct sense. I also wrote about gamification and physical visualization as ways to organize thought into structure people can steer.

That was not a metaphor. It was a real bet on how the last mile from mind to machine might close. What actually closed a usable last mile — sooner, and without an implant — was generative coding models. They did not replace Neuralink as an idea; they validated the process I cared about: channel thought into something a machine can realize as code.

One place the idea showed up: levels of code

One application: thought → pattern → DSL → orchestration → running compute. Not the only home for the idea.

Thinking in code is bigger than any one stack. DSLs, Kubernetes, and orchestration were not the whole thesis — they were a context where the idea seemed to apply to problems I was living in: how do you express control across a fabric of compute without drowning in leaf detail?

In that lane, the same idea showed up as levels of code:

• The ability to cycle through resources phase 2 (Oct 2020) — “second-level coders” write orchestration that moves lower-level logic across resources; multi-level chess.
• OpenKruise Sidecar set for data locality (Sep 2020) — containers as beans, Pods as local contexts, Kubernetes as parent context: OO thinking lifted into infrastructure.
• What can be created with CDK8s? (Jan 2021) — “configuration is code”; intent → DSL → object graph → running infrastructure.
• Finally took another look at MPS… (Jul 2023) — a DSL to write code, then ChatGPT to write the code that writes code: another force multiplier in the same spirit.

In that application, a useful chain looked like:

Thought → repeatable pattern → execution DSL → orchestration graph → running compute

Useful — and still only one surface. The core claim was that thought patterns, once channeled, can become verifiable reproductions in code, wherever that code lives.

Where the ideas landed: Uber Language of Compute

Over years of posts — DSLs for execution, resource, and data; multi-level orchestration; operators; locality; CDK8s; MPS — those threads did not stay as separate notes. They coalesced into a larger working model: the Uber Language of Compute (and later notes, v2.0 with AI-powered design).

That model is where details like “realizable in many languages” and “control flowing across a fabric of compute resources” belong — not as the definition of thinking in code, but as how the uber-language was meant to work in practice: patterns that compose across containers, graphs, and control planes. The smaller posts were applications and probes. The uber-language is the larger working model that held them.

And here is the part that still surprises me looking back: the blog series itself ended up becoming the spec. There was no separate requirements doc waiting offline. Writing the ideas in public — iterating titles, diagrams, “how would this work?” pieces — was specifying the larger system. Progress on the concept lived in the posts. Implementation could lag; the series kept the working model alive until tools (and later LLMs) could catch up.

That catch-up is no longer hypothetical. There is working code for the uber-language now: jmjava/uber-lang-of-compute. The blog was the spec; the repo is the implementation catching the concepts.

What actually arrived: LLMs validate the process

Same notebooks of intent — now with a model that can emit the leaf code.

Generative coding models did not replace the need to think. They validated the process — and removed the compromise. You can still progress on larger concepts first. Now you do not have to leave so much unimplemented. With an LLM in the loop, structured thought can become running systems at a velocity closer to the thinking itself — not via a brain interface, but via language, prompts, and review.

With an LLM in the loop:

• You still have to think in patterns — intent, constraints, structure, acceptance criteria (DSL when it fits; plain language when it does).
• The model can materialize large amounts of candidate code from that thought — so the concept does not sit unimplemented by default.
• You remain responsible for judgment — review, tests, architecture, what not to ship.

“I think more than I code” was the honest description of how progress used to work under a hard ceiling on implementation speed. That ceiling moved. The scarce resource is still the thinking; the model aids turning that think into code so the larger concept and the build can advance together. Neuralink remains a separate, literal bet about reading the brain. What we have now is different: natural language and structured artifacts as an encoding of thought, and the model as a compiler from that encoding toward running systems.

I did not see LLMs coming when I wrote the 2019 piece. Looking back, the multi-level orchestration posts — and the uber-language they fed — were already progress at the concept level, with the blog as living spec. LLMs make closing the implementation gap available far beyond any single DSL or platform.

What this is not

• It is not “the AI thinks so I don’t have to.” Unstructured vibes still produce slop.
• It is not claiming Neuralink was never serious — only that LLM-aided coding is what validated the thinking-in-code process first.
• It is not “DSLs were the answer.” They were an application. The process can exist in many contexts.
• It is not the end of craft. Contracts and verification still decide whether thought becomes a system you can trust.

Closing the loop

2019: progress larger concepts even when you cannot implement everything — think in code under that compromise.
2020: restated as bidirectional — and a real look at Neuralink as a possible path.
Along the way: the threads become the Uber Language of Compute — the blog series becomes the spec; uber-lang-of-compute is working code for that model.
Now: LLMs validate the process and lift the compromise — thinking can turn into code with model aid, so concept and implementation need not diverge by default. Not Neuralink. Review still matters.

Neuralink may or may not arrive later. The process was always the point. The series was already the blueprint — and the larger concept is no longer only half-built.

— John · earlier posts linked above on jmenke.blogspot.com

Coming soon — Guide as optional agent context for SDLC-SPDD

COMING SOON

File-based agent memory works — until you want cross-run lessons that stay auditable. We are expanding SDLC-SPDD Orchestrator with an optional context backend powered by Embabel Guide + Neo4j. Markdown stays canonical. Guide adds retrieval on top when you opt in.

Files first. Guide only when the marker is present and the service answers.

The idea

Today every /sdlc-spdd-* phase can already load context from indexes under agent-context/ and canvases under spdd/canvas/. That path stays the default forever.

The spike explores a DICE-style hybrid (Domain-Integrated Context Engineering): the same markdown the workflow already produces is projected into Guide twice — as RAG chunks and as typed domain entities — so the next session can ask not only “what text is similar?” but “what is connected to this Work ID or code area?”

Dual ingest: chunks for discovery, entities + edges for explainable inclusion.

Three retrieval legs, one join key

Work ID is the join key across lexical, embedding, and domain-graph legs.

1. Lexical / area index — what you have today: deterministic, auditable, exact identifiers.
2. Embedding discovery — Guide RAG (docs_textSearch / docs_vectorSearch) to find entry points by paraphrase.
3. Domain graph (DICE) — typed nodes such as WorkId, Canvas, Area, Decision, Pitfall, Pattern with edges (canvas, area, decision, pitfall, pattern, about). Inclusion is justified by a link, not only a cosine score.

That last point matters for agent trust: when a prior pitfall from FEAT-001 shows up while coding FEAT-009 on the same area, you should be able to say why it was pulled in.

Optional at runtime — never assumed

Installs opt in with a marker (agent-context/harness/guide-dice.md via init-project.sh --with-guide). Every command still probes first:

• No marker → CONTEXT_BACKEND=files (normal, not an error)
• Marker present but Guide down → same file fallback
• Guide live → augment analysis / architect / code / review with spdd_* tools

No slash command may block because Guide is absent. That is a hard design rule of the spike.

How our work relates to the ideas we ingested

SDLC-SPDD did not start as a RAG project. It started as a way to run Fowler’s SPDD workflow under Troy’s context limits, with slash commands and file indexes. The Guide spike is the next question: can optional hybrid retrieval make that same workflow better at remembering across Work IDs without abandoning auditability?

While dogfooding SPIKE-001 we appended those authors (and a smaller secondary set) into a local Guide corpus — so retrieval experiments hit methodology prose and our canvases, not only our own markdown. Below is the mapping we actually use.

Fowler SPDD → our lifecycle (files today)

Structured-Prompt-Driven Development says prompts are delivery artifacts: versioned, reviewed, improved. Our response: every Work ID gets a REASONS canvas under spdd/canvas/; /sdlc-spdd-analysis then /sdlc-spdd-plan then /sdlc-spdd-architect before /sdlc-spdd-code; code implements one approved operation; /sdlc-spdd-review and /sdlc-spdd-sync close the loop. That is the same contract, made runnable in Cursor / Copilot / Claude (see also engineered.at on SPDD).

I still care about the code and the wider Exploring Gen AI series argue that AI does not excuse sloppy ownership of design and tests. Our response: behavior changes are prompt-first; the canvas stays the source of truth; API-test and review phases exist so “the model wrote it” is never the definition of done.

Harness engineering, Sensors for coding agents, and Pushing AI autonomy describe feedback loops and limits around agents. Our response: workflow CLI + pointer + readiness gates + the prompt-optimization ledger (FEAT-004/005) are our harness and sensors. We do not grant more autonomy until the canvas says Ready For Coding.

The craft ladder we dogfood — make it work → make it right → make it fast — sits in the same lineage as Beck’s make it run / make it right and Fowler on evolutionary design (Is Design Dead?, Refactoring). Guide is a make-it-fast spike: only after the file workflow is right.

How Guide extends Fowler in our spike: SPDD already persists decisions in markdown. Guide dual-ingest projects those same artifacts so the next Work ID can retrieve prior Operations / Decisions / Safeguards by graph link — still Fowler’s “improve the prompt artifacts over time,” but searchable across sessions without pasting every canvas into the chat.

Chelsea Troy → why our indexes exist (and what Guide must not break)

What can we expect of LLMs as software engineers? argues models are aides to a rigorous process, not a substitute for judgment — and that large dumps fail (“lost in the middle,” unscoped pastes). Fowler gives workflow; Troy explains why that workflow must stay narrow. We documented the mapping in Chelsea Troy and the framework.

Troy’s point	What we already ship	What Guide must preserve
Don’t flood the context window	Tiered grounding; context-index / domain-index / session rotation	Retrieve a few linked lessons — never “all chunks similar to the prompt”
Work on cohesive slices	/sdlc-spdd-analysis → domain keywords → code areas	spdd_areaLessons keyed by area, not whole-repo RAG
Specific, testable problems	REASONS Requirements / Operations; one op per /sdlc-spdd-code	Surface pitfalls/decisions as Safeguards candidates — human still accepts
Judgment stays human	Architect readiness, review-against-canvas, confidence-stack testing	Optional backend; files fallback; no command fails if Guide is down
Don’t generate slop	Governed canvases, sync logs, prompt-first behavior change	Inclusions explained by typed edges, not opaque cosine alone

Related Troy pieces we ingested for the same reason: Avoiding technical debt (process debt is still debt — our canvases fight that), On code coverage tools (satisficing sentinels → our quality gates), debugging tactics (investigation is delivery work — our analysis phase).

How Guide extends Troy in our spike: file indexes already narrow context. Guide’s domain graph is how we pull cross-Work-ID lessons for the same area without violating Troy — an about edge to scripts/ is a scoped slice, not a history dump. If retrieval cannot explain the inclusion, it fails our Troy test even if the embedding score looks good.

Rod Johnson / Embabel — why the Guide shape is DICE, not “more RAG”

Context engineering needs domain understanding argues typed domain objects should drive context. Agent memory is not a greenfield problem argues you should ground agents in data you already keep. Our response: we do not invent a parallel memory store. We project the SPDD domain we already have — WorkId, Canvas, Area, Decision, Pitfall, Pattern — into Neo4j __Entity__ with typed edges, and keep markdown canonical. Chunk RAG (legs 1–2) is for discovery; the graph (leg 3) is for “why is this in the prompt?”

Jasper Blues — the Guide we actually integrate with

Our spike talks to a real Embabel Guide instance (fork + projection APIs), not a toy RAG stub. Jasper Blues’ From Docs to Dialog is the product story behind that service: Hub’s “talk to the docs” guide built with Embabel, graph-backed RAG on Neo4j via Drivine, and Toolish RAG — the model gets search tools (docs_textSearch, docs_vectorSearch, broaden/zoom) instead of a single black-box retrieve step. How that relates to our work: when CONTEXT_BACKEND=guide-dice, analysis/architect/code phases call those same tool-shaped retrieval surfaces (plus our spdd_* graph tools). We are not inventing a second RAG stack; we are hanging SPDD domain projection off the Guide Jasper describes.

The (Very Slowly) Ticking Time-Bomb in Your Graph Persistence Stack explains why Drivine’s use-case-specific Cypher/projections matter for graph persistence. Our response: leg 3 is a deliberate projection of SPDD markdown into __Entity__ nodes and typed edges — not hoping directory ingest alone fills a domain graph. That matches Jasper’s “write the graph shape you need” stance and is why our fork work adds projection load + spdd_workSubgraph / spdd_areaLessons rather than only chunk ingest.

The Voice, The Word, and The Wheel shows Guide as an evolving product surface (narration agents, command loops, Toolish RAG for speech). We are not shipping voice in SPIKE-001 — but it reinforces that Guide is a living context backend with MCP/tool loops, which is exactly the runtime we probe with resolve-context-backend.sh. Those three Jasper/Embabel pieces are also in Guide’s default supplementary ingest list alongside Rod’s posts — the same corpus family we extend with Fowler/Troy for the spike.

Thread: Fowler/Troy define how we work in files. Rod’s DICE framing defines why typed memory. Jasper’s Guide writing defines the system we plug into — Toolish RAG + Drivine/Neo4j — so our coming-soon path is “SPDD domain on Guide,” not a greenfield memory product.

Secondary ingest — sensors for the experiment, not the product story

We also appended Anthropic’s notes on context engineering / long-running harnesses, 12-Factor Agents, Willison on LLMs for code, and Hamel/Yan on evals. How that relates to our work: they score the spike (context cost, eval discipline, harness thinking). They do not redefine the operating model — Fowler + Troy still do. Go/no-go asks whether Guide hybrid beats file indexes on the same Troy criteria Fowler’s workflow already assumes.

Status: coming soon

This work lives on spike branches and open PRs — it is not the default on main yet:

• Orchestrator spike: PR #24 (SPIKE-001)
• Paired Guide projection / spdd_* tools: guide PR #2
• Flow overview: docs/guide-flow.md on the spike branch

Much of the operator path is already dogfooded (ingest, projection, runtime probe, A/B spot-checks). The remaining gate is a formal go / no-go before anything becomes a recommended install option for adopters. Until then: treat it as preview, keep shipping file-based SDLC-SPDD on main, and watch this space.

What you can do today

1. Use SDLC-SPDD with the file indexes — that path is production for the framework.
2. Read the spike docs / PRs if you want the design early.
3. Expect a follow-up post when go/no-go lands and the opt-in path is documented for adopters.

— John · github.com/jmjava/sdlc-spdd-orchestrator

Introducing tekton-dag — stack-aware Tekton CI for local and multi-team PoCs

Most local Kubernetes CI demos stop at “build one service.” Real stacks are graphs: shared libraries, polyglot apps, intercept routing for pull requests, and teams that need isolation without forking the platform.

tekton-dag is a standalone Tekton pipeline system for local development and proof-of-concept work. It models your apps as a DAG, runs stack-aware Bootstrap / PR / Merge pipelines, and can route PR traffic through Telepresence or mirrord while the rest of the stack stays on normal paths.

Three pipelines, one stack: bootstrap once, test on PRs, release on merge.

What it is

Pipeline	Purpose
Bootstrap	Deploy the full stack once — prerequisite for PR runs
PR (stack-pr-test)	Build the changed app with a snapshot tag, deploy intercepts, validate, test, comment on the PR — no version bump
Merge (stack-merge-release)	Promote RC → release, tag images, push the next dev-cycle version

Around those pipelines sits an in-cluster orchestration service (webhooks → stack resolution → PipelineRuns), Helm packaging for multi-team namespaces, baggage middleware across Spring / Node / Flask / PHP, and a testing ecosystem (Newman, Playwright, Artillery) with optional Neo4j-backed blast-radius selection.

PR traffic can follow intercept headers; everyone else keeps the default path.

# Kind + Tekton + stack tasks (local)
./scripts/kind-with-registry.sh
./scripts/install-tekton.sh
./scripts/publish-build-images.sh
kubectl apply -f tasks/ -f pipeline/

What we shipped recently

1. Dual intercept backends (M7)

PR flows can use Telepresence (default) or mirrord, selected with the pipeline param intercept-backend. Both paths are E2E-verified, so teams can pick the tool that fits their local debugging story without changing the DAG model.

2. Multi-team orchestration + Helm (M10)

An in-cluster Flask orchestrator receives GitHub webhooks, maps repos to stacks, and creates PipelineRuns. Helm charts plus ArgoCD ApplicationSet patterns cover team isolation, namespace scoping, and batched builds — so “one Kind cluster” can still look like several teams.

Management GUI: team switcher, DAG view, runs, triggers, and tests.

3. Management GUI (M11)

Vue 3 + Flask replaces the older reporting UI: multi-team / multi-cluster views, DAG visualization, runs and triggers, test status, and a Git browser. Covered by a solid pytest + Playwright suite so GUI changes stay reviewable.

4. Architecture customization (M12)

Shared Python packaging, Helm ConfigMap/PVC templates, parameterized pipelines (no hardcoded localhost), build-image variants (Java / Node / Python / PHP ranges), and custom pre/post hook tasks. Stack JSON schema + onboarding docs make “add a team” a configuration problem instead of a fork.

5. Narrated demos on GitHub Pages (M8 / docgen)

Eighteen Manim + TTS segments walk architecture, quick start, bootstrap dataflow, PR flow, intercept routing, local debug, merge/release, orchestrator API, Helm, baggage, testing, the test-trace graph, Results DB, customization, regression, the GUI, and what’s next. Watch them at jmjava.github.io/tekton-dag.

What’s next

Milestone 13 — production hardening is planned: retries on transient build/deploy failures, precise build-image sizing, multi-cluster promotion, timeouts and cleanup, Prometheus-oriented observability, secrets injection (ESO / Sealed Secrets), and per-app config per environment.

Try it

1. Clone jmjava/tekton-dag
2. Follow the README quick start (Kind + Tekton + publish build images)
3. Bootstrap a stack, then run a PR pipeline against a changed app
4. Skim the demo videos if you want the architecture before you touch YAML

If you need stack-aware Tekton locally — with real intercept behavior and a path toward multi-team Helm — start with bootstrap, then let a PR run prove the DAG.

— John · github.com/jmjava/tekton-dag

Introducing docgen — narrated demos from Markdown to Manim

Long-form demos should explain how a system works. Narrated diagram videos age well when the script and the visuals are first-class artifacts you can regenerate in CI — not one-off recordings that rot with every UI tweak.

docgen (documentation-generator) is a reusable Python library and CLI for that job: Markdown narration, OpenAI TTS, Whisper-aligned timing, Manim scenes, ffmpeg composition, and validation you can run before you ship. Install it, point it at a docgen.yaml, and build demos from the shell — no IDE plugin required.

Library, not app: pip-installable CLI + YAML + shell/CI.

What it is

docgen ships the video stack you need for scripted explainers:

• TTS narration — Markdown scripts → MP3 via OpenAI (gpt-4o-mini-tts)
• Whisper-style timestamps — word-level timing so visuals can wait on real speech
• Manim animations — the primary visual surface for diagram-heavy segments
• ffmpeg compose / concat — mux audio + video, stitch segments, freeze-tail guard
• validate — A/V drift, freeze ratio, narration lint, Manim layout hints, pre-push checks
• pages — static preview HTML for demo assets
• wizard — optional local web UI to bootstrap narration from project docs

North-star constraints matter as much as features: stable CLI contracts, hybrid config (deterministic merges plus optional OpenAI where it helps), and a hard rule that generated assets come from the tool — not hand-edited “fixes” that paper over generator gaps.

Typical path: narration → TTS → timestamps → Manim → compose → validate.

cd your-project/docs/demos
docgen yaml-generate          # merge hints/defaults into docgen.yaml
docgen narration-generate …   # optional LLM narration from hints
docgen scene-spec-generate …  # declarative Manim YAML
docgen generate-all           # TTS → timestamps → Manim → compose → validate
docgen validate --pre-push

What we shipped recently

1. Declarative Manim: scene-spec-generate + scene-compile

Instead of hand-editing generated Manim classes, maintainers steer with hints and declarative *.scene.yaml specs. OpenAI can emit the YAML; the engine compiles it into _TimedScene classes inside marked regions of scenes.py.

YAML in, timed Manim scenes out — with layout budgets and Whisper wait_word alignment.

The compiler is opinionated in useful ways: rows auto-paginate when they exceed the frame stack budget, oversized specs are rejected, and (when timing.json has Whisper words) each row’s first label can map to a wait_word index so boxes appear with the narration.

docgen scene-spec-generate --segment 01 --compile
docgen scene-compile animations/specs/01-overview.scene.yaml
docgen manim --scene YourGeneratedScene

2. Hints + yaml-generate as the maintainer surface

Demo bundles (typically docs/demos/) prefer hint files with YAML front matter over ad-hoc surgery on merged docgen.yaml. docgen yaml-generate merges segment lists, visual_map, and paths; narration-generate and scene-spec-generate read those hints. Generated narration, scenes, audio, and recordings stay tool-owned so Git review stays honest.

3. Handbook diagrams + Pages-friendly demos

The repo also ships a suite handbook under docs/suite/ (PlantUML sources with Graphviz/CI rendering) and Manim-oriented demo media for GitHub Pages — so architecture stories can ship as diagrams and narrated segments, not only as markdown.

Try it

1. Install from source or git: see jmjava/documentation-generator
2. docgen init a demos bundle (or adopt an existing docs/demos/)
3. Author hints → yaml-generate → narration / scene specs → generate-all
4. Run docgen validate --pre-push before you ship media

If you want demos that explain architecture with speech and diagrams — and you want a pipeline you can re-run instead of re-record — start with a Manim segment and let docgen own the rest.

— John · github.com/jmjava/documentation-generator