The DIF test engine — prove the three layers without collapsing them

A plan that cannot fail in an interesting way is just more markdown. The last post said DIF, the orchestrator, and Embabel answer different questions. This post is the test engine that keeps them from collapsing into one runtime.

Follow-up to Three layers, one day. Source: the working test flow in jmjava/embabel-dif and the integration ladder in docs/ORCH_INTEGRATION_ROADMAP.md.

What “test engine” means here

Not a new product. Not a second daily driver. A stacked set of checks where each rung is allowed to fail before we spend complexity on the next:

./mvnw test                 # unit + CLI + FoldContractTest
                            # EmbabelLivePlatformTest skipped unless DIF_LIVE_EMBABEL=1
./scripts/dif-orch-smoke.sh # FEAT-001 ready + T03; FEAT-099 exit 1
./scripts/dif-orch-day.sh   # fold twice / architect / review / plan --projection
                            # skip when CLI or snapshots missing
./scripts/dif-live-e2e.sh   # orch Guide+Neo4j + JSONL quote + live GOAP

Default CI is the first two boxes. Live Guide and Embabel are opt-in. They reuse the orchestrator’s existing tests/test-guide-stack-live.sh. They do not put Embabel or Guide inside sdlc.sh next.

The engine’s job. Prove the three systems can talk. Prove a missing DIF checkout is skip, not a broken day. Prove a contradictory canvas cannot earn Ready For Coding. Never start a JVM to run next.

Rung A — five named checks, not prose

FoldContractTest is step 1 of the fold iteration plan. The five success criteria are tests:

Check	What fails if we are wrong
Same accepted canvas → same model	Nothing downstream is trustworthy
Review fails without “looks correct”	login-auth-broken still prints RESULT: PASS
Syntax variance does not flip invariants	A DTO rename (FEAT-070) changes what must stay true
Open T## is a MissingObligation	T03 on FEAT-001 disappears into a checklist
Requirement vs non-goal blocks Ready For Coding	FEAT-099 pagination clash still looks green

Harvested canvases under examples/canvases/ are the corpus. The fold learns from real REASONS files, not imagined IR. Adding a backend must not change the CLI or the canvas schema.

Rungs C–D — a script can trust the gate

dif-fold.sh writes a projection and a stable .gate.json. Smoke does not parse stdout for meaning. It reads JSON and exit codes:

{
  "workId": "FEAT-001-order-status-api",
  "readyForImplementation": true,
  "blockingConflicts": [],
  "missingObligations": ["T03"]
}

dif-orch-smoke.sh folds FEAT-001 (exit 0, ready, T03 missing) and FEAT-099 (exit 1, blocking pagination clash). If a sibling orch checkout is present, it folds the live examples/spring-boot-order-api canvas too. Then it hits the silent attach:

DIF_DISABLED=1 check-canvas.sh …   →  dif=skipped   (exit 0)
check-canvas.sh FEAT-001           →  dif=ready     (exit 0)
check-canvas.sh FEAT-099           →  dif=blocked   (exit 1)

One line. Agents do not get a fold dump. Missing DIF is skip, not a new ritual. That is the same opt-in shape as Guide.

Rung G — a scripted day, no Embabel

dif-orch-day.sh is the cheap “full day.” It does not start Embabel, Guide MCP, or replace sdlc.sh next.

1. Fold the same FEAT-001 canvas twice. The two .gate.json files must cmp equal.
2. Architect FEAT-001 → dif=ready, T03 still a missing obligation.
3. Architect FEAT-099 → exit 1, dif=blocked.
4. Review the orch order-status snapshots: dropping auth fails. A DTO rename still passes.
5. plan --projection builds a VerificationPlan from the folded model. No markdown re-parse.
6. Guide JSONL is an optional quote (Decision / Pitfall), not a gate.
7. Missing CLI or missing snapshots → dif=skipped. Present snapshots that drop a safeguard → dif=blocked.

Review uses examples/snapshots/order-status-*.json and the canvas safeguard paths — not the old login fixtures. Syntax-ok vs auth-broken is the whole point: a rename is legal; a dropped safeguard is not.

Rungs H–J — live, still not inside next

dif-live-e2e.sh is the three-way path that already passed here. First it asserts sdlc-engine is not a JVM — help must not mention Spring or Embabel. Then it reuses the orch Guide+Neo4j harness, runs the scripted day, quotes DIF JSONL through GuideClient under a unique Work ID (FEAT-DIF-LIVE-… so it does not collide with orch’s already-projected FEAT-001), and boots the Embabel Spring platform.

Live Embabel (EmbabelLivePlatformTest, DIF_LIVE_EMBABEL=1) runs the fixture GOAP path:

UserInput
  → captureRequest → interpretIntent → foldIntent
  → analyzeRepository → planVerification
  → VerificationPlan (readyForImplementation, missing rotation IT)

The refresh-token wording uses FixtureIntentInterpreter — no LLM. A second test plans an already-folded orch canvas without re-parsing markdown. Conflicts stay on the VerificationPlan (readyForImplementation=false). They are not a GOAP precondition Embabel 1.5 cannot treat as an action post.

Orch CI does not need Maven. It uses tests/fixtures/dif-fold-stub.sh so detect-and-skip / fail-closed can be proven with a fake CLI: skipped, ready, or blocked.

What would fail the engine

The ladder is the falsification list from the last post, turned into commands:

• Two folds of the same canvas disagree → day step 1 fails.
• A requirement vs non-goal pair still looks ready → smoke / architect on FEAT-099 fails.
• An open T03 does not show up → gate assertion fails.
• A DTO rename flips an invariant → SyntaxVarianceTest / review syntax-ok fails.
• Review cannot fail a dropped safeguard without “looks correct” → auth-broken still passes.
• sdlc-engine --help mentions Embabel → live E2E step 0 fails.
• Missing DIF breaks the orch day → skip tests fail.

If people stop reading the canvas because they treat the JSON as source of truth, we failed even if every script is green. The projection stays regenerable and disposable.

Source: github.com/jmjava/embabel-dif
Previous: Three layers, one day — DIF, the orchestrator, and Embabel
Related: sdlc-spdd-orchestrator · Embabel