Files

medusa bbaf914259 Update tech_docs/template_kindom.md

2025-08-06 06:07:53 -05:00

17 KiB

Raw Permalink Blame History

The “One-Tool Mousetrap” is a concise manifesto for adopting CUE (Configure-Unify-Execute) as the single source of truth for configuration and policy.
It attacks the common multi-stage YAML stack:

YAML hand-edited by humans
Jinja2/Go-template rendering → JSON
Some external validator (Kubeval, OPA, custom scripts)
If validation fails, loop back and edit again (runtime surprise)

The anti-pattern is that you only discover type/syntax/semantic errors after rendering, when the cost of repair is highest.

CUE inverts the flow:

CUE schema + data (both expressed in CUE’s own language)
cue export (a single binary)
Immediately emits valid YAML/JSON/Dockerfile/INI, already linted against the schema.

Thus the same surface (CUE) is:

• The linter (static analysis)
• The schema (type & constraint def)
• The template (data transformation)

Zero runtime surprises because validation and generation happen in one atomic step.

In short: “CUE collapses the stack.”

Supremacy Verdict
(inside the Deterministic Serendipity Kingdom, where YAML is king and zero leakage is law)

──────────────────────────────

Scoring Table (0–5, 5 best)

Criterion	Jinja2	Go	Jsonnet	CUE	Dhall
Determinism lock	4	5	5	5	5
YAML round-trip fidelity	5	4	5	5	5
Zero side-effect surface	3	4	5	5	5
Macro / recursion depth	5	3	5	3	4
Tooling friction	5	5	3	3	2
Schema + validation	2	2	3	5	4

Total

Dhall: 25
Jsonnet: 24
CUE: 23
Go: 22
Jinja2: 21

──────────────────────────────
2. Crown Holder: Dhall

Why

Purity by design – no IO, no random, no exceptions.
Determinism by construction – same inputs → same output, period.
Native YAML emitter – canonical, sorted, no tags.
Functions as first-class citizens – richer than CUE, safer than Jsonnet.
Single static binary – cross-compiled dhall-to-yaml < 20 MB.

Trade-off: steeper learning curve, but once learned it cannot break the kingdom’s law.

──────────────────────────────
3. First Runner-Up: Jsonnet

If you need deep recursion or heavier arithmetic logic, Jsonnet edges Dhall.
Still pure, still deterministic, but larger surface area (stdlib) and heavier runtime.

──────────────────────────────
4. Final Decree

In the Deterministic Serendipity Kingdom Dhall reigns supreme.
Its crown is forged from mathematical purity; no steward can leak entropy, no subject can smuggle side effects.

──────────────────────────────────────────────
DETERMINISTIC SERENDIPITY – META TOOLKIT
(“YAML is still king; the crown can change heads.”)
──────────────────────────────────────────────

Guiding Question
“Which templating engines can replace Jinja2 without breaking the deterministic-serendipity contract?”

YAML-only input / output
Single deterministic seed
No embedded imperative code in the objects

Below is a matrix + migration cookbook for the four engines most often considered.

──────────────────────────────────────────────

Candidate Matrix (feature vs. Jinja2)

Engine	Deterministic Seed	YAML Load/Emit	Inline Expressions	Recursion	Notes
Jinja2	✅ (custom globals)	✅ (PyYAML)	`{{ }}`	✅	reference
Go text/template + sprig	✅ (rand.Seed)	✅ (gopkg.in/yaml.v3)	`{{ }}`	✅	binary embed
CUE templates	✅ (cue.Value)	✅ (native)	`{{ }}` & CUE constraints	✅	schema + template
Jsonnet	✅ (std.md5 + seed)	✅ (std.manifestYaml)	`+` / `	`/`std.*`	✅
Dhall	✅ (deterministic by design)	✅ (dhall-to-yaml)	λ-expressions	✅	no side effects

──────────────────────────────────────────────
2. Migration Rules (preserve determinism)

Rule 1 – Seed Isolation

Jinja2 / Go / sprig: call rand.Seed(n) or provide seeded rand.Rand instance.
Jsonnet: pass seed as top-level function argument.
Dhall: seed is an ordinary parameter (Dhall is pure).
CUE: seed is injected via let seed = <int> at template root.

Rule 2 – YAML Round-Trip
Every engine must:

Load raw YAML → AST (strict keys, no anchors).
Render expressions.
Emit canonical YAML (sorted keys, no tags).
Provide a validator script that compares SHA-256 before & after.

Rule 3 – Forbidden Features
Disable anything that leaks entropy or I/O:

Jinja2: turn off Undefined fallback, block import.
Go: prune sprig crypto & date funcs.
Jsonnet: wrap stdlib, remove std.native.
Dhall: already side-effect-free.
CUE: disallow command & http builtins.

──────────────────────────────────────────────
3. Drop-In Skeletons

3.1 Go template (rules/resolve.go.tmpl)

{{- $seed := .seed -}}
{{- $rand := rand.New(rand.NewSource(int64 $seed)) }}
{{- range $obj := .objects }}
{{ $obj | sprig.mustToPrettyJson | yamlFromJSON }}
{{- end }}

Build helper:

go run ./render.go -seed $KINGDOM_SEED

3.2 Jsonnet (resolve.jsonnet)

local seed = std.parseInt(std.extVar('seed'));
local rng  = std.native('seededRand')(seed);
[ x + {color: rng.choice(['silver','obsidian'])} for x in yaml.load('objects/*.yaml') ]

3.3 CUE (resolve.cue)

package kingdom

seed: int @tag(seed)
objects: [ for f in os.Glob("objects/*.yaml") { yaml.Unmarshal(os.ReadFile(f)) & {
	color: *rng.choice(["silver","obsidian"]) | _
}}]

3.4 Dhall (resolve.dhall)

let seed = 42
let rng  = λ(n → Natural/fold seed …)  -- deterministic pseudo RNG
in  ./objects/*.yaml
    : List { id : Text, color : Text }
    |> List/map (λ(o → o // { color = rng.choice ["silver","obsidian"] }))

──────────────────────────────────────────────
4. Validation Harness (language-agnostic)

#!/usr/bin/env bash
set -e
SEED=42
for engine in jinja2 go jsonnet cue dhall; do
  ./render-$engine --seed $SEED > out-$engine.yaml
  echo "$engine SHA256: $(sha256sum out-$engine.yaml | cut -d' ' -f1)"
done
diff -q out-*.yaml || echo "⚠️  non-deterministic drift"

──────────────────────────────────────────────
5. Decision Heuristics

Situation	Pick
Stay in Python, need macros & inheritance	Jinja2
Single static binary, embed templates	Go templates
Want schemas + data validation baked in	CUE
Pure functional, no side effects ever	Dhall
Deep recursion & arithmetic logic	Jsonnet

──────────────────────────────────────────────
6. One-Line Meta-Moral

The crown (YAML) never changes; only the steward (template engine) does—choose the steward whose leakage surface matches the risk budget of your deterministic kingdom.

The Deterministic Serendipity Guide

Using Jinja2 to Rule a Pure-YAML Kingdom

(Private reference v1.0 – for your eyes only)

0. Executive Purpose

Demonstrate how Jinja2 becomes the single source of truth in a closed system where

YAML is the only artifact (input, intermediate, output)
Objects are pure data (no code, no imports)
Determinism is absolute (same seed → same bits)
Serendipity is guaranteed (drop any new .yaml and re-render)

1. Kingdom Layout (single directory tree)

serendipity/
├─ kingdom.yml        # the royal charter
├─ objects/           # every object you will ever need
│  ├─ _manifest.yml   # optional index
│  ├─ unicorn.yaml
│  ├─ dragon.yaml
│  └─ …
└─ rules/
   └─ resolve.j2      # the only executable artifact

2. The Royal Charter (`kingdom.yml`)

kingdom:
  # determinism knobs
  seed: 42
  rng:
    engine: Random
    module: random
  magic:
    engine: Magic
    module: hashlib

  # path conventions
  objects: "objects/**/*.yaml"   # glob pattern
  rules:   "rules/resolve.j2"

  # post-processing
  exports:
    - format: yaml
      to:   stdout
    - format: yaml
      to:   out/kingdom_rendered.yaml

Nothing outside this file ever changes the deterministic output except the seed.

3. Object Contract (what your YAML may contain)

Every object MUST be valid YAML and MAY include Jinja2 expressions inside quoted scalars only.

# objects/unicorn.yaml
id: unicorn
color: "{{ rng.choice(['silver','iridescent']) }}"
blessing: "{{ magic.uuid4() }}"

Important:

Expressions are strings → safe for YAML parsers.
No top-level keys named jinja2 or __ (reserved for rulebook).

4. The Single Rulebook (`rules/resolve.j2`)

{#- 1. Bootstrap deterministic engines -#}
{% set rng   = load_rng(kingdom.rng) %}
{% set magic = load_magic(kingdom.magic) %}

{#- 2. Load and expand every object -#}
{% set court = [] %}
{% for path in glob(kingdom.objects) %}
  {% set raw  = read_file(path) %}
  {% set yaml = from_yaml(render_str(raw, rng=rng, magic=magic)) %}
  {{ court.append(yaml) or '' }}
{% endfor %}

{#- 3. Emit final deterministic YAML -#}
{{ court | to_yaml }}

Helper filters (load_rng, load_magic, glob, read_file, render_str) are registered once at startup; they never change and are not serialisable, so they cannot leak into objects.

5. Bootstrap Script (one-time, never edited)

#!/usr/bin/env python3
# bootstrap.py  (kept outside kingdom/)
import os, yaml, jinja2, glob, hashlib, random

# deterministic seed from kingdom.yml
meta = yaml.safe_load(open('kingdom.yml'))
random.seed(meta['kingdom']['seed'])

# Jinja2 env
env = jinja2.Environment(
    loader=jinja2.FileSystemLoader('rules'),
    finalize=lambda x: '' if x is None else x,
)
env.filters.update({
    'to_yaml': lambda d: yaml.dump(d, sort_keys=True),
})
env.globals.update({
    'load_rng':   lambda cfg: random,
    'load_magic': lambda cfg: type('Magic', (), {'uuid4': lambda: hashlib.md5(os.urandom(8)).hexdigest()[:8]})(),
    'glob':       lambda p: glob.glob(p, recursive=True),
    'read_file':  lambda f: open(f).read(),
    'render_str': lambda s, **kw: jinja2.Template(s, undefined=jinja2.StrictUndefined).render(**kw),
})

template = env.get_template('resolve.j2')
print(env.from_string(template.render(kingdom=meta['kingdom'])).render())

Run once → deterministic stream of bytes.
Commit the stream if you need reproducibility without Python.

6. Deep Powers Inside Jinja2 (exploited here)

Power	Usage in Kingdom
StrictUndefined	Every undefined var raises → no silent typos
Custom globals/filters	Inject rng, magic, filesystem helpers
Macro imports	`{% from 'macros.j2' import mutate %}` inside object templates
Whitespace control	`{%- … -%}` keeps emitted YAML clean
Recursive rendering	Object YAML contains Jinja2 that is re-parsed
Immutable context	No `{% set %}` leaks between object renders

7. Serendipity Workflow (non-breaking)

Drop objects/new_artifact.yaml
Re-run python bootstrap.py
Diff the two deterministic outputs → only deltas from the new object appear.
Version-control the diff, not the code.

8. Security & Reproducibility Checklist

No Python code inside YAML → safe for linters like yamllint.
Seed is single source of entropy → store it in a repo tag for audit.
Read-only filesystem → templates and bootstrap script are immutable at runtime.
One-way data flow: YAML → Jinja2 → expanded YAML → stdout/file.
No external network calls → reproducible in air-gapped CI.

9. Single-Command Invocation (CI / Makefile)

render:
	@python bootstrap.py > kingdom_rendered.yaml
	@echo "SHA256: $$(sha256sum kingdom_rendered.yaml)"

10. Closing Axiom

In this kingdom the template is the constitution,
the seed is the crown jewel,
and every .yaml is a citizen whose fate is deterministically serendipitous.

────────────────────────────────────────
THE ONE-TOOL MOUSETRAP
“CUE is the linter, the schema, and the template.”
────────────────────────────────────────

Thesis
Traditional pipelines

YAML → Jinja2 → JSON → Validator → (retry)

Anti-pattern: type errors are discovered after rendering.

CUE collapses the stack

CUE spec + data → `cue export` → Valid YAML/JSON/Dockerfile/INI

One binary, zero runtime surprises.

────────────────────────────────────────

Single Source of Truth (spec.cue)

package kingdom

// Universal schema: any object must satisfy this
#Object: {
    id:   string & =~"^[a-z0-9-]+$"
    kind: string
    meta?: {...}
}

// Domain specialisations
#Dockerfile: #Object & {
    kind: "Dockerfile"
    stages: [string]: {
        from: string
        run: [...string]
    }
}

#CiscoCfg: #Object & {
    kind: "Cisco"
    lines: [...string]
}

#FastAPISite: #Object & {
    kind: "FastAPI"
    routes: [...{path: string, method: "GET"|"POST"}]
}

#EmailTpl: #Object & {
    kind: "Email"
    subject: string
    body_html: string
}

// Definition of the entire universe
objects: [...#Object]

────────────────────────────────────────
2. Concrete Data (objects/anything.yaml)

objects:
  - id: web
    kind: Dockerfile
    stages:
      base:
        from: "python:3.11-slim"
        run: ["pip install fastapi uvicorn"]
  - id: api
    kind: FastAPI
    routes:
      - path: /health
        method: GET
  - id: alert
    kind: Email
    subject: "Build {{ .build_id }} succeeded"
    body_html: "<h1>🎉</h1>"

────────────────────────────────────────
3. One Command to Rule Them All

# Validate + render + lint in one shot
cue export objects/anything.yaml spec.cue --out yaml

Validation: fails before render if any field is wrong.
Rendering: substitutes {{ }} using native CUE interpolation.
Linting: built-in cue vet ensures every object is well-formed.

────────────────────────────────────────
4. Template Engine Inside the Schema (no Jinja2)

CUE supports string interpolation directly:

#EmailTpl: {
    subject: string
    body_html: """
    <html>
      <body>
        <h1>Build {{ .build_id }} succeeded</h1>
        <p>Timestamp: {{ .timestamp }}</p>
      </body>
    </html>
    """
}

Inject data:

cue export --with-context build_id=12345 --with-context timestamp=2024-06-13T12:34:56Z

────────────────────────────────────────
5. Multi-Format Output

# Dockerfile
cue export objects/anything.yaml spec.cue -e 'objects[0]' --out dockerfile > Dockerfile

# Cisco IOS
cue export objects/anything.yaml spec.cue -e 'objects[1]' --out text > router.cfg

# GitHub Actions
cue export objects/anything.yaml spec.cue -e 'objects[2]' --out yaml > .github/workflows/deploy.yml

────────────────────────────────────────
6. CI Guardrails (single 15-line job)

- name: One tool to rule them all
  uses: cue-lang/setup-cue@main
  with:
    version: 'v0.8.0'
  run: |
    cue vet objects/ spec.cue
    cue export objects/ spec.cue --out yaml > rendered.yml
    sha256sum rendered.yml > checksum.sha256

────────────────────────────────────────
7. The Mousetrap in One Sentence

With CUE, the schema is the linter, the template, and the runtime contract—every artifact (Dockerfile, Cisco config, FastAPI schema, email, GitHub Action) is guaranteed correct at render time, not discovered later.

17 KiB Raw Permalink Blame History Unescape Escape