Supremacy Verdict (inside the **Deterministic Serendipity Kingdom**, where YAML is king and *zero leakage* is law) ────────────────────────────── 1. Scoring Table (0–5, 5 best) | Criterion | Jinja2 | Go | Jsonnet | CUE | Dhall | |---|---|---|---|---|---| | **Determinism lock** | 4 | 5 | 5 | 5 | 5 | | **YAML round-trip fidelity** | 5 | 4 | 5 | 5 | 5 | | **Zero side-effect surface** | 3 | 4 | 5 | 5 | 5 | | **Macro / recursion depth** | 5 | 3 | 5 | 3 | 4 | | **Tooling friction** | 5 | 5 | 3 | 3 | 2 | | **Schema + validation** | 2 | 2 | 3 | 5 | 4 | Total - **Dhall**: 25 - **Jsonnet**: 24 - **CUE**: 23 - **Go**: 22 - **Jinja2**: 21 ────────────────────────────── 2. Crown Holder: **Dhall** Why - **Purity by design** – no IO, no random, no exceptions. - **Determinism by construction** – same inputs → same output, period. - **Native YAML emitter** – canonical, sorted, no tags. - **Functions as first-class citizens** – richer than CUE, safer than Jsonnet. - **Single static binary** – cross-compiled `dhall-to-yaml` < 20 MB. Trade-off: steeper learning curve, but once learned **it cannot break the kingdom’s law**. ────────────────────────────── 3. First Runner-Up: **Jsonnet** If you need deep recursion or heavier arithmetic logic, Jsonnet edges Dhall. Still pure, still deterministic, but larger surface area (stdlib) and heavier runtime. ────────────────────────────── 4. Final Decree > In the Deterministic Serendipity Kingdom **Dhall reigns supreme**. > Its crown is forged from mathematical purity; no steward can leak entropy, no subject can smuggle side effects. --- ────────────────────────────────────────────── DETERMINISTIC SERENDIPITY – META TOOLKIT (“YAML is still king; the crown can change heads.”) ────────────────────────────────────────────── 0. Guiding Question “Which templating engines can **replace** Jinja2 **without breaking** the deterministic-serendipity contract?” - YAML-only input / output - Single deterministic seed - No embedded imperative code in the objects Below is a **matrix + migration cookbook** for the four engines most often considered. ────────────────────────────────────────────── 1. Candidate Matrix (feature vs. Jinja2) | Engine | Deterministic Seed | YAML Load/Emit | Inline Expressions | Recursion | Notes | |---|---|---|---|---|---| | **Jinja2** | ✅ (custom globals) | ✅ (PyYAML) | `{{ }}` | ✅ | reference | | **Go text/template + sprig** | ✅ (rand.Seed) | ✅ (gopkg.in/yaml.v3) | `{{ }}` | ✅ | binary embed | | **CUE templates** | ✅ (cue.Value) | ✅ (native) | `{{ }}` & CUE constraints | ✅ | schema + template | | **Jsonnet** | ✅ (std.md5 + seed) | ✅ (std.manifestYaml) | `+` / `|` / `std.*` | ✅ | pure functional | | **Dhall** | ✅ (deterministic by design) | ✅ (dhall-to-yaml) | λ-expressions | ✅ | no side effects | ────────────────────────────────────────────── 2. Migration Rules (preserve determinism) Rule 1 – Seed Isolation - **Jinja2 / Go / sprig**: call `rand.Seed(n)` or provide seeded `rand.Rand` instance. - **Jsonnet**: pass seed as top-level function argument. - **Dhall**: seed is an ordinary parameter (Dhall is pure). - **CUE**: seed is injected via `let seed = ` at template root. Rule 2 – YAML Round-Trip Every engine must: 1. Load raw YAML → AST (strict keys, no anchors). 2. Render expressions. 3. Emit canonical YAML (sorted keys, no tags). Provide a **validator** script that compares SHA-256 before & after. Rule 3 – Forbidden Features Disable anything that leaks entropy or I/O: - Jinja2: turn off `Undefined` fallback, block `import`. - Go: prune `sprig` crypto & date funcs. - Jsonnet: wrap stdlib, remove `std.native`. - Dhall: already side-effect-free. - CUE: disallow `command` & `http` builtins. ────────────────────────────────────────────── 3. Drop-In Skeletons 3.1 Go template (rules/resolve.go.tmpl) ```go {{- $seed := .seed -}} {{- $rand := rand.New(rand.NewSource(int64 $seed)) }} {{- range $obj := .objects }} {{ $obj | sprig.mustToPrettyJson | yamlFromJSON }} {{- end }} ``` Build helper: ```bash go run ./render.go -seed $KINGDOM_SEED ``` 3.2 Jsonnet (resolve.jsonnet) ```jsonnet local seed = std.parseInt(std.extVar('seed')); local rng = std.native('seededRand')(seed); [ x + {color: rng.choice(['silver','obsidian'])} for x in yaml.load('objects/*.yaml') ] ``` 3.3 CUE (resolve.cue) ```cue package kingdom seed: int @tag(seed) objects: [ for f in os.Glob("objects/*.yaml") { yaml.Unmarshal(os.ReadFile(f)) & { color: *rng.choice(["silver","obsidian"]) | _ }}] ``` 3.4 Dhall (resolve.dhall) ```dhall let seed = 42 let rng = λ(n → Natural/fold seed …) -- deterministic pseudo RNG in ./objects/*.yaml : List { id : Text, color : Text } |> List/map (λ(o → o // { color = rng.choice ["silver","obsidian"] })) ``` ────────────────────────────────────────────── 4. Validation Harness (language-agnostic) ```bash #!/usr/bin/env bash set -e SEED=42 for engine in jinja2 go jsonnet cue dhall; do ./render-$engine --seed $SEED > out-$engine.yaml echo "$engine SHA256: $(sha256sum out-$engine.yaml | cut -d' ' -f1)" done diff -q out-*.yaml || echo "⚠️ non-deterministic drift" ``` ────────────────────────────────────────────── 5. Decision Heuristics | Situation | Pick | |---|---| | Stay in Python, need macros & inheritance | Jinja2 | | Single static binary, embed templates | Go templates | | Want schemas + data validation baked in | CUE | | Pure functional, no side effects ever | Dhall | | Deep recursion & arithmetic logic | Jsonnet | ────────────────────────────────────────────── 6. One-Line Meta-Moral The crown (YAML) never changes; only the steward (template engine) does—choose the steward whose **leakage surface** matches the **risk budget** of your deterministic kingdom. --- # ***The Deterministic Serendipity Guide*** ## Using Jinja2 to Rule a Pure-YAML Kingdom *(Private reference v1.0 – for your eyes only)* --- ### 0. Executive Purpose Demonstrate how **Jinja2 becomes the single source of truth** in a closed system where - **YAML is the only artifact** (input, intermediate, output) - **Objects are pure data** (no code, no imports) - **Determinism is absolute** (same seed → same bits) - **Serendipity is guaranteed** (drop any new `.yaml` and re-render) --- ### 1. Kingdom Layout (single directory tree) ``` serendipity/ ├─ kingdom.yml # the royal charter ├─ objects/ # every object you will ever need │ ├─ _manifest.yml # optional index │ ├─ unicorn.yaml │ ├─ dragon.yaml │ └─ … └─ rules/ └─ resolve.j2 # the only executable artifact ``` --- ### 2. The Royal Charter (`kingdom.yml`) ```yaml kingdom: # determinism knobs seed: 42 rng: engine: Random module: random magic: engine: Magic module: hashlib # path conventions objects: "objects/**/*.yaml" # glob pattern rules: "rules/resolve.j2" # post-processing exports: - format: yaml to: stdout - format: yaml to: out/kingdom_rendered.yaml ``` Nothing outside this file ever changes the deterministic output except the **seed**. --- ### 3. Object Contract (what your YAML may contain) Every object **MUST** be valid YAML and **MAY** include Jinja2 expressions inside **quoted** scalars only. ```yaml # objects/unicorn.yaml id: unicorn color: "{{ rng.choice(['silver','iridescent']) }}" blessing: "{{ magic.uuid4() }}" ``` Important: - Expressions are **strings** → safe for YAML parsers. - No top-level keys named `jinja2` or `__` (reserved for rulebook). --- ### 4. The Single Rulebook (`rules/resolve.j2`) ```jinja2 {#- 1. Bootstrap deterministic engines -#} {% set rng = load_rng(kingdom.rng) %} {% set magic = load_magic(kingdom.magic) %} {#- 2. Load and expand every object -#} {% set court = [] %} {% for path in glob(kingdom.objects) %} {% set raw = read_file(path) %} {% set yaml = from_yaml(render_str(raw, rng=rng, magic=magic)) %} {{ court.append(yaml) or '' }} {% endfor %} {#- 3. Emit final deterministic YAML -#} {{ court | to_yaml }} ``` Helper filters (`load_rng`, `load_magic`, `glob`, `read_file`, `render_str`) are registered once at startup; **they never change** and are **not serialisable**, so they cannot leak into objects. --- ### 5. Bootstrap Script (one-time, never edited) ```python #!/usr/bin/env python3 # bootstrap.py (kept outside kingdom/) import os, yaml, jinja2, glob, hashlib, random # deterministic seed from kingdom.yml meta = yaml.safe_load(open('kingdom.yml')) random.seed(meta['kingdom']['seed']) # Jinja2 env env = jinja2.Environment( loader=jinja2.FileSystemLoader('rules'), finalize=lambda x: '' if x is None else x, ) env.filters.update({ 'to_yaml': lambda d: yaml.dump(d, sort_keys=True), }) env.globals.update({ 'load_rng': lambda cfg: random, 'load_magic': lambda cfg: type('Magic', (), {'uuid4': lambda: hashlib.md5(os.urandom(8)).hexdigest()[:8]})(), 'glob': lambda p: glob.glob(p, recursive=True), 'read_file': lambda f: open(f).read(), 'render_str': lambda s, **kw: jinja2.Template(s, undefined=jinja2.StrictUndefined).render(**kw), }) template = env.get_template('resolve.j2') print(env.from_string(template.render(kingdom=meta['kingdom'])).render()) ``` Run once → deterministic stream of bytes. Commit the stream if you need reproducibility without Python. --- ### 6. Deep Powers Inside Jinja2 (exploited here) | Power | Usage in Kingdom | |---|---| | **StrictUndefined** | Every undefined var raises → no silent typos | | **Custom globals/filters** | Inject rng, magic, filesystem helpers | | **Macro imports** | `{% from 'macros.j2' import mutate %}` inside object templates | | **Whitespace control** | `{%- … -%}` keeps emitted YAML clean | | **Recursive rendering** | Object YAML contains Jinja2 that is re-parsed | | **Immutable context** | No `{% set %}` leaks between object renders | --- ### 7. Serendipity Workflow (non-breaking) 1. Drop `objects/new_artifact.yaml` 2. Re-run `python bootstrap.py` 3. Diff the two deterministic outputs → only deltas from the new object appear. 4. Version-control the diff, not the code. --- ### 8. Security & Reproducibility Checklist - **No Python code inside YAML** → safe for linters like yamllint. - **Seed is single source of entropy** → store it in a repo tag for audit. - **Read-only filesystem** → templates and bootstrap script are immutable at runtime. - **One-way data flow**: YAML → Jinja2 → expanded YAML → stdout/file. - **No external network calls** → reproducible in air-gapped CI. --- ### 9. Single-Command Invocation (CI / Makefile) ```makefile render: @python bootstrap.py > kingdom_rendered.yaml @echo "SHA256: $$(sha256sum kingdom_rendered.yaml)" ``` --- ### 10. Closing Axiom > In this kingdom the **template is the constitution**, > the seed is the crown jewel, > and every `.yaml` is a citizen whose fate is **deterministically serendipitous**.