the_information_nexus/tech_docs/template_kindom.md

Supremacy Verdict  
(inside the **Deterministic Serendipity Kingdom**, where YAML is king and *zero leakage* is law)

──────────────────────────────  
1. Scoring Table (0–5, 5 best)

| Criterion | Jinja2 | Go | Jsonnet | CUE | Dhall |
|---|---|---|---|---|---|
| **Determinism lock** | 4 | 5 | 5 | 5 | 5 |
| **YAML round-trip fidelity** | 5 | 4 | 5 | 5 | 5 |
| **Zero side-effect surface** | 3 | 4 | 5 | 5 | 5 |
| **Macro / recursion depth** | 5 | 3 | 5 | 3 | 4 |
| **Tooling friction** | 5 | 5 | 3 | 3 | 2 |
| **Schema + validation** | 2 | 2 | 3 | 5 | 4 |

Total  
- **Dhall**: 25  
- **Jsonnet**: 24  
- **CUE**: 23  
- **Go**: 22  
- **Jinja2**: 21  

──────────────────────────────  
2. Crown Holder: **Dhall**

Why  
- **Purity by design** – no IO, no random, no exceptions.  
- **Determinism by construction** – same inputs → same output, period.  
- **Native YAML emitter** – canonical, sorted, no tags.  
- **Functions as first-class citizens** – richer than CUE, safer than Jsonnet.  
- **Single static binary** – cross-compiled `dhall-to-yaml` < 20 MB.

Trade-off: steeper learning curve, but once learned **it cannot break the kingdom’s law**.

──────────────────────────────  
3. First Runner-Up: **Jsonnet**

If you need deep recursion or heavier arithmetic logic, Jsonnet edges Dhall.  
Still pure, still deterministic, but larger surface area (stdlib) and heavier runtime.

──────────────────────────────  
4. Final Decree

> In the Deterministic Serendipity Kingdom **Dhall reigns supreme**.  
> Its crown is forged from mathematical purity; no steward can leak entropy, no subject can smuggle side effects.

---

──────────────────────────────────────────────  
DETERMINISTIC SERENDIPITY – META TOOLKIT  
(“YAML is still king; the crown can change heads.”)  
──────────────────────────────────────────────  

0.  Guiding Question  
“Which templating engines can **replace** Jinja2 **without breaking** the deterministic-serendipity contract?”  
- YAML-only input / output  
- Single deterministic seed  
- No embedded imperative code in the objects  

Below is a **matrix + migration cookbook** for the four engines most often considered.

──────────────────────────────────────────────  
1.  Candidate Matrix (feature vs. Jinja2)

| Engine | Deterministic Seed | YAML Load/Emit | Inline Expressions | Recursion | Notes |
|---|---|---|---|---|---|
| **Jinja2** | ✅ (custom globals) | ✅ (PyYAML) | `{{ }}` | ✅ | reference |
| **Go text/template + sprig** | ✅ (rand.Seed) | ✅ (gopkg.in/yaml.v3) | `{{ }}` | ✅ | binary embed |
| **CUE templates** | ✅ (cue.Value) | ✅ (native) | `{{ }}` & CUE constraints | ✅ | schema + template |
| **Jsonnet** | ✅ (std.md5 + seed) | ✅ (std.manifestYaml) | `+` / `|` / `std.*` | ✅ | pure functional |
| **Dhall** | ✅ (deterministic by design) | ✅ (dhall-to-yaml) | λ-expressions | ✅ | no side effects |

──────────────────────────────────────────────  
2.  Migration Rules (preserve determinism)

Rule 1 – Seed Isolation  
- **Jinja2 / Go / sprig**: call `rand.Seed(n)` or provide seeded `rand.Rand` instance.  
- **Jsonnet**: pass seed as top-level function argument.  
- **Dhall**: seed is an ordinary parameter (Dhall is pure).  
- **CUE**: seed is injected via `let seed = <int>` at template root.

Rule 2 – YAML Round-Trip  
Every engine must:  
1. Load raw YAML → AST (strict keys, no anchors).  
2. Render expressions.  
3. Emit canonical YAML (sorted keys, no tags).  
Provide a **validator** script that compares SHA-256 before & after.

Rule 3 – Forbidden Features  
Disable anything that leaks entropy or I/O:  
- Jinja2: turn off `Undefined` fallback, block `import`.  
- Go: prune `sprig` crypto & date funcs.  
- Jsonnet: wrap stdlib, remove `std.native`.  
- Dhall: already side-effect-free.  
- CUE: disallow `command` & `http` builtins.

──────────────────────────────────────────────  
3.  Drop-In Skeletons

3.1  Go template (rules/resolve.go.tmpl)  
```go
{{- $seed := .seed -}}
{{- $rand := rand.New(rand.NewSource(int64 $seed)) }}
{{- range $obj := .objects }}
{{ $obj | sprig.mustToPrettyJson | yamlFromJSON }}
{{- end }}
```
Build helper:  
```bash
go run ./render.go -seed $KINGDOM_SEED
```

3.2  Jsonnet (resolve.jsonnet)  
```jsonnet
local seed = std.parseInt(std.extVar('seed'));
local rng  = std.native('seededRand')(seed);
[ x + {color: rng.choice(['silver','obsidian'])} for x in yaml.load('objects/*.yaml') ]
```

3.3  CUE (resolve.cue)  
```cue
package kingdom

seed: int @tag(seed)
objects: [ for f in os.Glob("objects/*.yaml") { yaml.Unmarshal(os.ReadFile(f)) & {
	color: *rng.choice(["silver","obsidian"]) | _
}}]
```

3.4  Dhall (resolve.dhall)  
```dhall
let seed = 42
let rng  = λ(n → Natural/fold seed …)  -- deterministic pseudo RNG
in  ./objects/*.yaml
    : List { id : Text, color : Text }
    |> List/map (λ(o → o // { color = rng.choice ["silver","obsidian"] }))
```

──────────────────────────────────────────────  
4.  Validation Harness (language-agnostic)

```bash
#!/usr/bin/env bash
set -e
SEED=42
for engine in jinja2 go jsonnet cue dhall; do
  ./render-$engine --seed $SEED > out-$engine.yaml
  echo "$engine SHA256: $(sha256sum out-$engine.yaml | cut -d' ' -f1)"
done
diff -q out-*.yaml || echo "⚠️  non-deterministic drift"
```

──────────────────────────────────────────────  
5.  Decision Heuristics

| Situation | Pick |
|---|---|
| Stay in Python, need macros & inheritance | Jinja2 |
| Single static binary, embed templates | Go templates |
| Want schemas + data validation baked in | CUE |
| Pure functional, no side effects ever | Dhall |
| Deep recursion & arithmetic logic | Jsonnet |

──────────────────────────────────────────────  
6.  One-Line Meta-Moral

The crown (YAML) never changes; only the steward (template engine) does—choose the steward whose **leakage surface** matches the **risk budget** of your deterministic kingdom.

---

# ***The Deterministic Serendipity Guide***  
## Using Jinja2 to Rule a Pure-YAML Kingdom  
*(Private reference v1.0 – for your eyes only)*

---

### 0. Executive Purpose
Demonstrate how **Jinja2 becomes the single source of truth** in a closed system where  
- **YAML is the only artifact** (input, intermediate, output)  
- **Objects are pure data** (no code, no imports)  
- **Determinism is absolute** (same seed → same bits)  
- **Serendipity is guaranteed** (drop any new `.yaml` and re-render)

---

### 1. Kingdom Layout (single directory tree)

```
serendipity/
├─ kingdom.yml        # the royal charter
├─ objects/           # every object you will ever need
│  ├─ _manifest.yml   # optional index
│  ├─ unicorn.yaml
│  ├─ dragon.yaml
│  └─ …
└─ rules/
   └─ resolve.j2      # the only executable artifact
```

---

### 2. The Royal Charter (`kingdom.yml`)

```yaml
kingdom:
  # determinism knobs
  seed: 42
  rng:
    engine: Random
    module: random
  magic:
    engine: Magic
    module: hashlib

  # path conventions
  objects: "objects/**/*.yaml"   # glob pattern
  rules:   "rules/resolve.j2"

  # post-processing
  exports:
    - format: yaml
      to:   stdout
    - format: yaml
      to:   out/kingdom_rendered.yaml
```

Nothing outside this file ever changes the deterministic output except the **seed**.

---

### 3. Object Contract (what your YAML may contain)

Every object **MUST** be valid YAML and **MAY** include Jinja2 expressions inside **quoted** scalars only.

```yaml
# objects/unicorn.yaml
id: unicorn
color: "{{ rng.choice(['silver','iridescent']) }}"
blessing: "{{ magic.uuid4() }}"
```

Important:  
- Expressions are **strings** → safe for YAML parsers.  
- No top-level keys named `jinja2` or `__` (reserved for rulebook).

---

### 4. The Single Rulebook (`rules/resolve.j2`)

```jinja2
{#- 1. Bootstrap deterministic engines -#}
{% set rng   = load_rng(kingdom.rng) %}
{% set magic = load_magic(kingdom.magic) %}

{#- 2. Load and expand every object -#}
{% set court = [] %}
{% for path in glob(kingdom.objects) %}
  {% set raw  = read_file(path) %}
  {% set yaml = from_yaml(render_str(raw, rng=rng, magic=magic)) %}
  {{ court.append(yaml) or '' }}
{% endfor %}

{#- 3. Emit final deterministic YAML -#}
{{ court | to_yaml }}
```

Helper filters (`load_rng`, `load_magic`, `glob`, `read_file`, `render_str`) are registered once at startup; **they never change** and are **not serialisable**, so they cannot leak into objects.

---

### 5. Bootstrap Script (one-time, never edited)

```python
#!/usr/bin/env python3
# bootstrap.py  (kept outside kingdom/)
import os, yaml, jinja2, glob, hashlib, random

# deterministic seed from kingdom.yml
meta = yaml.safe_load(open('kingdom.yml'))
random.seed(meta['kingdom']['seed'])

# Jinja2 env
env = jinja2.Environment(
    loader=jinja2.FileSystemLoader('rules'),
    finalize=lambda x: '' if x is None else x,
)
env.filters.update({
    'to_yaml': lambda d: yaml.dump(d, sort_keys=True),
})
env.globals.update({
    'load_rng':   lambda cfg: random,
    'load_magic': lambda cfg: type('Magic', (), {'uuid4': lambda: hashlib.md5(os.urandom(8)).hexdigest()[:8]})(),
    'glob':       lambda p: glob.glob(p, recursive=True),
    'read_file':  lambda f: open(f).read(),
    'render_str': lambda s, **kw: jinja2.Template(s, undefined=jinja2.StrictUndefined).render(**kw),
})

template = env.get_template('resolve.j2')
print(env.from_string(template.render(kingdom=meta['kingdom'])).render())
```

Run once → deterministic stream of bytes.  
Commit the stream if you need reproducibility without Python.

---

### 6. Deep Powers Inside Jinja2 (exploited here)

| Power | Usage in Kingdom |
|---|---|
| **StrictUndefined** | Every undefined var raises → no silent typos |
| **Custom globals/filters** | Inject rng, magic, filesystem helpers |
| **Macro imports** | `{% from 'macros.j2' import mutate %}` inside object templates |
| **Whitespace control** | `{%- … -%}` keeps emitted YAML clean |
| **Recursive rendering** | Object YAML contains Jinja2 that is re-parsed |
| **Immutable context** | No `{% set %}` leaks between object renders |

---

### 7. Serendipity Workflow (non-breaking)

1. Drop `objects/new_artifact.yaml`  
2. Re-run `python bootstrap.py`  
3. Diff the two deterministic outputs → only deltas from the new object appear.  
4. Version-control the diff, not the code.

---

### 8. Security & Reproducibility Checklist

- **No Python code inside YAML** → safe for linters like yamllint.  
- **Seed is single source of entropy** → store it in a repo tag for audit.  
- **Read-only filesystem** → templates and bootstrap script are immutable at runtime.  
- **One-way data flow**: YAML → Jinja2 → expanded YAML → stdout/file.  
- **No external network calls** → reproducible in air-gapped CI.

---

### 9. Single-Command Invocation (CI / Makefile)

```makefile
render:
	@python bootstrap.py > kingdom_rendered.yaml
	@echo "SHA256: $$(sha256sum kingdom_rendered.yaml)"
```

---

### 10. Closing Axiom

> In this kingdom the **template is the constitution**,  
> the seed is the crown jewel,  
> and every `.yaml` is a citizen whose fate is **deterministically serendipitous**.

---

────────────────────────────────────────  
THE ONE-TOOL MOUSETRAP  
“CUE is the linter, the schema, *and* the template.”  
────────────────────────────────────────  

0.  Thesis  
Traditional pipelines  
```
YAML → Jinja2 → JSON → Validator → (retry)
```
Anti-pattern: **type errors are discovered *after* rendering**.  

CUE collapses the stack  
```
CUE spec + data → `cue export` → Valid YAML/JSON/Dockerfile/INI
```
One binary, zero runtime surprises.

────────────────────────────────────────  
1.  Single Source of Truth (`spec.cue`)

```cue
package kingdom

// Universal schema: any object must satisfy this
#Object: {
    id:   string & =~"^[a-z0-9-]+$"
    kind: string
    meta?: {...}
}

// Domain specialisations
#Dockerfile: #Object & {
    kind: "Dockerfile"
    stages: [string]: {
        from: string
        run: [...string]
    }
}

#CiscoCfg: #Object & {
    kind: "Cisco"
    lines: [...string]
}

#FastAPISite: #Object & {
    kind: "FastAPI"
    routes: [...{path: string, method: "GET"|"POST"}]
}

#EmailTpl: #Object & {
    kind: "Email"
    subject: string
    body_html: string
}

// Definition of the entire universe
objects: [...#Object]
```

────────────────────────────────────────  
2.  Concrete Data (`objects/anything.yaml`)

```yaml
objects:
  - id: web
    kind: Dockerfile
    stages:
      base:
        from: "python:3.11-slim"
        run: ["pip install fastapi uvicorn"]
  - id: api
    kind: FastAPI
    routes:
      - path: /health
        method: GET
  - id: alert
    kind: Email
    subject: "Build {{ .build_id }} succeeded"
    body_html: "<h1>🎉</h1>"
```

────────────────────────────────────────  
3.  One Command to Rule Them All

```bash
# Validate + render + lint in one shot
cue export objects/anything.yaml spec.cue --out yaml
```

- **Validation**: fails before render if any field is wrong.  
- **Rendering**: substitutes `{{ }}` using native CUE interpolation.  
- **Linting**: built-in `cue vet` ensures every object is well-formed.

────────────────────────────────────────  
4.  Template Engine Inside the Schema (no Jinja2)

CUE supports string interpolation directly:

```cue
#EmailTpl: {
    subject: string
    body_html: """
    <html>
      <body>
        <h1>Build {{ .build_id }} succeeded</h1>
        <p>Timestamp: {{ .timestamp }}</p>
      </body>
    </html>
    """
}
```

Inject data:

```bash
cue export --with-context build_id=12345 --with-context timestamp=2024-06-13T12:34:56Z
```

────────────────────────────────────────  
5.  Multi-Format Output

```bash
# Dockerfile
cue export objects/anything.yaml spec.cue -e 'objects[0]' --out dockerfile > Dockerfile

# Cisco IOS
cue export objects/anything.yaml spec.cue -e 'objects[1]' --out text > router.cfg

# GitHub Actions
cue export objects/anything.yaml spec.cue -e 'objects[2]' --out yaml > .github/workflows/deploy.yml
```

────────────────────────────────────────  
6.  CI Guardrails (single 15-line job)

```yaml
- name: One tool to rule them all
  uses: cue-lang/setup-cue@main
  with:
    version: 'v0.8.0'
  run: |
    cue vet objects/ spec.cue
    cue export objects/ spec.cue --out yaml > rendered.yml
    sha256sum rendered.yml > checksum.sha256
```

────────────────────────────────────────  
7.  The Mousetrap in One Sentence

> With **CUE**, the schema *is* the linter, the template, and the runtime contract—every artifact (Dockerfile, Cisco config, FastAPI schema, email, GitHub Action) is **guaranteed correct at render time**, not discovered later.