From 39900c70c69df2dcf7ab5d0c91c1fdf73c4f0d1d Mon Sep 17 00:00:00 2001 From: medusa Date: Wed, 6 Aug 2025 05:56:36 -0500 Subject: [PATCH] Add tech_docs/template_kindom.txt --- tech_docs/template_kindom.txt | 353 ++++++++++++++++++++++++++++++++++ 1 file changed, 353 insertions(+) create mode 100644 tech_docs/template_kindom.txt diff --git a/tech_docs/template_kindom.txt b/tech_docs/template_kindom.txt new file mode 100644 index 0000000..c7427b0 --- /dev/null +++ b/tech_docs/template_kindom.txt @@ -0,0 +1,353 @@ +Supremacy Verdict +(inside the **Deterministic Serendipity Kingdom**, where YAML is king and *zero leakage* is law) + +────────────────────────────── +1. Scoring Table (0–5, 5 best) + +| Criterion | Jinja2 | Go | Jsonnet | CUE | Dhall | +|---|---|---|---|---|---| +| **Determinism lock** | 4 | 5 | 5 | 5 | 5 | +| **YAML round-trip fidelity** | 5 | 4 | 5 | 5 | 5 | +| **Zero side-effect surface** | 3 | 4 | 5 | 5 | 5 | +| **Macro / recursion depth** | 5 | 3 | 5 | 3 | 4 | +| **Tooling friction** | 5 | 5 | 3 | 3 | 2 | +| **Schema + validation** | 2 | 2 | 3 | 5 | 4 | + +Total +- **Dhall**: 25 +- **Jsonnet**: 24 +- **CUE**: 23 +- **Go**: 22 +- **Jinja2**: 21 + +────────────────────────────── +2. Crown Holder: **Dhall** + +Why +- **Purity by design** – no IO, no random, no exceptions. +- **Determinism by construction** – same inputs → same output, period. +- **Native YAML emitter** – canonical, sorted, no tags. +- **Functions as first-class citizens** – richer than CUE, safer than Jsonnet. +- **Single static binary** – cross-compiled `dhall-to-yaml` < 20 MB. + +Trade-off: steeper learning curve, but once learned **it cannot break the kingdom’s law**. + +────────────────────────────── +3. First Runner-Up: **Jsonnet** + +If you need deep recursion or heavier arithmetic logic, Jsonnet edges Dhall. +Still pure, still deterministic, but larger surface area (stdlib) and heavier runtime. + +────────────────────────────── +4. Final Decree + +> In the Deterministic Serendipity Kingdom **Dhall reigns supreme**. +> Its crown is forged from mathematical purity; no steward can leak entropy, no subject can smuggle side effects. + +--- + +────────────────────────────────────────────── +DETERMINISTIC SERENDIPITY – META TOOLKIT +(“YAML is still king; the crown can change heads.”) +────────────────────────────────────────────── + +0. Guiding Question +“Which templating engines can **replace** Jinja2 **without breaking** the deterministic-serendipity contract?” +- YAML-only input / output +- Single deterministic seed +- No embedded imperative code in the objects + +Below is a **matrix + migration cookbook** for the four engines most often considered. + +────────────────────────────────────────────── +1. Candidate Matrix (feature vs. Jinja2) + +| Engine | Deterministic Seed | YAML Load/Emit | Inline Expressions | Recursion | Notes | +|---|---|---|---|---|---| +| **Jinja2** | ✅ (custom globals) | ✅ (PyYAML) | `{{ }}` | ✅ | reference | +| **Go text/template + sprig** | ✅ (rand.Seed) | ✅ (gopkg.in/yaml.v3) | `{{ }}` | ✅ | binary embed | +| **CUE templates** | ✅ (cue.Value) | ✅ (native) | `{{ }}` & CUE constraints | ✅ | schema + template | +| **Jsonnet** | ✅ (std.md5 + seed) | ✅ (std.manifestYaml) | `+` / `|` / `std.*` | ✅ | pure functional | +| **Dhall** | ✅ (deterministic by design) | ✅ (dhall-to-yaml) | λ-expressions | ✅ | no side effects | + +────────────────────────────────────────────── +2. Migration Rules (preserve determinism) + +Rule 1 – Seed Isolation +- **Jinja2 / Go / sprig**: call `rand.Seed(n)` or provide seeded `rand.Rand` instance. +- **Jsonnet**: pass seed as top-level function argument. +- **Dhall**: seed is an ordinary parameter (Dhall is pure). +- **CUE**: seed is injected via `let seed = ` at template root. + +Rule 2 – YAML Round-Trip +Every engine must: +1. Load raw YAML → AST (strict keys, no anchors). +2. Render expressions. +3. Emit canonical YAML (sorted keys, no tags). +Provide a **validator** script that compares SHA-256 before & after. + +Rule 3 – Forbidden Features +Disable anything that leaks entropy or I/O: +- Jinja2: turn off `Undefined` fallback, block `import`. +- Go: prune `sprig` crypto & date funcs. +- Jsonnet: wrap stdlib, remove `std.native`. +- Dhall: already side-effect-free. +- CUE: disallow `command` & `http` builtins. + +────────────────────────────────────────────── +3. Drop-In Skeletons + +3.1 Go template (rules/resolve.go.tmpl) +```go +{{- $seed := .seed -}} +{{- $rand := rand.New(rand.NewSource(int64 $seed)) }} +{{- range $obj := .objects }} +{{ $obj | sprig.mustToPrettyJson | yamlFromJSON }} +{{- end }} +``` +Build helper: +```bash +go run ./render.go -seed $KINGDOM_SEED +``` + +3.2 Jsonnet (resolve.jsonnet) +```jsonnet +local seed = std.parseInt(std.extVar('seed')); +local rng = std.native('seededRand')(seed); +[ x + {color: rng.choice(['silver','obsidian'])} for x in yaml.load('objects/*.yaml') ] +``` + +3.3 CUE (resolve.cue) +```cue +package kingdom + +seed: int @tag(seed) +objects: [ for f in os.Glob("objects/*.yaml") { yaml.Unmarshal(os.ReadFile(f)) & { + color: *rng.choice(["silver","obsidian"]) | _ +}}] +``` + +3.4 Dhall (resolve.dhall) +```dhall +let seed = 42 +let rng = λ(n → Natural/fold seed …) -- deterministic pseudo RNG +in ./objects/*.yaml + : List { id : Text, color : Text } + |> List/map (λ(o → o // { color = rng.choice ["silver","obsidian"] })) +``` + +────────────────────────────────────────────── +4. Validation Harness (language-agnostic) + +```bash +#!/usr/bin/env bash +set -e +SEED=42 +for engine in jinja2 go jsonnet cue dhall; do + ./render-$engine --seed $SEED > out-$engine.yaml + echo "$engine SHA256: $(sha256sum out-$engine.yaml | cut -d' ' -f1)" +done +diff -q out-*.yaml || echo "⚠️ non-deterministic drift" +``` + +────────────────────────────────────────────── +5. Decision Heuristics + +| Situation | Pick | +|---|---| +| Stay in Python, need macros & inheritance | Jinja2 | +| Single static binary, embed templates | Go templates | +| Want schemas + data validation baked in | CUE | +| Pure functional, no side effects ever | Dhall | +| Deep recursion & arithmetic logic | Jsonnet | + +────────────────────────────────────────────── +6. One-Line Meta-Moral + +The crown (YAML) never changes; only the steward (template engine) does—choose the steward whose **leakage surface** matches the **risk budget** of your deterministic kingdom. + +--- + +# ***The Deterministic Serendipity Guide*** +## Using Jinja2 to Rule a Pure-YAML Kingdom +*(Private reference v1.0 – for your eyes only)* + +--- + +### 0. Executive Purpose +Demonstrate how **Jinja2 becomes the single source of truth** in a closed system where +- **YAML is the only artifact** (input, intermediate, output) +- **Objects are pure data** (no code, no imports) +- **Determinism is absolute** (same seed → same bits) +- **Serendipity is guaranteed** (drop any new `.yaml` and re-render) + +--- + +### 1. Kingdom Layout (single directory tree) + +``` +serendipity/ +├─ kingdom.yml # the royal charter +├─ objects/ # every object you will ever need +│ ├─ _manifest.yml # optional index +│ ├─ unicorn.yaml +│ ├─ dragon.yaml +│ └─ … +└─ rules/ + └─ resolve.j2 # the only executable artifact +``` + +--- + +### 2. The Royal Charter (`kingdom.yml`) + +```yaml +kingdom: + # determinism knobs + seed: 42 + rng: + engine: Random + module: random + magic: + engine: Magic + module: hashlib + + # path conventions + objects: "objects/**/*.yaml" # glob pattern + rules: "rules/resolve.j2" + + # post-processing + exports: + - format: yaml + to: stdout + - format: yaml + to: out/kingdom_rendered.yaml +``` + +Nothing outside this file ever changes the deterministic output except the **seed**. + +--- + +### 3. Object Contract (what your YAML may contain) + +Every object **MUST** be valid YAML and **MAY** include Jinja2 expressions inside **quoted** scalars only. + +```yaml +# objects/unicorn.yaml +id: unicorn +color: "{{ rng.choice(['silver','iridescent']) }}" +blessing: "{{ magic.uuid4() }}" +``` + +Important: +- Expressions are **strings** → safe for YAML parsers. +- No top-level keys named `jinja2` or `__` (reserved for rulebook). + +--- + +### 4. The Single Rulebook (`rules/resolve.j2`) + +```jinja2 +{#- 1. Bootstrap deterministic engines -#} +{% set rng = load_rng(kingdom.rng) %} +{% set magic = load_magic(kingdom.magic) %} + +{#- 2. Load and expand every object -#} +{% set court = [] %} +{% for path in glob(kingdom.objects) %} + {% set raw = read_file(path) %} + {% set yaml = from_yaml(render_str(raw, rng=rng, magic=magic)) %} + {{ court.append(yaml) or '' }} +{% endfor %} + +{#- 3. Emit final deterministic YAML -#} +{{ court | to_yaml }} +``` + +Helper filters (`load_rng`, `load_magic`, `glob`, `read_file`, `render_str`) are registered once at startup; **they never change** and are **not serialisable**, so they cannot leak into objects. + +--- + +### 5. Bootstrap Script (one-time, never edited) + +```python +#!/usr/bin/env python3 +# bootstrap.py (kept outside kingdom/) +import os, yaml, jinja2, glob, hashlib, random + +# deterministic seed from kingdom.yml +meta = yaml.safe_load(open('kingdom.yml')) +random.seed(meta['kingdom']['seed']) + +# Jinja2 env +env = jinja2.Environment( + loader=jinja2.FileSystemLoader('rules'), + finalize=lambda x: '' if x is None else x, +) +env.filters.update({ + 'to_yaml': lambda d: yaml.dump(d, sort_keys=True), +}) +env.globals.update({ + 'load_rng': lambda cfg: random, + 'load_magic': lambda cfg: type('Magic', (), {'uuid4': lambda: hashlib.md5(os.urandom(8)).hexdigest()[:8]})(), + 'glob': lambda p: glob.glob(p, recursive=True), + 'read_file': lambda f: open(f).read(), + 'render_str': lambda s, **kw: jinja2.Template(s, undefined=jinja2.StrictUndefined).render(**kw), +}) + +template = env.get_template('resolve.j2') +print(env.from_string(template.render(kingdom=meta['kingdom'])).render()) +``` + +Run once → deterministic stream of bytes. +Commit the stream if you need reproducibility without Python. + +--- + +### 6. Deep Powers Inside Jinja2 (exploited here) + +| Power | Usage in Kingdom | +|---|---| +| **StrictUndefined** | Every undefined var raises → no silent typos | +| **Custom globals/filters** | Inject rng, magic, filesystem helpers | +| **Macro imports** | `{% from 'macros.j2' import mutate %}` inside object templates | +| **Whitespace control** | `{%- … -%}` keeps emitted YAML clean | +| **Recursive rendering** | Object YAML contains Jinja2 that is re-parsed | +| **Immutable context** | No `{% set %}` leaks between object renders | + +--- + +### 7. Serendipity Workflow (non-breaking) + +1. Drop `objects/new_artifact.yaml` +2. Re-run `python bootstrap.py` +3. Diff the two deterministic outputs → only deltas from the new object appear. +4. Version-control the diff, not the code. + +--- + +### 8. Security & Reproducibility Checklist + +- **No Python code inside YAML** → safe for linters like yamllint. +- **Seed is single source of entropy** → store it in a repo tag for audit. +- **Read-only filesystem** → templates and bootstrap script are immutable at runtime. +- **One-way data flow**: YAML → Jinja2 → expanded YAML → stdout/file. +- **No external network calls** → reproducible in air-gapped CI. + +--- + +### 9. Single-Command Invocation (CI / Makefile) + +```makefile +render: + @python bootstrap.py > kingdom_rendered.yaml + @echo "SHA256: $$(sha256sum kingdom_rendered.yaml)" +``` + +--- + +### 10. Closing Axiom + +> In this kingdom the **template is the constitution**, +> the seed is the crown jewel, +> and every `.yaml` is a citizen whose fate is **deterministically serendipitous**. \ No newline at end of file