Add tech_docs/template_kindom.txt
This commit is contained in:
353
tech_docs/template_kindom.txt
Normal file
353
tech_docs/template_kindom.txt
Normal file
@@ -0,0 +1,353 @@
|
||||
Supremacy Verdict
|
||||
(inside the **Deterministic Serendipity Kingdom**, where YAML is king and *zero leakage* is law)
|
||||
|
||||
──────────────────────────────
|
||||
1. Scoring Table (0–5, 5 best)
|
||||
|
||||
| Criterion | Jinja2 | Go | Jsonnet | CUE | Dhall |
|
||||
|---|---|---|---|---|---|
|
||||
| **Determinism lock** | 4 | 5 | 5 | 5 | 5 |
|
||||
| **YAML round-trip fidelity** | 5 | 4 | 5 | 5 | 5 |
|
||||
| **Zero side-effect surface** | 3 | 4 | 5 | 5 | 5 |
|
||||
| **Macro / recursion depth** | 5 | 3 | 5 | 3 | 4 |
|
||||
| **Tooling friction** | 5 | 5 | 3 | 3 | 2 |
|
||||
| **Schema + validation** | 2 | 2 | 3 | 5 | 4 |
|
||||
|
||||
Total
|
||||
- **Dhall**: 25
|
||||
- **Jsonnet**: 24
|
||||
- **CUE**: 23
|
||||
- **Go**: 22
|
||||
- **Jinja2**: 21
|
||||
|
||||
──────────────────────────────
|
||||
2. Crown Holder: **Dhall**
|
||||
|
||||
Why
|
||||
- **Purity by design** – no IO, no random, no exceptions.
|
||||
- **Determinism by construction** – same inputs → same output, period.
|
||||
- **Native YAML emitter** – canonical, sorted, no tags.
|
||||
- **Functions as first-class citizens** – richer than CUE, safer than Jsonnet.
|
||||
- **Single static binary** – cross-compiled `dhall-to-yaml` < 20 MB.
|
||||
|
||||
Trade-off: steeper learning curve, but once learned **it cannot break the kingdom’s law**.
|
||||
|
||||
──────────────────────────────
|
||||
3. First Runner-Up: **Jsonnet**
|
||||
|
||||
If you need deep recursion or heavier arithmetic logic, Jsonnet edges Dhall.
|
||||
Still pure, still deterministic, but larger surface area (stdlib) and heavier runtime.
|
||||
|
||||
──────────────────────────────
|
||||
4. Final Decree
|
||||
|
||||
> In the Deterministic Serendipity Kingdom **Dhall reigns supreme**.
|
||||
> Its crown is forged from mathematical purity; no steward can leak entropy, no subject can smuggle side effects.
|
||||
|
||||
---
|
||||
|
||||
──────────────────────────────────────────────
|
||||
DETERMINISTIC SERENDIPITY – META TOOLKIT
|
||||
(“YAML is still king; the crown can change heads.”)
|
||||
──────────────────────────────────────────────
|
||||
|
||||
0. Guiding Question
|
||||
“Which templating engines can **replace** Jinja2 **without breaking** the deterministic-serendipity contract?”
|
||||
- YAML-only input / output
|
||||
- Single deterministic seed
|
||||
- No embedded imperative code in the objects
|
||||
|
||||
Below is a **matrix + migration cookbook** for the four engines most often considered.
|
||||
|
||||
──────────────────────────────────────────────
|
||||
1. Candidate Matrix (feature vs. Jinja2)
|
||||
|
||||
| Engine | Deterministic Seed | YAML Load/Emit | Inline Expressions | Recursion | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| **Jinja2** | ✅ (custom globals) | ✅ (PyYAML) | `{{ }}` | ✅ | reference |
|
||||
| **Go text/template + sprig** | ✅ (rand.Seed) | ✅ (gopkg.in/yaml.v3) | `{{ }}` | ✅ | binary embed |
|
||||
| **CUE templates** | ✅ (cue.Value) | ✅ (native) | `{{ }}` & CUE constraints | ✅ | schema + template |
|
||||
| **Jsonnet** | ✅ (std.md5 + seed) | ✅ (std.manifestYaml) | `+` / `|` / `std.*` | ✅ | pure functional |
|
||||
| **Dhall** | ✅ (deterministic by design) | ✅ (dhall-to-yaml) | λ-expressions | ✅ | no side effects |
|
||||
|
||||
──────────────────────────────────────────────
|
||||
2. Migration Rules (preserve determinism)
|
||||
|
||||
Rule 1 – Seed Isolation
|
||||
- **Jinja2 / Go / sprig**: call `rand.Seed(n)` or provide seeded `rand.Rand` instance.
|
||||
- **Jsonnet**: pass seed as top-level function argument.
|
||||
- **Dhall**: seed is an ordinary parameter (Dhall is pure).
|
||||
- **CUE**: seed is injected via `let seed = <int>` at template root.
|
||||
|
||||
Rule 2 – YAML Round-Trip
|
||||
Every engine must:
|
||||
1. Load raw YAML → AST (strict keys, no anchors).
|
||||
2. Render expressions.
|
||||
3. Emit canonical YAML (sorted keys, no tags).
|
||||
Provide a **validator** script that compares SHA-256 before & after.
|
||||
|
||||
Rule 3 – Forbidden Features
|
||||
Disable anything that leaks entropy or I/O:
|
||||
- Jinja2: turn off `Undefined` fallback, block `import`.
|
||||
- Go: prune `sprig` crypto & date funcs.
|
||||
- Jsonnet: wrap stdlib, remove `std.native`.
|
||||
- Dhall: already side-effect-free.
|
||||
- CUE: disallow `command` & `http` builtins.
|
||||
|
||||
──────────────────────────────────────────────
|
||||
3. Drop-In Skeletons
|
||||
|
||||
3.1 Go template (rules/resolve.go.tmpl)
|
||||
```go
|
||||
{{- $seed := .seed -}}
|
||||
{{- $rand := rand.New(rand.NewSource(int64 $seed)) }}
|
||||
{{- range $obj := .objects }}
|
||||
{{ $obj | sprig.mustToPrettyJson | yamlFromJSON }}
|
||||
{{- end }}
|
||||
```
|
||||
Build helper:
|
||||
```bash
|
||||
go run ./render.go -seed $KINGDOM_SEED
|
||||
```
|
||||
|
||||
3.2 Jsonnet (resolve.jsonnet)
|
||||
```jsonnet
|
||||
local seed = std.parseInt(std.extVar('seed'));
|
||||
local rng = std.native('seededRand')(seed);
|
||||
[ x + {color: rng.choice(['silver','obsidian'])} for x in yaml.load('objects/*.yaml') ]
|
||||
```
|
||||
|
||||
3.3 CUE (resolve.cue)
|
||||
```cue
|
||||
package kingdom
|
||||
|
||||
seed: int @tag(seed)
|
||||
objects: [ for f in os.Glob("objects/*.yaml") { yaml.Unmarshal(os.ReadFile(f)) & {
|
||||
color: *rng.choice(["silver","obsidian"]) | _
|
||||
}}]
|
||||
```
|
||||
|
||||
3.4 Dhall (resolve.dhall)
|
||||
```dhall
|
||||
let seed = 42
|
||||
let rng = λ(n → Natural/fold seed …) -- deterministic pseudo RNG
|
||||
in ./objects/*.yaml
|
||||
: List { id : Text, color : Text }
|
||||
|> List/map (λ(o → o // { color = rng.choice ["silver","obsidian"] }))
|
||||
```
|
||||
|
||||
──────────────────────────────────────────────
|
||||
4. Validation Harness (language-agnostic)
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
SEED=42
|
||||
for engine in jinja2 go jsonnet cue dhall; do
|
||||
./render-$engine --seed $SEED > out-$engine.yaml
|
||||
echo "$engine SHA256: $(sha256sum out-$engine.yaml | cut -d' ' -f1)"
|
||||
done
|
||||
diff -q out-*.yaml || echo "⚠️ non-deterministic drift"
|
||||
```
|
||||
|
||||
──────────────────────────────────────────────
|
||||
5. Decision Heuristics
|
||||
|
||||
| Situation | Pick |
|
||||
|---|---|
|
||||
| Stay in Python, need macros & inheritance | Jinja2 |
|
||||
| Single static binary, embed templates | Go templates |
|
||||
| Want schemas + data validation baked in | CUE |
|
||||
| Pure functional, no side effects ever | Dhall |
|
||||
| Deep recursion & arithmetic logic | Jsonnet |
|
||||
|
||||
──────────────────────────────────────────────
|
||||
6. One-Line Meta-Moral
|
||||
|
||||
The crown (YAML) never changes; only the steward (template engine) does—choose the steward whose **leakage surface** matches the **risk budget** of your deterministic kingdom.
|
||||
|
||||
---
|
||||
|
||||
# ***The Deterministic Serendipity Guide***
|
||||
## Using Jinja2 to Rule a Pure-YAML Kingdom
|
||||
*(Private reference v1.0 – for your eyes only)*
|
||||
|
||||
---
|
||||
|
||||
### 0. Executive Purpose
|
||||
Demonstrate how **Jinja2 becomes the single source of truth** in a closed system where
|
||||
- **YAML is the only artifact** (input, intermediate, output)
|
||||
- **Objects are pure data** (no code, no imports)
|
||||
- **Determinism is absolute** (same seed → same bits)
|
||||
- **Serendipity is guaranteed** (drop any new `.yaml` and re-render)
|
||||
|
||||
---
|
||||
|
||||
### 1. Kingdom Layout (single directory tree)
|
||||
|
||||
```
|
||||
serendipity/
|
||||
├─ kingdom.yml # the royal charter
|
||||
├─ objects/ # every object you will ever need
|
||||
│ ├─ _manifest.yml # optional index
|
||||
│ ├─ unicorn.yaml
|
||||
│ ├─ dragon.yaml
|
||||
│ └─ …
|
||||
└─ rules/
|
||||
└─ resolve.j2 # the only executable artifact
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. The Royal Charter (`kingdom.yml`)
|
||||
|
||||
```yaml
|
||||
kingdom:
|
||||
# determinism knobs
|
||||
seed: 42
|
||||
rng:
|
||||
engine: Random
|
||||
module: random
|
||||
magic:
|
||||
engine: Magic
|
||||
module: hashlib
|
||||
|
||||
# path conventions
|
||||
objects: "objects/**/*.yaml" # glob pattern
|
||||
rules: "rules/resolve.j2"
|
||||
|
||||
# post-processing
|
||||
exports:
|
||||
- format: yaml
|
||||
to: stdout
|
||||
- format: yaml
|
||||
to: out/kingdom_rendered.yaml
|
||||
```
|
||||
|
||||
Nothing outside this file ever changes the deterministic output except the **seed**.
|
||||
|
||||
---
|
||||
|
||||
### 3. Object Contract (what your YAML may contain)
|
||||
|
||||
Every object **MUST** be valid YAML and **MAY** include Jinja2 expressions inside **quoted** scalars only.
|
||||
|
||||
```yaml
|
||||
# objects/unicorn.yaml
|
||||
id: unicorn
|
||||
color: "{{ rng.choice(['silver','iridescent']) }}"
|
||||
blessing: "{{ magic.uuid4() }}"
|
||||
```
|
||||
|
||||
Important:
|
||||
- Expressions are **strings** → safe for YAML parsers.
|
||||
- No top-level keys named `jinja2` or `__` (reserved for rulebook).
|
||||
|
||||
---
|
||||
|
||||
### 4. The Single Rulebook (`rules/resolve.j2`)
|
||||
|
||||
```jinja2
|
||||
{#- 1. Bootstrap deterministic engines -#}
|
||||
{% set rng = load_rng(kingdom.rng) %}
|
||||
{% set magic = load_magic(kingdom.magic) %}
|
||||
|
||||
{#- 2. Load and expand every object -#}
|
||||
{% set court = [] %}
|
||||
{% for path in glob(kingdom.objects) %}
|
||||
{% set raw = read_file(path) %}
|
||||
{% set yaml = from_yaml(render_str(raw, rng=rng, magic=magic)) %}
|
||||
{{ court.append(yaml) or '' }}
|
||||
{% endfor %}
|
||||
|
||||
{#- 3. Emit final deterministic YAML -#}
|
||||
{{ court | to_yaml }}
|
||||
```
|
||||
|
||||
Helper filters (`load_rng`, `load_magic`, `glob`, `read_file`, `render_str`) are registered once at startup; **they never change** and are **not serialisable**, so they cannot leak into objects.
|
||||
|
||||
---
|
||||
|
||||
### 5. Bootstrap Script (one-time, never edited)
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
# bootstrap.py (kept outside kingdom/)
|
||||
import os, yaml, jinja2, glob, hashlib, random
|
||||
|
||||
# deterministic seed from kingdom.yml
|
||||
meta = yaml.safe_load(open('kingdom.yml'))
|
||||
random.seed(meta['kingdom']['seed'])
|
||||
|
||||
# Jinja2 env
|
||||
env = jinja2.Environment(
|
||||
loader=jinja2.FileSystemLoader('rules'),
|
||||
finalize=lambda x: '' if x is None else x,
|
||||
)
|
||||
env.filters.update({
|
||||
'to_yaml': lambda d: yaml.dump(d, sort_keys=True),
|
||||
})
|
||||
env.globals.update({
|
||||
'load_rng': lambda cfg: random,
|
||||
'load_magic': lambda cfg: type('Magic', (), {'uuid4': lambda: hashlib.md5(os.urandom(8)).hexdigest()[:8]})(),
|
||||
'glob': lambda p: glob.glob(p, recursive=True),
|
||||
'read_file': lambda f: open(f).read(),
|
||||
'render_str': lambda s, **kw: jinja2.Template(s, undefined=jinja2.StrictUndefined).render(**kw),
|
||||
})
|
||||
|
||||
template = env.get_template('resolve.j2')
|
||||
print(env.from_string(template.render(kingdom=meta['kingdom'])).render())
|
||||
```
|
||||
|
||||
Run once → deterministic stream of bytes.
|
||||
Commit the stream if you need reproducibility without Python.
|
||||
|
||||
---
|
||||
|
||||
### 6. Deep Powers Inside Jinja2 (exploited here)
|
||||
|
||||
| Power | Usage in Kingdom |
|
||||
|---|---|
|
||||
| **StrictUndefined** | Every undefined var raises → no silent typos |
|
||||
| **Custom globals/filters** | Inject rng, magic, filesystem helpers |
|
||||
| **Macro imports** | `{% from 'macros.j2' import mutate %}` inside object templates |
|
||||
| **Whitespace control** | `{%- … -%}` keeps emitted YAML clean |
|
||||
| **Recursive rendering** | Object YAML contains Jinja2 that is re-parsed |
|
||||
| **Immutable context** | No `{% set %}` leaks between object renders |
|
||||
|
||||
---
|
||||
|
||||
### 7. Serendipity Workflow (non-breaking)
|
||||
|
||||
1. Drop `objects/new_artifact.yaml`
|
||||
2. Re-run `python bootstrap.py`
|
||||
3. Diff the two deterministic outputs → only deltas from the new object appear.
|
||||
4. Version-control the diff, not the code.
|
||||
|
||||
---
|
||||
|
||||
### 8. Security & Reproducibility Checklist
|
||||
|
||||
- **No Python code inside YAML** → safe for linters like yamllint.
|
||||
- **Seed is single source of entropy** → store it in a repo tag for audit.
|
||||
- **Read-only filesystem** → templates and bootstrap script are immutable at runtime.
|
||||
- **One-way data flow**: YAML → Jinja2 → expanded YAML → stdout/file.
|
||||
- **No external network calls** → reproducible in air-gapped CI.
|
||||
|
||||
---
|
||||
|
||||
### 9. Single-Command Invocation (CI / Makefile)
|
||||
|
||||
```makefile
|
||||
render:
|
||||
@python bootstrap.py > kingdom_rendered.yaml
|
||||
@echo "SHA256: $$(sha256sum kingdom_rendered.yaml)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 10. Closing Axiom
|
||||
|
||||
> In this kingdom the **template is the constitution**,
|
||||
> the seed is the crown jewel,
|
||||
> and every `.yaml` is a citizen whose fate is **deterministically serendipitous**.
|
||||
Reference in New Issue
Block a user