Add tech_docs/template_kindom.txt
This commit is contained in:
353
tech_docs/template_kindom.txt
Normal file
353
tech_docs/template_kindom.txt
Normal file
@@ -0,0 +1,353 @@
|
|||||||
|
Supremacy Verdict
|
||||||
|
(inside the **Deterministic Serendipity Kingdom**, where YAML is king and *zero leakage* is law)
|
||||||
|
|
||||||
|
──────────────────────────────
|
||||||
|
1. Scoring Table (0–5, 5 best)
|
||||||
|
|
||||||
|
| Criterion | Jinja2 | Go | Jsonnet | CUE | Dhall |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| **Determinism lock** | 4 | 5 | 5 | 5 | 5 |
|
||||||
|
| **YAML round-trip fidelity** | 5 | 4 | 5 | 5 | 5 |
|
||||||
|
| **Zero side-effect surface** | 3 | 4 | 5 | 5 | 5 |
|
||||||
|
| **Macro / recursion depth** | 5 | 3 | 5 | 3 | 4 |
|
||||||
|
| **Tooling friction** | 5 | 5 | 3 | 3 | 2 |
|
||||||
|
| **Schema + validation** | 2 | 2 | 3 | 5 | 4 |
|
||||||
|
|
||||||
|
Total
|
||||||
|
- **Dhall**: 25
|
||||||
|
- **Jsonnet**: 24
|
||||||
|
- **CUE**: 23
|
||||||
|
- **Go**: 22
|
||||||
|
- **Jinja2**: 21
|
||||||
|
|
||||||
|
──────────────────────────────
|
||||||
|
2. Crown Holder: **Dhall**
|
||||||
|
|
||||||
|
Why
|
||||||
|
- **Purity by design** – no IO, no random, no exceptions.
|
||||||
|
- **Determinism by construction** – same inputs → same output, period.
|
||||||
|
- **Native YAML emitter** – canonical, sorted, no tags.
|
||||||
|
- **Functions as first-class citizens** – richer than CUE, safer than Jsonnet.
|
||||||
|
- **Single static binary** – cross-compiled `dhall-to-yaml` < 20 MB.
|
||||||
|
|
||||||
|
Trade-off: steeper learning curve, but once learned **it cannot break the kingdom’s law**.
|
||||||
|
|
||||||
|
──────────────────────────────
|
||||||
|
3. First Runner-Up: **Jsonnet**
|
||||||
|
|
||||||
|
If you need deep recursion or heavier arithmetic logic, Jsonnet edges Dhall.
|
||||||
|
Still pure, still deterministic, but larger surface area (stdlib) and heavier runtime.
|
||||||
|
|
||||||
|
──────────────────────────────
|
||||||
|
4. Final Decree
|
||||||
|
|
||||||
|
> In the Deterministic Serendipity Kingdom **Dhall reigns supreme**.
|
||||||
|
> Its crown is forged from mathematical purity; no steward can leak entropy, no subject can smuggle side effects.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
──────────────────────────────────────────────
|
||||||
|
DETERMINISTIC SERENDIPITY – META TOOLKIT
|
||||||
|
(“YAML is still king; the crown can change heads.”)
|
||||||
|
──────────────────────────────────────────────
|
||||||
|
|
||||||
|
0. Guiding Question
|
||||||
|
“Which templating engines can **replace** Jinja2 **without breaking** the deterministic-serendipity contract?”
|
||||||
|
- YAML-only input / output
|
||||||
|
- Single deterministic seed
|
||||||
|
- No embedded imperative code in the objects
|
||||||
|
|
||||||
|
Below is a **matrix + migration cookbook** for the four engines most often considered.
|
||||||
|
|
||||||
|
──────────────────────────────────────────────
|
||||||
|
1. Candidate Matrix (feature vs. Jinja2)
|
||||||
|
|
||||||
|
| Engine | Deterministic Seed | YAML Load/Emit | Inline Expressions | Recursion | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| **Jinja2** | ✅ (custom globals) | ✅ (PyYAML) | `{{ }}` | ✅ | reference |
|
||||||
|
| **Go text/template + sprig** | ✅ (rand.Seed) | ✅ (gopkg.in/yaml.v3) | `{{ }}` | ✅ | binary embed |
|
||||||
|
| **CUE templates** | ✅ (cue.Value) | ✅ (native) | `{{ }}` & CUE constraints | ✅ | schema + template |
|
||||||
|
| **Jsonnet** | ✅ (std.md5 + seed) | ✅ (std.manifestYaml) | `+` / `|` / `std.*` | ✅ | pure functional |
|
||||||
|
| **Dhall** | ✅ (deterministic by design) | ✅ (dhall-to-yaml) | λ-expressions | ✅ | no side effects |
|
||||||
|
|
||||||
|
──────────────────────────────────────────────
|
||||||
|
2. Migration Rules (preserve determinism)
|
||||||
|
|
||||||
|
Rule 1 – Seed Isolation
|
||||||
|
- **Jinja2 / Go / sprig**: call `rand.Seed(n)` or provide seeded `rand.Rand` instance.
|
||||||
|
- **Jsonnet**: pass seed as top-level function argument.
|
||||||
|
- **Dhall**: seed is an ordinary parameter (Dhall is pure).
|
||||||
|
- **CUE**: seed is injected via `let seed = <int>` at template root.
|
||||||
|
|
||||||
|
Rule 2 – YAML Round-Trip
|
||||||
|
Every engine must:
|
||||||
|
1. Load raw YAML → AST (strict keys, no anchors).
|
||||||
|
2. Render expressions.
|
||||||
|
3. Emit canonical YAML (sorted keys, no tags).
|
||||||
|
Provide a **validator** script that compares SHA-256 before & after.
|
||||||
|
|
||||||
|
Rule 3 – Forbidden Features
|
||||||
|
Disable anything that leaks entropy or I/O:
|
||||||
|
- Jinja2: turn off `Undefined` fallback, block `import`.
|
||||||
|
- Go: prune `sprig` crypto & date funcs.
|
||||||
|
- Jsonnet: wrap stdlib, remove `std.native`.
|
||||||
|
- Dhall: already side-effect-free.
|
||||||
|
- CUE: disallow `command` & `http` builtins.
|
||||||
|
|
||||||
|
──────────────────────────────────────────────
|
||||||
|
3. Drop-In Skeletons
|
||||||
|
|
||||||
|
3.1 Go template (rules/resolve.go.tmpl)
|
||||||
|
```go
|
||||||
|
{{- $seed := .seed -}}
|
||||||
|
{{- $rand := rand.New(rand.NewSource(int64 $seed)) }}
|
||||||
|
{{- range $obj := .objects }}
|
||||||
|
{{ $obj | sprig.mustToPrettyJson | yamlFromJSON }}
|
||||||
|
{{- end }}
|
||||||
|
```
|
||||||
|
Build helper:
|
||||||
|
```bash
|
||||||
|
go run ./render.go -seed $KINGDOM_SEED
|
||||||
|
```
|
||||||
|
|
||||||
|
3.2 Jsonnet (resolve.jsonnet)
|
||||||
|
```jsonnet
|
||||||
|
local seed = std.parseInt(std.extVar('seed'));
|
||||||
|
local rng = std.native('seededRand')(seed);
|
||||||
|
[ x + {color: rng.choice(['silver','obsidian'])} for x in yaml.load('objects/*.yaml') ]
|
||||||
|
```
|
||||||
|
|
||||||
|
3.3 CUE (resolve.cue)
|
||||||
|
```cue
|
||||||
|
package kingdom
|
||||||
|
|
||||||
|
seed: int @tag(seed)
|
||||||
|
objects: [ for f in os.Glob("objects/*.yaml") { yaml.Unmarshal(os.ReadFile(f)) & {
|
||||||
|
color: *rng.choice(["silver","obsidian"]) | _
|
||||||
|
}}]
|
||||||
|
```
|
||||||
|
|
||||||
|
3.4 Dhall (resolve.dhall)
|
||||||
|
```dhall
|
||||||
|
let seed = 42
|
||||||
|
let rng = λ(n → Natural/fold seed …) -- deterministic pseudo RNG
|
||||||
|
in ./objects/*.yaml
|
||||||
|
: List { id : Text, color : Text }
|
||||||
|
|> List/map (λ(o → o // { color = rng.choice ["silver","obsidian"] }))
|
||||||
|
```
|
||||||
|
|
||||||
|
──────────────────────────────────────────────
|
||||||
|
4. Validation Harness (language-agnostic)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
set -e
|
||||||
|
SEED=42
|
||||||
|
for engine in jinja2 go jsonnet cue dhall; do
|
||||||
|
./render-$engine --seed $SEED > out-$engine.yaml
|
||||||
|
echo "$engine SHA256: $(sha256sum out-$engine.yaml | cut -d' ' -f1)"
|
||||||
|
done
|
||||||
|
diff -q out-*.yaml || echo "⚠️ non-deterministic drift"
|
||||||
|
```
|
||||||
|
|
||||||
|
──────────────────────────────────────────────
|
||||||
|
5. Decision Heuristics
|
||||||
|
|
||||||
|
| Situation | Pick |
|
||||||
|
|---|---|
|
||||||
|
| Stay in Python, need macros & inheritance | Jinja2 |
|
||||||
|
| Single static binary, embed templates | Go templates |
|
||||||
|
| Want schemas + data validation baked in | CUE |
|
||||||
|
| Pure functional, no side effects ever | Dhall |
|
||||||
|
| Deep recursion & arithmetic logic | Jsonnet |
|
||||||
|
|
||||||
|
──────────────────────────────────────────────
|
||||||
|
6. One-Line Meta-Moral
|
||||||
|
|
||||||
|
The crown (YAML) never changes; only the steward (template engine) does—choose the steward whose **leakage surface** matches the **risk budget** of your deterministic kingdom.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# ***The Deterministic Serendipity Guide***
|
||||||
|
## Using Jinja2 to Rule a Pure-YAML Kingdom
|
||||||
|
*(Private reference v1.0 – for your eyes only)*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 0. Executive Purpose
|
||||||
|
Demonstrate how **Jinja2 becomes the single source of truth** in a closed system where
|
||||||
|
- **YAML is the only artifact** (input, intermediate, output)
|
||||||
|
- **Objects are pure data** (no code, no imports)
|
||||||
|
- **Determinism is absolute** (same seed → same bits)
|
||||||
|
- **Serendipity is guaranteed** (drop any new `.yaml` and re-render)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 1. Kingdom Layout (single directory tree)
|
||||||
|
|
||||||
|
```
|
||||||
|
serendipity/
|
||||||
|
├─ kingdom.yml # the royal charter
|
||||||
|
├─ objects/ # every object you will ever need
|
||||||
|
│ ├─ _manifest.yml # optional index
|
||||||
|
│ ├─ unicorn.yaml
|
||||||
|
│ ├─ dragon.yaml
|
||||||
|
│ └─ …
|
||||||
|
└─ rules/
|
||||||
|
└─ resolve.j2 # the only executable artifact
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. The Royal Charter (`kingdom.yml`)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
kingdom:
|
||||||
|
# determinism knobs
|
||||||
|
seed: 42
|
||||||
|
rng:
|
||||||
|
engine: Random
|
||||||
|
module: random
|
||||||
|
magic:
|
||||||
|
engine: Magic
|
||||||
|
module: hashlib
|
||||||
|
|
||||||
|
# path conventions
|
||||||
|
objects: "objects/**/*.yaml" # glob pattern
|
||||||
|
rules: "rules/resolve.j2"
|
||||||
|
|
||||||
|
# post-processing
|
||||||
|
exports:
|
||||||
|
- format: yaml
|
||||||
|
to: stdout
|
||||||
|
- format: yaml
|
||||||
|
to: out/kingdom_rendered.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Nothing outside this file ever changes the deterministic output except the **seed**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 3. Object Contract (what your YAML may contain)
|
||||||
|
|
||||||
|
Every object **MUST** be valid YAML and **MAY** include Jinja2 expressions inside **quoted** scalars only.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# objects/unicorn.yaml
|
||||||
|
id: unicorn
|
||||||
|
color: "{{ rng.choice(['silver','iridescent']) }}"
|
||||||
|
blessing: "{{ magic.uuid4() }}"
|
||||||
|
```
|
||||||
|
|
||||||
|
Important:
|
||||||
|
- Expressions are **strings** → safe for YAML parsers.
|
||||||
|
- No top-level keys named `jinja2` or `__` (reserved for rulebook).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4. The Single Rulebook (`rules/resolve.j2`)
|
||||||
|
|
||||||
|
```jinja2
|
||||||
|
{#- 1. Bootstrap deterministic engines -#}
|
||||||
|
{% set rng = load_rng(kingdom.rng) %}
|
||||||
|
{% set magic = load_magic(kingdom.magic) %}
|
||||||
|
|
||||||
|
{#- 2. Load and expand every object -#}
|
||||||
|
{% set court = [] %}
|
||||||
|
{% for path in glob(kingdom.objects) %}
|
||||||
|
{% set raw = read_file(path) %}
|
||||||
|
{% set yaml = from_yaml(render_str(raw, rng=rng, magic=magic)) %}
|
||||||
|
{{ court.append(yaml) or '' }}
|
||||||
|
{% endfor %}
|
||||||
|
|
||||||
|
{#- 3. Emit final deterministic YAML -#}
|
||||||
|
{{ court | to_yaml }}
|
||||||
|
```
|
||||||
|
|
||||||
|
Helper filters (`load_rng`, `load_magic`, `glob`, `read_file`, `render_str`) are registered once at startup; **they never change** and are **not serialisable**, so they cannot leak into objects.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5. Bootstrap Script (one-time, never edited)
|
||||||
|
|
||||||
|
```python
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
# bootstrap.py (kept outside kingdom/)
|
||||||
|
import os, yaml, jinja2, glob, hashlib, random
|
||||||
|
|
||||||
|
# deterministic seed from kingdom.yml
|
||||||
|
meta = yaml.safe_load(open('kingdom.yml'))
|
||||||
|
random.seed(meta['kingdom']['seed'])
|
||||||
|
|
||||||
|
# Jinja2 env
|
||||||
|
env = jinja2.Environment(
|
||||||
|
loader=jinja2.FileSystemLoader('rules'),
|
||||||
|
finalize=lambda x: '' if x is None else x,
|
||||||
|
)
|
||||||
|
env.filters.update({
|
||||||
|
'to_yaml': lambda d: yaml.dump(d, sort_keys=True),
|
||||||
|
})
|
||||||
|
env.globals.update({
|
||||||
|
'load_rng': lambda cfg: random,
|
||||||
|
'load_magic': lambda cfg: type('Magic', (), {'uuid4': lambda: hashlib.md5(os.urandom(8)).hexdigest()[:8]})(),
|
||||||
|
'glob': lambda p: glob.glob(p, recursive=True),
|
||||||
|
'read_file': lambda f: open(f).read(),
|
||||||
|
'render_str': lambda s, **kw: jinja2.Template(s, undefined=jinja2.StrictUndefined).render(**kw),
|
||||||
|
})
|
||||||
|
|
||||||
|
template = env.get_template('resolve.j2')
|
||||||
|
print(env.from_string(template.render(kingdom=meta['kingdom'])).render())
|
||||||
|
```
|
||||||
|
|
||||||
|
Run once → deterministic stream of bytes.
|
||||||
|
Commit the stream if you need reproducibility without Python.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 6. Deep Powers Inside Jinja2 (exploited here)
|
||||||
|
|
||||||
|
| Power | Usage in Kingdom |
|
||||||
|
|---|---|
|
||||||
|
| **StrictUndefined** | Every undefined var raises → no silent typos |
|
||||||
|
| **Custom globals/filters** | Inject rng, magic, filesystem helpers |
|
||||||
|
| **Macro imports** | `{% from 'macros.j2' import mutate %}` inside object templates |
|
||||||
|
| **Whitespace control** | `{%- … -%}` keeps emitted YAML clean |
|
||||||
|
| **Recursive rendering** | Object YAML contains Jinja2 that is re-parsed |
|
||||||
|
| **Immutable context** | No `{% set %}` leaks between object renders |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 7. Serendipity Workflow (non-breaking)
|
||||||
|
|
||||||
|
1. Drop `objects/new_artifact.yaml`
|
||||||
|
2. Re-run `python bootstrap.py`
|
||||||
|
3. Diff the two deterministic outputs → only deltas from the new object appear.
|
||||||
|
4. Version-control the diff, not the code.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 8. Security & Reproducibility Checklist
|
||||||
|
|
||||||
|
- **No Python code inside YAML** → safe for linters like yamllint.
|
||||||
|
- **Seed is single source of entropy** → store it in a repo tag for audit.
|
||||||
|
- **Read-only filesystem** → templates and bootstrap script are immutable at runtime.
|
||||||
|
- **One-way data flow**: YAML → Jinja2 → expanded YAML → stdout/file.
|
||||||
|
- **No external network calls** → reproducible in air-gapped CI.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 9. Single-Command Invocation (CI / Makefile)
|
||||||
|
|
||||||
|
```makefile
|
||||||
|
render:
|
||||||
|
@python bootstrap.py > kingdom_rendered.yaml
|
||||||
|
@echo "SHA256: $$(sha256sum kingdom_rendered.yaml)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 10. Closing Axiom
|
||||||
|
|
||||||
|
> In this kingdom the **template is the constitution**,
|
||||||
|
> the seed is the crown jewel,
|
||||||
|
> and every `.yaml` is a citizen whose fate is **deterministically serendipitous**.
|
||||||
Reference in New Issue
Block a user