353 lines
12 KiB
Plaintext
353 lines
12 KiB
Plaintext
Supremacy Verdict
|
||
(inside the **Deterministic Serendipity Kingdom**, where YAML is king and *zero leakage* is law)
|
||
|
||
──────────────────────────────
|
||
1. Scoring Table (0–5, 5 best)
|
||
|
||
| Criterion | Jinja2 | Go | Jsonnet | CUE | Dhall |
|
||
|---|---|---|---|---|---|
|
||
| **Determinism lock** | 4 | 5 | 5 | 5 | 5 |
|
||
| **YAML round-trip fidelity** | 5 | 4 | 5 | 5 | 5 |
|
||
| **Zero side-effect surface** | 3 | 4 | 5 | 5 | 5 |
|
||
| **Macro / recursion depth** | 5 | 3 | 5 | 3 | 4 |
|
||
| **Tooling friction** | 5 | 5 | 3 | 3 | 2 |
|
||
| **Schema + validation** | 2 | 2 | 3 | 5 | 4 |
|
||
|
||
Total
|
||
- **Dhall**: 25
|
||
- **Jsonnet**: 24
|
||
- **CUE**: 23
|
||
- **Go**: 22
|
||
- **Jinja2**: 21
|
||
|
||
──────────────────────────────
|
||
2. Crown Holder: **Dhall**
|
||
|
||
Why
|
||
- **Purity by design** – no IO, no random, no exceptions.
|
||
- **Determinism by construction** – same inputs → same output, period.
|
||
- **Native YAML emitter** – canonical, sorted, no tags.
|
||
- **Functions as first-class citizens** – richer than CUE, safer than Jsonnet.
|
||
- **Single static binary** – cross-compiled `dhall-to-yaml` < 20 MB.
|
||
|
||
Trade-off: steeper learning curve, but once learned **it cannot break the kingdom’s law**.
|
||
|
||
──────────────────────────────
|
||
3. First Runner-Up: **Jsonnet**
|
||
|
||
If you need deep recursion or heavier arithmetic logic, Jsonnet edges Dhall.
|
||
Still pure, still deterministic, but larger surface area (stdlib) and heavier runtime.
|
||
|
||
──────────────────────────────
|
||
4. Final Decree
|
||
|
||
> In the Deterministic Serendipity Kingdom **Dhall reigns supreme**.
|
||
> Its crown is forged from mathematical purity; no steward can leak entropy, no subject can smuggle side effects.
|
||
|
||
---
|
||
|
||
──────────────────────────────────────────────
|
||
DETERMINISTIC SERENDIPITY – META TOOLKIT
|
||
(“YAML is still king; the crown can change heads.”)
|
||
──────────────────────────────────────────────
|
||
|
||
0. Guiding Question
|
||
“Which templating engines can **replace** Jinja2 **without breaking** the deterministic-serendipity contract?”
|
||
- YAML-only input / output
|
||
- Single deterministic seed
|
||
- No embedded imperative code in the objects
|
||
|
||
Below is a **matrix + migration cookbook** for the four engines most often considered.
|
||
|
||
──────────────────────────────────────────────
|
||
1. Candidate Matrix (feature vs. Jinja2)
|
||
|
||
| Engine | Deterministic Seed | YAML Load/Emit | Inline Expressions | Recursion | Notes |
|
||
|---|---|---|---|---|---|
|
||
| **Jinja2** | ✅ (custom globals) | ✅ (PyYAML) | `{{ }}` | ✅ | reference |
|
||
| **Go text/template + sprig** | ✅ (rand.Seed) | ✅ (gopkg.in/yaml.v3) | `{{ }}` | ✅ | binary embed |
|
||
| **CUE templates** | ✅ (cue.Value) | ✅ (native) | `{{ }}` & CUE constraints | ✅ | schema + template |
|
||
| **Jsonnet** | ✅ (std.md5 + seed) | ✅ (std.manifestYaml) | `+` / `|` / `std.*` | ✅ | pure functional |
|
||
| **Dhall** | ✅ (deterministic by design) | ✅ (dhall-to-yaml) | λ-expressions | ✅ | no side effects |
|
||
|
||
──────────────────────────────────────────────
|
||
2. Migration Rules (preserve determinism)
|
||
|
||
Rule 1 – Seed Isolation
|
||
- **Jinja2 / Go / sprig**: call `rand.Seed(n)` or provide seeded `rand.Rand` instance.
|
||
- **Jsonnet**: pass seed as top-level function argument.
|
||
- **Dhall**: seed is an ordinary parameter (Dhall is pure).
|
||
- **CUE**: seed is injected via `let seed = <int>` at template root.
|
||
|
||
Rule 2 – YAML Round-Trip
|
||
Every engine must:
|
||
1. Load raw YAML → AST (strict keys, no anchors).
|
||
2. Render expressions.
|
||
3. Emit canonical YAML (sorted keys, no tags).
|
||
Provide a **validator** script that compares SHA-256 before & after.
|
||
|
||
Rule 3 – Forbidden Features
|
||
Disable anything that leaks entropy or I/O:
|
||
- Jinja2: turn off `Undefined` fallback, block `import`.
|
||
- Go: prune `sprig` crypto & date funcs.
|
||
- Jsonnet: wrap stdlib, remove `std.native`.
|
||
- Dhall: already side-effect-free.
|
||
- CUE: disallow `command` & `http` builtins.
|
||
|
||
──────────────────────────────────────────────
|
||
3. Drop-In Skeletons
|
||
|
||
3.1 Go template (rules/resolve.go.tmpl)
|
||
```go
|
||
{{- $seed := .seed -}}
|
||
{{- $rand := rand.New(rand.NewSource(int64 $seed)) }}
|
||
{{- range $obj := .objects }}
|
||
{{ $obj | sprig.mustToPrettyJson | yamlFromJSON }}
|
||
{{- end }}
|
||
```
|
||
Build helper:
|
||
```bash
|
||
go run ./render.go -seed $KINGDOM_SEED
|
||
```
|
||
|
||
3.2 Jsonnet (resolve.jsonnet)
|
||
```jsonnet
|
||
local seed = std.parseInt(std.extVar('seed'));
|
||
local rng = std.native('seededRand')(seed);
|
||
[ x + {color: rng.choice(['silver','obsidian'])} for x in yaml.load('objects/*.yaml') ]
|
||
```
|
||
|
||
3.3 CUE (resolve.cue)
|
||
```cue
|
||
package kingdom
|
||
|
||
seed: int @tag(seed)
|
||
objects: [ for f in os.Glob("objects/*.yaml") { yaml.Unmarshal(os.ReadFile(f)) & {
|
||
color: *rng.choice(["silver","obsidian"]) | _
|
||
}}]
|
||
```
|
||
|
||
3.4 Dhall (resolve.dhall)
|
||
```dhall
|
||
let seed = 42
|
||
let rng = λ(n → Natural/fold seed …) -- deterministic pseudo RNG
|
||
in ./objects/*.yaml
|
||
: List { id : Text, color : Text }
|
||
|> List/map (λ(o → o // { color = rng.choice ["silver","obsidian"] }))
|
||
```
|
||
|
||
──────────────────────────────────────────────
|
||
4. Validation Harness (language-agnostic)
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
set -e
|
||
SEED=42
|
||
for engine in jinja2 go jsonnet cue dhall; do
|
||
./render-$engine --seed $SEED > out-$engine.yaml
|
||
echo "$engine SHA256: $(sha256sum out-$engine.yaml | cut -d' ' -f1)"
|
||
done
|
||
diff -q out-*.yaml || echo "⚠️ non-deterministic drift"
|
||
```
|
||
|
||
──────────────────────────────────────────────
|
||
5. Decision Heuristics
|
||
|
||
| Situation | Pick |
|
||
|---|---|
|
||
| Stay in Python, need macros & inheritance | Jinja2 |
|
||
| Single static binary, embed templates | Go templates |
|
||
| Want schemas + data validation baked in | CUE |
|
||
| Pure functional, no side effects ever | Dhall |
|
||
| Deep recursion & arithmetic logic | Jsonnet |
|
||
|
||
──────────────────────────────────────────────
|
||
6. One-Line Meta-Moral
|
||
|
||
The crown (YAML) never changes; only the steward (template engine) does—choose the steward whose **leakage surface** matches the **risk budget** of your deterministic kingdom.
|
||
|
||
---
|
||
|
||
# ***The Deterministic Serendipity Guide***
|
||
## Using Jinja2 to Rule a Pure-YAML Kingdom
|
||
*(Private reference v1.0 – for your eyes only)*
|
||
|
||
---
|
||
|
||
### 0. Executive Purpose
|
||
Demonstrate how **Jinja2 becomes the single source of truth** in a closed system where
|
||
- **YAML is the only artifact** (input, intermediate, output)
|
||
- **Objects are pure data** (no code, no imports)
|
||
- **Determinism is absolute** (same seed → same bits)
|
||
- **Serendipity is guaranteed** (drop any new `.yaml` and re-render)
|
||
|
||
---
|
||
|
||
### 1. Kingdom Layout (single directory tree)
|
||
|
||
```
|
||
serendipity/
|
||
├─ kingdom.yml # the royal charter
|
||
├─ objects/ # every object you will ever need
|
||
│ ├─ _manifest.yml # optional index
|
||
│ ├─ unicorn.yaml
|
||
│ ├─ dragon.yaml
|
||
│ └─ …
|
||
└─ rules/
|
||
└─ resolve.j2 # the only executable artifact
|
||
```
|
||
|
||
---
|
||
|
||
### 2. The Royal Charter (`kingdom.yml`)
|
||
|
||
```yaml
|
||
kingdom:
|
||
# determinism knobs
|
||
seed: 42
|
||
rng:
|
||
engine: Random
|
||
module: random
|
||
magic:
|
||
engine: Magic
|
||
module: hashlib
|
||
|
||
# path conventions
|
||
objects: "objects/**/*.yaml" # glob pattern
|
||
rules: "rules/resolve.j2"
|
||
|
||
# post-processing
|
||
exports:
|
||
- format: yaml
|
||
to: stdout
|
||
- format: yaml
|
||
to: out/kingdom_rendered.yaml
|
||
```
|
||
|
||
Nothing outside this file ever changes the deterministic output except the **seed**.
|
||
|
||
---
|
||
|
||
### 3. Object Contract (what your YAML may contain)
|
||
|
||
Every object **MUST** be valid YAML and **MAY** include Jinja2 expressions inside **quoted** scalars only.
|
||
|
||
```yaml
|
||
# objects/unicorn.yaml
|
||
id: unicorn
|
||
color: "{{ rng.choice(['silver','iridescent']) }}"
|
||
blessing: "{{ magic.uuid4() }}"
|
||
```
|
||
|
||
Important:
|
||
- Expressions are **strings** → safe for YAML parsers.
|
||
- No top-level keys named `jinja2` or `__` (reserved for rulebook).
|
||
|
||
---
|
||
|
||
### 4. The Single Rulebook (`rules/resolve.j2`)
|
||
|
||
```jinja2
|
||
{#- 1. Bootstrap deterministic engines -#}
|
||
{% set rng = load_rng(kingdom.rng) %}
|
||
{% set magic = load_magic(kingdom.magic) %}
|
||
|
||
{#- 2. Load and expand every object -#}
|
||
{% set court = [] %}
|
||
{% for path in glob(kingdom.objects) %}
|
||
{% set raw = read_file(path) %}
|
||
{% set yaml = from_yaml(render_str(raw, rng=rng, magic=magic)) %}
|
||
{{ court.append(yaml) or '' }}
|
||
{% endfor %}
|
||
|
||
{#- 3. Emit final deterministic YAML -#}
|
||
{{ court | to_yaml }}
|
||
```
|
||
|
||
Helper filters (`load_rng`, `load_magic`, `glob`, `read_file`, `render_str`) are registered once at startup; **they never change** and are **not serialisable**, so they cannot leak into objects.
|
||
|
||
---
|
||
|
||
### 5. Bootstrap Script (one-time, never edited)
|
||
|
||
```python
|
||
#!/usr/bin/env python3
|
||
# bootstrap.py (kept outside kingdom/)
|
||
import os, yaml, jinja2, glob, hashlib, random
|
||
|
||
# deterministic seed from kingdom.yml
|
||
meta = yaml.safe_load(open('kingdom.yml'))
|
||
random.seed(meta['kingdom']['seed'])
|
||
|
||
# Jinja2 env
|
||
env = jinja2.Environment(
|
||
loader=jinja2.FileSystemLoader('rules'),
|
||
finalize=lambda x: '' if x is None else x,
|
||
)
|
||
env.filters.update({
|
||
'to_yaml': lambda d: yaml.dump(d, sort_keys=True),
|
||
})
|
||
env.globals.update({
|
||
'load_rng': lambda cfg: random,
|
||
'load_magic': lambda cfg: type('Magic', (), {'uuid4': lambda: hashlib.md5(os.urandom(8)).hexdigest()[:8]})(),
|
||
'glob': lambda p: glob.glob(p, recursive=True),
|
||
'read_file': lambda f: open(f).read(),
|
||
'render_str': lambda s, **kw: jinja2.Template(s, undefined=jinja2.StrictUndefined).render(**kw),
|
||
})
|
||
|
||
template = env.get_template('resolve.j2')
|
||
print(env.from_string(template.render(kingdom=meta['kingdom'])).render())
|
||
```
|
||
|
||
Run once → deterministic stream of bytes.
|
||
Commit the stream if you need reproducibility without Python.
|
||
|
||
---
|
||
|
||
### 6. Deep Powers Inside Jinja2 (exploited here)
|
||
|
||
| Power | Usage in Kingdom |
|
||
|---|---|
|
||
| **StrictUndefined** | Every undefined var raises → no silent typos |
|
||
| **Custom globals/filters** | Inject rng, magic, filesystem helpers |
|
||
| **Macro imports** | `{% from 'macros.j2' import mutate %}` inside object templates |
|
||
| **Whitespace control** | `{%- … -%}` keeps emitted YAML clean |
|
||
| **Recursive rendering** | Object YAML contains Jinja2 that is re-parsed |
|
||
| **Immutable context** | No `{% set %}` leaks between object renders |
|
||
|
||
---
|
||
|
||
### 7. Serendipity Workflow (non-breaking)
|
||
|
||
1. Drop `objects/new_artifact.yaml`
|
||
2. Re-run `python bootstrap.py`
|
||
3. Diff the two deterministic outputs → only deltas from the new object appear.
|
||
4. Version-control the diff, not the code.
|
||
|
||
---
|
||
|
||
### 8. Security & Reproducibility Checklist
|
||
|
||
- **No Python code inside YAML** → safe for linters like yamllint.
|
||
- **Seed is single source of entropy** → store it in a repo tag for audit.
|
||
- **Read-only filesystem** → templates and bootstrap script are immutable at runtime.
|
||
- **One-way data flow**: YAML → Jinja2 → expanded YAML → stdout/file.
|
||
- **No external network calls** → reproducible in air-gapped CI.
|
||
|
||
---
|
||
|
||
### 9. Single-Command Invocation (CI / Makefile)
|
||
|
||
```makefile
|
||
render:
|
||
@python bootstrap.py > kingdom_rendered.yaml
|
||
@echo "SHA256: $$(sha256sum kingdom_rendered.yaml)"
|
||
```
|
||
|
||
---
|
||
|
||
### 10. Closing Axiom
|
||
|
||
> In this kingdom the **template is the constitution**,
|
||
> the seed is the crown jewel,
|
||
> and every `.yaml` is a citizen whose fate is **deterministically serendipitous**. |