the_information_nexus/Jinja2_guide.md at main

Files

medusa 5a8be80f43 Update tech_docs/Jinja2_guide.md

2025-08-05 23:39:33 -05:00

17 KiB

Raw Permalink Blame History

Below is a “cheat-sheet” style upgrade to the playbook you already have.
It is organized by pain points I see teams hit once they move past the first 5–10 devices.
Everything is copy-paste ready (CLI one-liners, tiny Python/Go helpers, Ansible snippets).
If you only have 15 min right now, jump to the “Tonight” section at the end.

────────────────────────────────────────────────────────
1. YAML → Jinja2 → Device in <10 s
────────────────────────────────────────────────────────
Problem: you still open an editor to change a variable file.  
Fix: keep YAML inside Git and treat it as the *single source of truth*.

A. Git-based variable loader (avoids “where did I save that file?”)
```bash
# vars live in ./inventory/host_vars/$hostname.yml
git ls-files -z '*.yml' | xargs -0 yamllint   # fast lint

B. One-shot render + diff

#!/usr/bin/env bash
host=$1
jinja2 templates/ios_base.j2 \
       inventory/host_vars/${host}.yml \
       --format yaml \
       -o /tmp/${host}.cfg
colordiff -u <(ssh $host show run) /tmp/${host}.cfg | less -R

C. Atomic push with Ansible “deploy_if_changed”

- name: Deploy only if rendered differs
  ios_config:
    src: "/tmp/{{ inventory_hostname }}.cfg"
  vars:
    ansible_check_mode: false
  when: >
    lookup('pipe', 'diff -q /tmp/{{ inventory_hostname }}.cfg
                     <(ssh {{ inventory_hostname }} show run)')
    is not search('identical')

──────────────────────────────────────────────────────── 2. Multi-vendor, one variable set ──────────────────────────────────────────────────────── Trick: store capability flags instead of vendor syntax.

vars/host_vars/leaf01.yml

interfaces:
  - name: Ethernet1/1
    port_type: access
    vlan: 200
    poe: true
    mtu: 9214

templates/shared/interface_capabilities.j2

{# Works on IOS-XE, Junos, EOS #}
{% set vendor = hostvars[inventory_hostname].vendor %}
{% for i in interfaces %}
interface {{ i.name | re_replace('^Eth','') if vendor=='arista' else
             i.name | re_replace('^Eth','ge-') if vendor=='juniper' else
             i.name }}
{% if i.port_type == 'access' %}
 switchport mode access
 switchport access vlan {{ i.vlan }}
{% endif %}
{% if i.poe and (vendor == 'cisco' or vendor == 'arista') %}
 power inline never
{% endif %}
 mtu {{ i.mtu }}
!
{% endfor %}

──────────────────────────────────────────────────────── 3. NetBox as a live inventory ──────────────────────────────────────────────────────── Inventory plugin (ships with Ansible ≥2.10) in ansible.cfg:

[inventory]
enable_plugins = netbox.netbox.nb_inventory

Group by site + role:

# group_vars/netbox_site_edge.yml
ntp_servers: ["10.0.0.10", "10.0.0.11"]

Now zero extra YAML for new sites.

──────────────────────────────────────────────────────── 4. Rego in CI to block bad configs ──────────────────────────────────────────────────────── .gitlab-ci.yml (runs in <5 s):

validate:
  image: openpolicyagent/opa:0.59
  script:
    - opa test policies/
    - >
      find . -name "*.cfg" -print0 |
      xargs -0 -I{} opa eval --fail-defined --format pretty
      --data policies/network.rego --input {} "data.network.deny[_]"

Example policy (policies/network.rego):

package network
deny[msg] {
  input.kind == "interface"
  input.mtu > 9216
  msg := sprintf("MTU %d too high on %s", [input.mtu, input.name])
}

──────────────────────────────────────────────────────── 5. Secrets without Ansible Vault pain ──────────────────────────────────────────────────────── Use sops + age (age is easier than GPG).

# encrypt vars/secrets.yml
sops --age $(cat ~/.age/key.txt | grep public | cut -d' ' -f4) \
     --encrypt vars/secrets.yml > vars/secrets.enc.yml

Ansible automatically decrypts when ansible_sops_enabled=true.

──────────────────────────────────────────────────────── 6. Tonight → Next week → Go Pro (15 min each) ──────────────────────────────────────────────────────── Tonight (literally 15 min):

pipx install jinja2-cli yq
git init network-auto && cd network-auto
mkdir templates vars
cat > vars/lab.yml <<EOF
vlans:
  - id: 100
    name: STAFF
EOF
cat > templates/ios_vlan.j2 <<'EOF'
{% for v in vlans %}
vlan {{ v.id }}
 name {{ v.name }}
{% endfor %}
EOF
jinja2 templates/ios_vlan.j2 vars/lab.yml

You now have version-controlled VLANs.

Next week:

Replace static vars/lab.yml with NetBox inventory plugin.
Add a single playbook that runs against “site:lab” and pushes the VLANs.

Go Pro:

Add Rego policy to reject VLAN 1 or duplicate names.
GitLab CI job auto-pushes when MR merges (use Ansible’s ios_config with backup: yes so you can rollback).

──────────────────────────────────────────────────────── Quick reference card ──────────────────────────────────────────────────────── Render: jinja2 tpl.j2 vars.yml --format yaml
Lint: yamllint vars/
Test: opa test policies/
Encrypt: sops -e vars/secrets.yml
Diff: colordiff -u live.cfg new.cfg

Copy the snippets above into your repo and you’re already ahead of most teams.

---

This is **exactly** the kind of tactical, network-focused Jinja2 guide I wish I had when I started automating configs. Let’s sharpen it further with battle-tested patterns and CLI-ready examples. Here’s your playbook:

---

### **1. Immediate Wins: Stop Hand-Editing Configs**
#### **A. Interface Templating (Multi-Vendor)**
**Problem:** Vendor-specific syntax for the same logical interface.  
**Solution:** One YAML → Cisco/Juniper/Arista templates.

**`vars/interface.yml`**
```yaml
interfaces:
  - name: Eth1/1
    description: "Server Farm Uplink"
    mode: trunk
    vlans: [100, 200]
    mtu: 9214

templates/cisco_interface.j2

interface {{ interface.name }}
 description {{ interface.description }}
 {% if interface.mode == 'trunk' %}
 switchport mode trunk
 switchport trunk allowed vlan {{ interface.vlans | join(',') }}
 {% endif %}
 mtu {{ interface.mtu }}

templates/juniper_interface.j2

interfaces {
    {{ interface.name | replace('Eth','ge-') }} {
        description "{{ interface.description }}";
        {% if interface.mode == 'trunk' %}
        unit 0 {
            family ethernet-switching {
                vlan members [ {{ interface.vlans | join(' ') }} ];
            }
        }
        {% endif %}
        mtu {{ interface.mtu }};
    }
}

Render Both:

# Cisco
jinja2 templates/cisco_interface.j2 vars/interface.yml
# Juniper
jinja2 templates/juniper_interface.j2 vars/interface.yml

B. BGP Config Generation (With Error-Prone Logic)

Problem: Complex BGP configs with neighbor policies.
Solution: Template + YAML with validation.

vars/bgp.yml

bgp:
  asn: 65001
  neighbors:
    - ip: 10.0.0.2
      remote_as: 65002
      policies: [ "PREVENT_LEAK" ]
    - ip: 192.168.1.1
      remote_as: 65123
      policies: [ "CUSTOMER_ROUTES" ]

templates/bgp.j2

router bgp {{ bgp.asn }}
  {% for neighbor in bgp.neighbors %}
  neighbor {{ neighbor.ip }} remote-as {{ neighbor.remote_as }}
  {% if "PREVENT_LEAK" in neighbor.policies %}
  neighbor {{ neighbor.ip }} route-map BLOCK_DEFAULT in
  {% endif %}
  {% endfor %}

Key Trick: Use | selectattr to filter neighbors:

{% for neighbor in bgp.neighbors | selectattr("policies", "contains", "CUSTOMER_ROUTES") %}
...customer-specific config...
{% endfor %}

2. Advanced: Network-as-Code Patterns

A. Generate Device-Specific Configs from NetBox API

Problem: NetBox has device data, but CLI configs are manual.
Solution: Fetch NetBox data → Jinja2.

fetch_netbox_data.py

import requests
import json

devices = requests.get("https://netbox/api/dcim/devices/").json()
with open('vars/netbox_devices.yml', 'w') as f:
    json.dump(devices, f)

templates/netbox_cisco.j2

hostname {{ device.name }}
{% for iface in device.interfaces %}
interface {{ iface.name }}
 description {{ iface.description }}
{% endfor %}

Render All Devices:

python fetch_netbox_data.py
jinja2 templates/netbox_cisco.j2 vars/netbox_devices.yml

B. Auto-Generate Port-Channel Configs (LACP)

Problem: Port-channel members change frequently.
Solution: Dynamic YAML + template.

vars/portchannel.yml

portchannel:
  id: 10
  members: [ "Eth1/1", "Eth1/2" ]
  mode: "active"
  vlan: 100

templates/portchannel.j2

interface Port-channel{{ portchannel.id }}
  channel-group {{ portchannel.id }} mode {{ portchannel.mode }}
  switchport access vlan {{ portchannel.vlan }}

{% for member in portchannel.members %}
interface {{ member }}
  channel-group {{ portchannel.id }} mode {{ portchannel.mode }}
{% endfor %}

3. Pro Tooling: Network-Specific Tricks

A. Ansible + Jinja2 for Zero-Touch Deployment

ansible_playbook.yml

- name: Push Interface Configs
  hosts: switches
  tasks:
    - name: Generate Config
      template:
        src: templates/cisco_interface.j2
        dest: /tmp/{{ inventory_hostname }}.cfg
    - name: Deploy Config
      cisco.ios.ios_config:
        src: /tmp/{{ inventory_hostname }}.cfg

B. Validate Configs Before Deployment

# Check for missing variables
jinja2 --format vars:yaml templates/interface.j2 vars/device.yml

# Dry-run with Ansible
ansible-playbook playbook.yml --check

C. Secret Management (Ansible Vault)

# Encrypted vars.yml
ansible-vault encrypt vars/secrets.yml

Template:

username {{ vaulted_username }}
password {{ vaulted_password }}

4. Your Battle Plan

Start Tonight:
- Pick 1 repetitive config (VLANs/ACLs) → template it.
- Render with jinja2-cli locally.
Next Week:
- Integrate with NetBox/Python to auto-generate configs.
- Use Ansible to push to devices.
Go Pro:
- Build a GitLab pipeline that auto-generates configs on NetBox changes.

Example Repo Structure:

network-automation/
├── templates/       # .j2 files
├── vars/            # .yml device vars
├── scripts/         # Python/Ansible helpers
└── outputs/         # Rendered configs

Key Takeaway: Jinja2 turns network configs from manual CLI work into version-controlled code. The more you template now, the more time you save later.

Great pivot! Rego (Open Policy Agent’s policy language) and Jinja2 serve different but complementary roles in infrastructure automation. Let’s break down how they compare, overlap, and where you’d use each—with gritty examples tailored to infrastructure/security use cases.

1. Quick Comparison: Jinja2 vs. Rego

Feature	Jinja2	Rego
Primary Use	Templating (generate configs/docs)	Policy enforcement (validate inputs)
Input	YAML/JSON/CSV	JSON/YAML (often from APIs)
Output	Rendered text (configs, CLI commands)	Policy decisions (allow/deny + detailed reasons)
Context	"Generate this network config"	"Is this network config compliant?"
Key Strength	Flexibility in text generation	Logic-based evaluation with auditing

2. Where Jinja2 and Rego Overlap

Both operate on structured data (YAML/JSON), but solve different problems in the pipeline:

Example Workflow: Network Change Automation

Jinja2: Generates a candidate BGP config from YAML variables.

router bgp {{ bgp.asn }}
  neighbor {{ neighbor.ip }} remote-as {{ neighbor.asn }}

Rego: Validates the generated config before deployment.

# Prevent BGP peers in untrusted ASNs
deny[msg] {
  input.kind == "bgp_config"
  not input.neighbor.asn in trusted_asns
  msg := sprintf("BGP peer ASN %v is not trusted", [input.neighbor.asn])
}

3. Rego Use Cases (Where It Shines Over Jinja2)

A. Pre-Deployment Validation

Problem: Ensure Jinja2-generated configs meet security/compliance rules.
Rego Policy (policies/networking.rego):

package networking

# Deny firewall rules that allow SSH from the internet
deny[msg] {
    input.kind == "firewall_rule"
    input.action == "allow"
    input.port == 22
    input.source == "0.0.0.0/0"
    msg := "SSH must not be open to the internet!"
}

# Require VLAN descriptions for auditability
deny[msg] {
    input.kind == "vlan_config"
    not input.description
    msg := "VLANs must have a description"
}

How to Use:

# Validate a Jinja2-rendered config against Rego
opa eval --data policies/networking.rego --input rendered_config.json "data.networking.deny"

B. Multi-Cloud Policy Enforcement

Problem: Enforce tagging standards across AWS/Azure/GCP.
Rego Policy (policies/cloud.rego):

package cloud

# Require 'CostCenter' tag on all resources
deny[msg] {
    input.resource.tags["CostCenter"] == ""
    msg := "All resources must have a CostCenter tag"
}

# Block public S3 buckets
deny[msg] {
    input.kind == "aws_s3_bucket"
    input.acl == "public-read"
    msg := "Public S3 buckets are prohibited"
}

Input (JSON from Terraform/Cloud API):

{
  "kind": "aws_s3_bucket",
  "acl": "public-read",
  "tags": { "CostCenter": "" }
}

Output:

{
  "deny": [
    "All resources must have a CostCenter tag",
    "Public S3 buckets are prohibited"
  ]
}

C. Network Topology Auditing

Problem: Validate that Jinja2-generated network designs follow best practices.
Rego Policy (policies/topology.rego):

package topology

# Core switches must have redundant links
deny[msg] {
    input.device.role == "core"
    count(input.device.links) < 2
    msg := "Core devices must have redundant links"
}

# VLAN 1 should not be used (security best practice)
deny[msg] {
    input.vlan.id == 1
    msg := "VLAN 1 is prohibited for security reasons"
}

4. When to Use Jinja2 vs. Rego

Scenario	Tool	Why
Generating Cisco/Juniper configs	Jinja2	Best at text templating.
Validating configs pre-deploy	Rego	Logic-based policy checks.
Creating API payloads	Jinja2	Structure dynamic JSON/YAML.
Auditing existing infra	Rego	Query live state for violations.
Documenting network designs	Jinja2	Render Markdown/HTML.
Enforcing security rules	Rego	Evaluate against compliance rules.

5. Pro Tips for Rego in Infrastructure

Test Policies Early: Use opa test to unit test your Rego rules.
```
opa test policies/ -v
```

Integrate with CI/CD:

# GitLab CI example
validate_policy:
  image: openpolicyagent/opa
  script:
    - opa eval --data policies/ --input $CHANGES "data.deny"

Debug with trace:

# Add to your policy to see evaluation steps
trace(sprintf("Input: %v", [input]))

6. Example End-to-End Flow

Jinja2 Generates Config:

jinja2 bgp_template.j2 vars.yml > bgp_config.json

Rego Validates It:

opa eval --data policies/ --input bgp_config.json "data.networking.deny"

Only deploy if Rego returns no denials.

7. Key Takeaway

Jinja2: Your "build" tool (creates configs/scripts/docs).
Rego: Your "guardrail" tool (ensures compliance/safety).

Together, they form a powerful pipeline:
Jinja2 (generate) → Rego (validate) → Ansible/Terraform (deploy)

Want a deep dive on a specific Rego use case (e.g., network security policies or cloud governance)? I can break it down with real code.

17 KiB Raw Permalink Blame History Unescape Escape