From e7b08d07ccd328ba52d2af5da98725b50b9ea31c Mon Sep 17 00:00:00 2001 From: medusa Date: Sat, 2 Aug 2025 14:55:27 -0500 Subject: [PATCH] Update tech_docs/networking/iac_github_project.md --- tech_docs/networking/iac_github_project.md | 352 +++++++++++++++++++++ 1 file changed, 352 insertions(+) diff --git a/tech_docs/networking/iac_github_project.md b/tech_docs/networking/iac_github_project.md index e78f9ea..1418959 100644 --- a/tech_docs/networking/iac_github_project.md +++ b/tech_docs/networking/iac_github_project.md @@ -76,6 +76,358 @@ The current organization of templates by file path (`core_settings/00_licensing. --- +Of course. Let's get meta. You're thinking about the core data structure itself—the YAML object. This is a great place to focus, as it's the "single source of truth" and the most critical component of the entire system. + +The core idea is that the YAML object is a nested, hierarchical data structure that models the entire state of a device. It's not just a flat list of variables; it's an object with keys that can contain other objects or lists. This nesting is what makes the system powerful, scalable, and manageable. + +### The YAML Object as a "Meta-Schema" + +Think of the YAML object not just as a data file, but as a "meta-schema" for your network. Each top-level key (`hostname`, `system`, `aaa`, `interfaces`, `routing`) represents a major configuration block. The values under these keys are either: + +1. **Simple scalars:** A single value, like `hostname: "SITE01-ASR01"`. +2. **Lists of objects:** A sequence of similar configuration blocks, like the `tacacs_servers` list. Each item in the list is its own object with keys like `host` and `key`. +3. **Nested objects:** A hierarchical structure where a key holds other keys, like the `routing` object containing `ospf` and `bgp`. + +The key insight is that this nested YAML structure is an *abstraction* of the Cisco CLI. It's a declarative representation of the desired state, which is far more human-readable and manageable than the procedural, line-by-line CLI. + +### The Overlap and How Nesting Solves It + +You mentioned "a lot of overlap etc." in the original prompt. The nested structure is designed to *eliminate* this kind of overlap and redundancy. + + * **Overlap in the CLI:** In the CLI, you might have to repeat a VRF name, an interface name, or an OSPF process ID in multiple places. + ``` + interface Tunnel1 + vrf forwarding DMVPN + ! + ip route vrf DMVPN 0.0.0.0 0.0.0.0 10.1.1.1 + ! + router bgp 65400 + vrf DMVPN + ``` + * **No Overlap in YAML:** In the YAML model, you define the VRF once, and then you reference it. The nesting itself organizes the configuration by functional group, so you don't have to think about where to place a `vrf forwarding` command. The `vrf` is an attribute of the `interface` object, which is an attribute of the `interfaces` object. + +Here's a conceptual view of how the nesting relates to the configuration: + +``` +# Top-level object for the entire device +device: + # This section configures system-wide settings. + system: + hostname: "SITE01-ASR01" + license_level: "network-advantage" + ... + # This section configures all interfaces. It's a list of objects. + interfaces: + - name: "GigabitEthernet0/0/0" + profile: "access_port" + desc: "Link to distribution switch" + # Interface-specific settings are nested here + ip: + address: "10.0.0.1/24" + vrf: "DMVPN" + ... + # This section configures all routing protocols. It's a nested object. + routing: + ospf: + # OSPF is a list of objects, because you can have multiple processes. + - pid: 1 + rid: "10.255.255.1" + vrf: "DMVPN" + networks: + - prefix: "10.0.0.0" + wc: "0.0.0.255" + area: 0 + bgp: + # BGP is a single object, as there's only one BGP process per device. + asn: 65400 + rid: "10.255.255.1" + neighbors: + - ip: "10.1.1.1" + as: 65400 + ... +``` + +### Key Recommendations for Thinking about the YAML Object + +1. **Embrace the Tree Structure:** Always think of the YAML data as a tree. The root is the device, and the branches are the major configuration areas (`system`, `interfaces`, `routing`). The leaves are the final configuration values (`hostname`, `ip`, `asn`). This mental model is crucial for building and navigating the schema. + +2. **Define a Canonical Schema:** The `schema.yml` file is the most important part of this entire system. It formalizes the nesting and structure. For example, it should define that `interfaces` must be a list, and each item in that list must be an object containing keys like `name` and `profile`. This is what makes the system robust and prevents errors. + +3. **Leverage Defaults for Consistency:** The `defaults.yml` file is another critical layer. It allows you to specify sane, global values that most devices will inherit. This keeps the per-device YAML files (`site-01.yml`) small and focused *only on the differences*. You only need to define a key in `site-01.yml` if you want to override the default. + +4. **The YAML is the API:** The YAML file is the "API" for your network configuration. Any engineer should be able to look at the `schema.yml` and a `site-01.yml` file and understand how to configure a device without ever needing to touch the Jinja2 templates. The templates are the implementation detail; the YAML is the policy. + +By adopting this perspective, you're not just organizing information; you're building a structured, declarative language for your network. The Jinja2 templates are just the compiler that translates this language into executable device configurations. + +--- + +Of course. Let's reverse-engineer the provided system to understand how all the pieces fit together to form a working solution. This will involve starting with the final output (a rendered configuration file) and tracing its creation back through the process. + +### The Goal: A Working Configuration File + +The ultimate goal of this entire system is to produce a clean, correct, and predictable configuration file (e.g., `site-01.cfg`) for a specific network device. This file is what you will push to the device. + +``` +# Final Output: site-01.cfg +hostname SITE01-ASR01 +service password-encryption +... +ip domain-name example.net +... +aaa group server tacacs+ GTAC + server-private 10.1.1.1 key SECRET1 +... +interface GigabitEthernet0/0/0 + description Link to Core Switch + switchport mode trunk +... +router ospf 1 + router-id 10.255.255.1 +... +``` + +### Step 1: The Render Pipeline + +The final configuration file is created by the rendering pipeline. The key command is: + +```bash +jinja2 templates/compile.j2 data/site-01.yml > site-01.cfg +``` + +This single command tells the Jinja2 rendering engine three things: + +1. **Which template to use?** `templates/compile.j2` +2. **Which data to use?** `data/site-01.yml` +3. **What to do with the output?** Redirect it to `site-01.cfg` + +This is the core "build" step. It takes the "policy" (templates) and the "data" (YAML) and combines them into the final output. + +### Step 2: The Data (`data/site-01.yml`) + +Where does the data come from? The `site-01.yml` file is the "Single Source of Truth." It's a structured representation of the configuration for a single device. + +```yaml +# The data/site-01.yml file +hostname: "SITE01-ASR01" +system: + license_level: "network-advantage" + domain_name: "example.net" + ... +interfaces: + - name: "GigabitEthernet0/0/0" + profile: "trunk_port" + desc: "Link to Core Switch" + ... +routing: + ospf: + - pid: 1 + rid: "10.255.255.1" + ... +``` + +This file is created by a human engineer, but it's not a free-form text file. It is governed by a strict schema. Before it's even rendered, it must pass a validation step: + +```bash +jsonschema -i data/site-01.yml schema.yml +``` + +This step ensures that the engineer hasn't made a mistake like misspelling a key, using an incorrect data type (e.g., a string instead of an integer), or omitting a mandatory field. The `schema.yml` file is the contract that defines the valid structure of the data. + +An even more powerful part of the data flow is the `defaults.yml` file. Each `site-01.yml` file is not just its own file—it's a merge of the global defaults and its own specific values. + +```yaml +# The data is actually a deep merge +data = defaults.yml | combine(site-01.yml, recursive=True) +``` + +This means that if you define a default NTP server in `defaults.yml`, you don't need to specify it in `site-01.yml` unless you need to override it for that specific device. This is a massive efficiency gain. + +### Step 3: The Templates (`templates/compile.j2` and its components) + +The rendering engine needs to know how to transform the YAML data into CLI syntax. This is the job of the Jinja2 templates. + +The `templates/compile.j2` file is the "master template." It's a simple text file that orchestrates the inclusion of all the other component templates in a deterministic order. + +```jinja +{# templates/compile.j2 #} +{% include 'config_components/core_settings/00_licensing.j2' %} +{% include 'config_components/core_settings/10_system_settings.j2' %} +{% include 'config_components/core_settings/20_aaa.j2' %} +{% include 'config_components/network_services/30_vlans.j2' %} +... +``` + +Why not just a `cat` command? Using `include` is better because it ensures that the Jinja2 context (the data from `site-01.yml`) is available to every component template. A simple `cat` wouldn't pass the data to each template individually. + +Each of the included component templates (e.g., `00_licensing.j2`, `10_system_settings.j2`) contains the actual CLI syntax with variables. + +```jinja +{# config_components/core_settings/10_system_settings.j2 #} +hostname {{ hostname }} +service password-encryption +ip domain-name {{ domain_name }} +{% for ns in name_servers %} +ip name-server {{ ns }} +{% endfor %} +``` + +When Jinja2 processes this file, it looks for the keys `hostname`, `domain_name`, and `name_servers` in the data object it was given (the merged `site-01.yml` data) and substitutes the values. + +### Step 4: The CI/CD Guardrails + +The final piece is the automation and quality control. This is where the whole system is protected from human error. + +1. **Engineer makes a change:** An engineer updates `data/site-01.yml` to change the hostname. +2. **Pre-commit hooks:** Before the change can be committed to Git, hooks automatically run `yamllint` and `jsonschema`. If the YAML is invalid or doesn't match the schema, the commit is rejected. +3. **Pull Request:** The engineer submits a pull request with the change. +4. **CI/CD Pipeline:** An automated pipeline is triggered. It performs the same validation checks. +5. **Render and Diff:** The pipeline renders the new `site-01.cfg` and compares it to the previous version. The resulting diff is shown in the pull request. This allows for a final human review of the *generated configuration* before it's deployed. +6. **Merge and Deploy:** Once approved and merged, the pipeline can automatically push the new configuration to the device using a tool like Nornir or Napalm. + +### The "Reverse-Engineered" Flow + +In summary, the entire solution works in this reverse-engineered order: + +1. A **desired state** is defined in a structured **YAML** data file (`site-01.yml`). +2. This data is validated against a formal **JSON Schema** (`schema.yml`). +3. A **Jinja2 "compiler"** (`jinja2-cli`) takes the YAML data as input. +4. It feeds this data into a master template (`compile.j2`) which includes a series of smaller **component templates** (`00_licensing.j2`, etc.). +5. Each component template contains the **pseudo-code CLI syntax**, which uses the variables from the YAML. +6. The templates are rendered, the variables are substituted with the YAML data, and the final **device configuration file** (`site-01.cfg`) is produced. +7. This final file is then pushed to the device via a **CI/CD pipeline**, which first performs a `diff` to ensure the changes are intentional. + +This entire process is "reverse-engineered" from the typical network management flow. Instead of starting with a desired configuration and hand-writing the CLI, you start with a desired state in a data file, and the system generates the CLI for you. This is the essence of a modern, declarative network automation solution. + +--- + +This is an excellent meta-question. You're hitting on the core philosophy behind modern, scalable IT. The YAML object isn't just a list of settings; it is the embodiment of several key, and often-hyped, principles of infrastructure management. + +Let's break down how this single YAML object represents all those buzzwords. + +### 1. Declarative Configuration + +This is perhaps the most fundamental principle. + +* **What it is:** Instead of telling the system *how* to do something (imperative), you tell it *what* the final state should look like (declarative). +* **How the YAML embodies it:** The YAML file is the "what." It doesn't contain a single CLI command like `configure terminal` or `router ospf 1`. It simply states the desired state: `routing.ospf.pid: 1`. The Jinja2 templates and the rendering engine are the "how"—they take that desired state and translate it into the procedural commands needed to achieve it. This separation is key. You don't care about the command sequence, only the end result. + +### 2. Single Source of Truth (SSOT) + +This is about consistency and reliability. + +* **What it is:** The idea that all information for a given entity (in this case, a network device) is stored in one, and only one, place. This eliminates the risk of conflicting data and "configuration drift." +* **How the YAML embodies it:** The `data/site-01.yml` file is the SSOT for that specific ASR-1002. All the facts about that device—its hostname, its IP addresses, its OSPF router ID—are contained within that single, structured file. If you need to know anything about the device's configuration, you look at this file. If you want to change something, you change it here. No more hunting through a dozen different text files, spreadsheets, or wiki pages. + +### 3. Infrastructure as Code (IaC) + +This is a broad principle that applies software development practices to infrastructure management. + +* **What it is:** Treating your infrastructure configuration files like you would application source code. This means using version control, peer review, and automated testing. +* **How the YAML embodies it:** The entire system is built on IaC principles. + * **Version Control:** The YAML files and templates live in a Git repository. Every change is a commit with a history, and you can roll back to any previous state. + * **Peer Review:** New configuration changes are submitted via pull requests, where another engineer can review the changes to the YAML and the resulting CLI `diff` before it's merged. + * **Automated Testing:** The CI/CD pipeline validates the YAML against the schema (`jsonschema`) and performs a dry-run render. This ensures the proposed change is valid *before* it ever touches a device. + +### 4. Data Modeling and Schema Validation + +This is about structure, safety, and consistency. + +* **What it is:** The process of defining the structure, data types, and constraints for your configuration data. +* **How the YAML embodies it:** The YAML is not a free-form text file; it is a structured data object. The `schema.yml` file is the formal definition of this structure. It dictates that `hostname` must be a string, `ospf.pid` must be an integer, and `interfaces` must be a list of objects, each containing a `name` and a `profile`. This schema is a powerful guardrail. It prevents invalid configurations, ensures consistency across all devices, and makes the system self-documenting. + +### 5. Modularity and Reusability + +This is about efficiency and scalability. + +* **What it is:** Breaking down complex configurations into smaller, reusable components. +* **How the YAML embodies it:** + * **Templates:** The Jinja2 templates are highly modular. Instead of a single, monolithic template, the configuration is built from dozens of smaller, single-purpose templates (e.g., `20_aaa.j2`, `access_port.j2`). This makes the system easier to maintain, as you only need to change one file to update the AAA configuration for all devices. + * **Data:** The `defaults.yml` file and the `port_profiles` are perfect examples of reusability on the data side. You define common values and profiles once, then reuse them across all your device configurations. + +In essence, the YAML file is a **declarative data model** that serves as the **single source of truth** for your device configuration. This data model is version-controlled and validated as part of an **Infrastructure as Code** workflow, which uses **modular templates** to render the final configuration. + +The entire system is a robust, end-to-end implementation of all these concepts, designed to move network configuration from a manual, error-prone process to an automated, safe, and scalable one. + +--- + +Based on the information provided, here are recommendations for organizing the content, addressing the overlap, and making it more structured and accessible. + +The provided text contains three distinct types of information that are currently intermingled: + +1. **A "how-to" guide or best-practice checklist.** This is the first section, "OPINIONATED, PRACTICAL GUIDE," which outlines a workflow and philosophy for managing network device configurations. +2. **A comprehensive list of configuration knobs.** This is the second section, "sanitized, pseudocode-style" view, which acts as a reference for all possible configuration options on an ASR-1002 DMVPN hub. +3. **A set of Jinja2 template examples.** This is the third section, "minimal, reusable skeleton," which provides concrete, copy-and-pasteable template files that demonstrate the principles from the first section. + +The overlap is in how these three sections relate to each other: the guide (1) describes a process that uses the templates (3) and the configuration knobs (2) as its building blocks. + +### Recommended Organization + +A logical structure would separate these three components into distinct, clearly labeled sections. This would make it easier for a user to find what they need, whether they are learning the philosophy, looking up a specific configuration command, or grabbing a template to get started. + +I recommend the following structure: + +--- + +### **Section 1: The "Golden" Configuration Management Workflow** + +This section should be the user-facing guide. It should be presented as a clear, numbered list of principles and steps, as it currently is. The title "OPINIONATED, PRACTICAL GUIDE" is good and should be kept. + +To improve readability and prevent confusion, explicitly state at the beginning that this section is the *recommended process*, and the following sections are the *reference materials* for that process. + +The "One-Sentence Rule" should be a prominent, highlighted principle at the top. The rest of the numbered items (0-10) are well-organized and should remain as a checklist. + +**Recommendations for this section:** + +* **Highlight key takeaways:** Use bolding or other formatting to emphasize crucial concepts like "Policy (templates) never changes," "One YAML file per box," and "If it’s not in YAML, it does **not** exist." +* **Clarify the purpose of `compile.j2`:** The mention of `compile.j2` is a critical part of the pipeline. It would be clearer to include a brief, commented example of its contents (e.g., `{% include 'core_settings/00_licensing.j2' %} \n {% include 'core_settings/10_system_settings.j2' %}`) directly in the text to show how it orchestrates the `find...sort...cat` process. This adds clarity to step 4. +* **Move "Usage (single command)" to the workflow:** The single-command example is part of the rendering pipeline. It's currently at the end of the third section. It makes more sense to place this command as the final step in the "Render Pipeline" part of the guide (Step 4), as it's the practical application of that step. + +--- + +### **Section 2: ASR-1002 Configuration Reference (Functional Knobs)** + +This section should be the comprehensive, non-opinionated reference. It is currently titled "sanitized, pseudocode-style view of every functional knob..." which is accurate but a bit verbose. A more direct title would be better, such as "**ASR-1002 DMVPN Hub Configuration Reference**" or "**Functional Knobs for ASR-1002 DMVPN**". + +This section is already well-organized into logical categories (GLOBAL/SYSTEM, AAA, VRFs, etc.). This structure is excellent and should be preserved. + +**Recommendations for this section:** + +* **Standardize the format:** Each bullet point is a configuration option. Using a consistent format like `knob_name → data_type` is good. This should be maintained throughout. +* **Explain the `→` notation:** Briefly explain that this section is a reference, and the `→ string`, `→ list`, etc., are placeholders for the data types that would go into the YAML file. +* **Explicitly state its purpose:** Add a short introductory sentence that says this is a complete list of configuration options, separate from the implementation details, and serves as a blueprint for the data model. + +--- + +### **Section 3: Reusable Jinja2 Templates (Code Snippets)** + +This section should contain the actual code. The current title, "minimal, reusable skeleton," is good. A more descriptive title could be "**Jinja2 Template Examples for ASR-1002 DMVPN Hubs**". + +The current organization of templates by file path (`core_settings/00_licensing.j2`, etc.) is perfect as it directly mirrors the recommended folder layout from the first section. This makes the connection between the guide and the code examples explicit. + +**Recommendations for this section:** + +* **Provide a full example:** The `interfaces/60_interface_assignments.j2` template is a great example of dynamic inclusion. To make the entire system more concrete, consider adding a full, *annotated* `site-01.yml` file that references these templates. This would show the "data" part of the "policy vs. data" rule in action and tie all three sections together perfectly. +* **Keep the comments:** The Jinja2 comments (`{# ... #}`) are very helpful for explaining the purpose of each template. These should be preserved. +* **Move the usage command:** As mentioned earlier, the `find ...` command should be moved to the first section, as it's a procedural step rather than a code example. + +### Summary of Recommended Structure + +1. **Title:** "OPINIONATED, PRACTICAL GUIDE: How to Ship the Right Config to Every ASR-1002 DMVPN Hub—Forever" + * **Sub-title:** A clear statement that this section outlines the recommended workflow, with subsequent sections providing the reference material and template code. + * **Content:** The 10-point checklist (One-Sentence Rule to Single Source of Truth). + * **Actionable Code:** The `find...sort...cat` command should be included in the "Render Pipeline" step. + +2. **Title:** "ASR-1002 DMVPN Hub Configuration Reference" + * **Sub-title:** A statement that this section is a comprehensive, non-opinionated list of configuration options. + * **Content:** The categorized list of functional knobs (GLOBAL, AAA, VRFs, etc.). + +3. **Title:** "Jinja2 Template Examples" + * **Sub-title:** A statement that these are reusable, copy-pasteable template files that implement the principles from the first section. + * **Content:** The code snippets, organized by file path. + * **Additional Content:** A full, example `data/site-01.yml` file. + +--- + Below is a vendor-agnostic, scalable-template review written in “meta-config” form. It is intentionally abstract (no literal values, no vendor CLI) so it can be mechanically translated to any NOS or rendered by an automation pipeline.