the_information_nexus/tech_docs/linux/vxlan.md

Certainly! Setting up a bridge-to-bridge connection across different hosts using tunneling technologies like VXLAN is an advanced networking topic that's particularly useful in modern data centers and cloud environments. Here, we'll cover this topic in detail, focusing on VXLAN as a popular choice due to its scalability, flexibility, and support across various networking hardware and software.

Understanding VXLAN

VXLAN (Virtual Extensible LAN) is a network overlay technology designed to provide the same services as VLAN but with greater extensibility and flexibility. It encapsulates Ethernet frames in UDP packets and uses a 24-bit VXLAN Network Identifier (VNI) to allow for about 16 million isolated Layer 2 networks within a common Layer 3 infrastructure.

Why Use VXLAN?

1. Scalability: Overcomes the 4096 VLAN ID limit, supporting up to 16 million virtual networks.
2. Flexibility: Can be used over any IP network and across different data centers or cloud environments.
3. Compatibility: Works with existing virtualization technologies and can be implemented in software or supported by physical network hardware.

Setting Up VXLAN for Bridge-to-Bridge Communication

Prerequisites:

• Two hosts, each with at least one network interface.
• IP connectivity between the hosts.
• Kernel support for VXLAN (common in modern Linux distributions).

Configuration Steps:

Step 1: Install Necessary Tools Ensure iproute2 is updated as it contains necessary tools for managing VXLAN interfaces.

sudo apt-get update
sudo apt-get install iproute2

Step 2: Create Bridges on Each Host First, you need to set up a bridge on each host. Here's how you might set up a bridge on both Host A and Host B:

# On Host A
sudo ip link add br0 type bridge
sudo ip link set br0 up

# On Host B
sudo ip link add br0 type bridge
sudo ip link set br0 up

Step 3: Create VXLAN Interface On each host, create a VXLAN interface. This example uses VXLAN ID 42 and assumes the source IP addresses are static and known.

# On Host A
sudo ip link add vxlan42 type vxlan id 42 dev eth0 dstport 4789 remote <IP_OF_HOST_B> local <IP_OF_HOST_A> nolearning
sudo ip link set vxlan42 up
sudo ip link set vxlan42 master br0

# On Host B
sudo ip link add vxlan42 type vxlan id 42 dev eth0 dstport 4789 remote <IP_OF_HOST_A> local <IP_OF_HOST_B> nolearning
sudo ip link set vxlan42 up
sudo ip link set vxlan42 master br0

Replace <IP_OF_HOST_A> and <IP_OF_HOST_B> with the respective IP addresses of your hosts.

Step 4: Assign IP Addresses (Optional) For management or testing, you might want to assign IP addresses to the bridge or to virtual interfaces attached to the bridge.

# On Host A
sudo ip addr add 192.168.1.1/24 dev br0

# On Host B
sudo ip addr add 192.168.1.2/24 dev br0

Step 5: Testing Connectivity Use ping or other network tools to test connectivity between the hosts.

# On Host A
ping 192.168.1.2

Advanced Topics

• Security: Consider using IPsec to secure VXLAN traffic, especially when traversing untrusted networks.
• Dynamic VXLAN Setup: For dynamic environments (like those managed by Kubernetes or OpenStack), look into automating VXLAN setup with network controllers or using protocols like EVPN.
• Performance: Monitoring and tuning the performance of VXLAN tunnels is crucial, especially in high-throughput environments. Techniques include offloading VXLAN processing to network hardware, tuning MTU settings, and using jumbo frames.

Conclusion

VXLAN provides a robust method for extending Layer 2 networks over Layer 3 infrastructures. When properly configured, it enables flexible, scalable, and secure network designs across geographically dispersed locations. This setup is especially beneficial in environments where virtualization and containerization are heavily used, allowing seamless connectivity across various hosts and clusters.

graph TD;
    subgraph Site A
        A_OPNsense[OPNsense Gateway A<br>192.168.10.1] --> A_Debian[Debian A<br>10.0.0.1<br>VXLAN ID 100]
    end
    subgraph Site B
        B_OPNsense[OPNsense Gateway B<br>192.168.20.1] --> B_Debian[Debian B<br>10.0.0.2<br>VXLAN ID 100]
    end
    subgraph Site C
        C_OPNsense[OPNsense Gateway C<br>192.168.30.1] --> C_Debian[Debian C<br>10.0.0.3<br>VXLAN ID 100]
    end
    A_Debian --- B_Debian
    B_Debian --- C_Debian
    C_Debian --- A_Debian

---

Routing traffic from VXLAN tunnels between Linux bridges and potentially to an OPNsense gateway involves several steps, focusing on ensuring proper encapsulation, decapsulation, and routing of packets. Here’s a detailed approach to handle this scenario effectively:

1. Handling VXLAN Traffic on Linux Hosts

When dealing with VXLAN tunnels on Linux, the key aspect is managing how traffic is encapsulated and decapsulated. This process typically involves:

• Creating VXLAN Interfaces: As discussed earlier, each Linux host will have a VXLAN interface configured. This interface encapsulates outgoing traffic and decapsulates incoming traffic based on the VXLAN Network Identifier (VNI).

• Bridging VXLAN and Ethernet Interfaces: Often, it might be necessary to bridge the VXLAN interface with one or more physical or virtual Ethernet interfaces. This setup allows all interfaces in the bridge to communicate as if they were in the same physical network segment.

sudo ip link add name br0 type bridge
sudo ip link set br0 up
sudo ip link set eth0 up
sudo ip link set vxlan0 master br0
sudo ip link set eth0 master br0

This command sequence sets up a bridge br0 and adds both the Ethernet interface eth0 and the VXLAN interface vxlan0 to this bridge.

2. Routing Traffic Between Bridges

To route traffic between different Linux bridges, which might be in different network namespaces or on different hosts:

• Configure IP Forwarding: Make sure IP forwarding is enabled on the Linux hosts to allow traffic to be routed between interfaces.

echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

• Set Up Routing Rules: If the bridges are in different subnets, set up static routing rules or use dynamic routing protocols to manage the routes.

sudo ip route add 192.168.2.0/24 via 192.168.1.2 dev br0

This command tells the system how to find the 192.168.2.0/24 network via the next-hop IP address 192.168.1.2, which is accessible via the br0 bridge interface.

3. Integrating with OPNsense

If you need to route traffic from the VXLAN to an OPNsense gateway, the approach will depend on whether the OPNsense device is acting as the edge router for the VXLAN network or if it's just another node within the network:

• As an Edge Router: Ensure that the OPNsense has routes back to the VXLAN network and that NAT (Network Address Translation) settings are configured if needed. This is especially important if the VXLAN IPs are not part of the routable address space managed by OPNsense.

• NAT Configuration: Configure NAT on OPNsense to allow devices outside the VXLAN (like the internet or other corporate networks) to communicate with devices inside the VXLAN.

• Firewall Rules: Modify firewall rules in OPNsense to allow traffic from the VXLAN networks. This can involve allowing specific ports or entire subnets.

4. Debugging and Validation

• Use tools like ping, traceroute, tcpdump, and ip link to test connectivity and monitor the traffic to ensure that the routing and bridging are configured correctly.
• Monitoring VXLAN Traffic: You can monitor VXLAN traffic specifically using tcpdump by filtering VXLAN traffic:

sudo tcpdump -ni any 'port 4789'

This setup provides a robust configuration for managing traffic flow between VXLAN segments, other network bridges, and an OPNsense gateway. Each step ensures that traffic is correctly managed, encapsulated, or decapsulated, and securely routed according to your network policies.