What is VXLAN? Packet Structure & Network Components

VXLAN stands for Virtual Extensible LAN. It is a network virtualization technology that addresses the challenges of extending Layer 2 networks over large-scale environments. 

It is essentially a tunnel technology that allows you to create a virtual network overlay on top of existing physical infrastructure. A VXLAN is handy when you need to connect data centers or manage virtual machines across different physical networks.

How does a VXLAN work?

We can visualize VXLAN as putting your Ethernet frames into a container (the UDP packet), and then sending this container across the existing IP network. 

When the container reaches its destination, it is opened, and the original Ethernet frame is retrieved and sent to its final destination. This process is transparent to the devices within each network, maintaining the same Layer 2 adjacency they're accustomed to. 

This means your applications and services can operate seamlessly, regardless of the underlying infrastructure changes or geographical distances.

To cite an actual VXLAN use case scenario, imagine you have several office locations around the globe. Traditionally, connecting these locations on the same Layer 2 network would be challenging, especially with the limitations of VLANs. 

While they are effective for smaller networks, VLANs can run into scalability issues due to their 12-bit VLAN ID field, capped at 4096 VLANs. 

VXLAN steps in with its 24-bit segment ID, or VNI (VXLAN Network Identifier), allowing for up to 16 million unique IDs. This scalability is a big advantage for large enterprises.

Interestingly, the use of VXLAN also provides isolation between different tenant networks. Each VNI (virtual network interface) acts like a separate VLAN, ensuring that traffic from one tenant does not interfere with another. 

For instance, if you have different departments within your company, such as HR, Finance, and Engineering, each can have its own isolated network segment, even though they all traverse the same physical infrastructure.

In short, VXLAN allows for a flexible, scalable, and efficient network design, enabling you to manage and expand your company's network infrastructure with ease. You can create a coherent virtual network across multiple locations, ensuring seamless connectivity and optimal performance for our applications and services.

VXLAN packet structure

The VXLAN network virtualization technology encapsulates Ethernet frames within UDP packets. This encapsulation adds several headers, resulting in additional overhead. Let's look at these headers in detail.

VXLAN Header

The VXLAN header is an essential component in encapsulating frames for the VXLAN network. It's an 8-byte header that carries VXLAN-specific information. To understand it better, we will break it down into its two main parts: the Flags and the 24-bit VXLAN ID.

First, the Flags field. The most crucial bit here is the I bit. When the I bit is set to 1, it signifies that the VXLAN ID is valid and should be considered. Conversely, if the I bit is 0, the VXLAN ID is invalid and should be ignored. 

This bit essentially acts as a toggle switch for the VXLAN ID's validity. All other bits in the Flags field are reserved and must be set to 0. This is non-negotiable. They serve no purpose other than being placeholders right now.

Next, let's talk about the 24-bit VXLAN ID, also known as the Virtual Network Identifier (VNI). This ID is critical because it identifies the specific VXLAN to which the frame belongs. It is like a unique street address for data packets within the virtual network. 

For example, if you have multiple virtual networks running over the same physical infrastructure, the VXLAN ID ensures that data packets reach their correct virtual destination.

The VXLAN header's simplicity in design belies its power. By encapsulating and distinguishing packets, it offers a scalable solution for creating isolated Layer 2 networks over a shared Layer 3 infrastructure. It’s the core of what makes VXLAN such a useful technology for modern data centers.

Outer IP/UDP header

The Outer IP and outer UDP headers in a VXLAN handle the nuts and bolts of how data travels across our network. It is the envelope carrying your letter (the payload) through a complex postal system.

The Outer IP header is the first layer. It's like the address on your envelope, telling routers exactly where to send your data. This header contains critical information such as Source and Destination IP addresses. 

For example, if you have a server with an IP address of 192.168.1.10 sending data to another server at 192.168.1.20, these addresses will be in the Outer IP header. This layer ensures the data packet navigates the broader network to the right destination.

Outer UDP header

UDP, or User Datagram Protocol, is a bit like writing "URGENT" on your envelope. It tells the network to deliver your data quickly, without doing extensive error-checking. 

The UDP header includes the Source Port and Destination Port. These ports act like doors on a building, guiding your packet to the correct application. For instance, if your application is listening on Port 4789 (which is the default port for VXLAN), the UDP header will list this as the Destination Port.

Let’s visualize this with an example. 

Suppose you have a virtual machine (VM) on one hypervisor communicating with a VM on another hypervisor across the network. The Outer IP header might have a Source IP of 10.1.1.1 and a Destination IP of 10.1.1.2. 

The UDP header would then have a Source Port of 12345 and a Destination Port of 4789, indicating it's a VXLAN packet. This setup tells every router and switch it encounters how to handle the packet and where to forward it.

Adding the VXLAN ID (VNI) to these headers is like adding a special code on your envelope that gets recognized by every post office in your network. This VNI ensures that, even though your data is traveling through shared infrastructure, it remains isolated and secure, just like each piece of mail stays confidential despite being part of a giant mail system.

So, the Outer IP and outer UDP headers are fundamental. They are your packet’s navigational guide, ensuring that data moves swiftly and securely from one point to another in your VXLAN-based company network.

VXLAN network components

Tunnel endpoints (VTEPs)

VTEPs are the devices that handle the encapsulation and de-encapsulation of VXLAN traffic. They act as translators, taking regular Layer 2 Ethernet frames and wrapping them in a VXLAN header to be sent across an IP network and reversing when the packets reach the other side.

Say you have two data centers, Data Center A and Data Center B. Both have their own local area networks (LANs) and you want devices in these separate LANs to communicate as if they're on the same network. 

To do this, you set up VTEPs in each data center. For instance, imagine you have a server in Data Center A that needs to communicate with another server in Data Center B. 

The server in Data Center A sends its Ethernet frame to the local VTEP. The VTEP encapsulates this frame into a VXLAN packet by adding a VXLAN header and delivers it over the IP network to the VTEP in Data Center B. Once it arrives, the VTEP in Data Center B removes the VXLAN header and sends the original Ethernet frame to the target server.

VTEPs can be either hardware-based, like switches, or software-based, like virtual switches running on a hypervisor. A hardware VTEP is designed to handle high throughput and offers robust performance for large-scale networks. On the other hand, a software VTEP could be commonly found in virtualized environments.

One key aspect of VTEPs is their mapping function. They map the VXLAN Network Identifier (VNI) to the appropriate VLAN on the local network. This mapping ensures that the encapsulated traffic reaches the correct destination within the data center. 

For instance, if the server in Data Center A is on VLAN 10, the VTEP will map this to a specific VNI – say VNI 5000. The VTEP in Data Center B will then know that VNI 5000 corresponds to VLAN 20 in its local setup, ensuring that the traffic is directed correctly.

Additionally, VTEPs have to maintain a forwarding table that maps MAC addresses to their corresponding VTEP IP addresses. This is crucial for ensuring that VTEP-to-VTEP communication is efficient and that traffic is routed accurately. 

For instance, if the VTEP in Data Center A receives a frame destined for a MAC address associated with a server in Data Center B, it consults its forwarding table to find the right VTEP IP address to send the encapsulated packet to.

So, VTEPs are essential for making VXLAN work. They handle the encapsulation and de-encapsulation, manage mappings between VNIs and VLANs, and maintain forwarding tables to ensure efficient routing of traffic. Without VTEPs, the seamless extension of Layer 2 networks across Layer 3 boundaries that VXLAN promises wouldn't be possible.

VXLAN network identifier (VNI)

The VXLAN header includes a crucial 24-bit field known as the VXLAN Network Identifier (VNI). The VNI serves as a unique ID for each VXLAN segment, much like VLAN IDs do for traditional VLANs. 

But there's a significant difference. While VLAN IDs are limited to 4094 due to their 12-bit length, the 24-bit length of the VNI allows for a staggering 16 million unique VXLAN segments.

Say you're running a large cloud infrastructure with thousands of tenants. Each tenant might need its own isolated network segment. With traditional VLANs, you would quickly hit the limit of 4094 VLANs. 

In contrast, by using VXLANs and their 24-bit VNI, you can theoretically support up to 16 million isolated segments. This scalability makes it especially ideal for large-scale data centers.

The VNI provides robust network segmentation on an immense scale. Think of a large enterprise with various departments, each needing its own isolated segment. 

With traditional VLANs, managing these segments would be limiting and cumbersome. But with VXLANs, you have a vast number of isolated segments at your disposal. Simply assign each department its own unique VNI, and you're good to go.

When a VTEP encapsulates an Ethernet frame into a VXLAN packet, it includes the VNI in the VXLAN header. This VNI identifies the specific VXLAN segment to which this packet belongs. 

So, if you have a virtual machine in New York and another in San Francisco, both can belong to the same VXLAN segment as long as they share the same VNI. This makes it easy to manage and move virtual machines across different locations without worrying about Layer 2 boundaries.

Remember, the VNI is essential for the encapsulation process. It ensures that the encapsulated packet is delivered correctly to its respective VXLAN segment. 

When the packet reaches its destination VTEP, the VTEP de-encapsulates it and forwards it to the appropriate layer within the network. This seamless process is powered by that tiny yet mighty 24-bit number.

Hardware vs. Software VXLAN. What are the differences

When setting up a hybrid network, deciding between software or hardware-based VXLAN can be tricky. With software-based VXLAN, only a single VTEP (VXLAN Tunnel Endpoint) L2 Gateway can be active for a given VXLAN instance. This setup confines all end-systems connected to the VLAN on the same leaf switch where the VTEP GW is attached. 

For example, if you have servers supporting the same application spread over multiple racks, they must all connect to the same leaf switch for communication to happen. If a server is on a different leaf switch, it can't communicate with the VTEP L2 GW, leading to isolated network segments. This can be a hassle with hybrid networks.

To bridge VNID and VLAN, the L2 network domain must span between the active VTEP L2 Gateway and all concerned servers sharing the same VLAN. This means you'd extend the Layer 2 domain via the fabric using a protocol like FabricPath. 

Take a scenario where Server 4 is on Leaf 4 and the VTEP L2 GW is on Leaf 1. Without Layer 2 extension, Server 4 can't communicate with the VTEP on Leaf 1, isolating it from the network.

Now, hardware-based VXLAN offers a neat solution. Here, the VTEP L2 GW is distributed and active on each leaf switch. This means communications between VTEPs on different switches use the VXLAN tunnel. 

So, in the same scenario, VNID 5000 can be bridged with VLAN 100 on Leaf 4, allowing VM-1 to communicate with Server 4 seamlessly. Each switch handles its VTEP, eliminating the need for an extended Layer 2 domain.

Another cool feature with hardware-based VXLAN is VLAN translation using VLAN stitching. Each leaf switch can map the same VNID with a different VLAN on its own side. 

For instance, VNID 5000 can be bridged with VLAN 100 on Leaf 1 and VLAN 200 on Leaf 6. This setup allows VLAN 100 and VLAN 200 to share the same broadcast domain, which is convenient for migration purposes.

In essence, while software-based VXLAN is great for fully virtualized environments, it doesn't always fit hybrid networks with mixed virtual and physical devices. Hardware-based VXLAN, with its distributed VTEP L2 GWs and VLAN stitching, offers a more flexible and robust solution.

VXLAN’s compatibility with network hardware and security tools

Switches

Many modern enterprise switches, such as the Cisco Nexus series or Arista's 7000 series, come with VXLAN support built-in. These switches can handle the encapsulation and decapsulation processes that VXLAN requires. 

For example, in a multi-site deployment, a Cisco Nexus switch can act as a VXLAN Tunnel Endpoint (VTEP), encapsulating traffic into VXLAN packets for transmission across the IP network. 

Additionally, setting up VXLAN on these switches can be done through simple configurations. These straightforward steps are well-documented, and many switch manufacturers provide comprehensive guides to ease the configuration process.

Another VXLAN benefit is flexibility. It gives you the freedom to deploy workloads anywhere within your network infrastructure without having to reconfigure the underlying physical network. 

This decouples the physical topology from the logical network design, making your lives much easier. For example, you can move a virtual machine from one server to another across different subnets without changing its IP address.

VXLAN also enhances data security. It supports segmentation and isolation of network traffic, which is vital for environments where data privacy is a concern. 

Combining VXLAN with access control lists (ACLs) on your switches can help you tightly control which segments can communicate with each other, adding an extra layer of protection against internal threats.

Therefore, incorporating VXLAN in your network setup with the right switches can significantly boost your scalability, flexibility, and security. It's a powerful tool that addresses many of the challenges you face with traditional VLANs, and with the right hardware, setting it up can be quite straightforward.

Routers

When setting up VXLAN, you'll need to make sure your routers support VXLAN. Many modern routers do, but it's always good to double-check the specs or documentation. 

With some routers you would start by enabling VXLAN globally. This is usually done with a simple command like `feature vxlan`. Once VXLAN is enabled, you'll need to configure the VXLAN Network Identifier (VNI). The VNI acts like a tag, identifying different VXLAN segments. 

So let's say you're setting up a VNI of 5000 for your finance department. You might use `vxlan vlan 100 vni 5000`. This ties VLAN 100 to your VNI 5000, effectively segmenting the finance department's traffic. 

Now, on to the NVE (Network Virtualization Edge) interface. This is a virtual interface that terminates VXLAN tunnels. You'd configure this with an IP address and link it to your physical interface. 

Don't forget about multicast for VXLAN. This is crucial for efficient network traffic distribution. You'll need to configure a multicast group address. For example, you might use `239.1.1.1` for your multicast group. On your router, it could look something like `member vni 5000 multicast-group 239.1.1.1`.

It's also worth setting up redundancy to avoid any single points of failure. Configuring your routers in an HA (High-Availability) setup ensures that if one router fails, another can take over. This often involves configuring HSRP (Hot Standby Router Protocol) along with your VXLAN settings.

You might need to tweak your MTU settings as well. VXLAN encapsulation adds overhead, so make sure your MTU is large enough to accommodate it. If your standard MTU is 1500 bytes, you might bump it up to 1600 bytes to handle the extra load without fragmentation.

All these settings and configurations ensure that your VXLAN setup on routers is both efficient and robust. They help segregate traffic, improve security, and scale your network architecture seamlessly.

Firewalls

In VXLAN network architectures, firewalls play a crucial role in securing the overlays. They ensure only the right people get in and out. With VXLAN, you have multiple tenants or segments, and the firewall ensures that traffic between these segments is properly controlled.

To use real-world examples, imagine you have two segments in your VXLAN—one for HR and another for engineering. The HR segment deals with sensitive employee information, while the engineering segment handles product development. 

You don't want just any traffic passing between these two segments. This is where your firewall steps in. It inspects and filters traffic based on policies you set up.

Suppose HR computers need to access a payroll application on the engineering segment. You can configure firewall rules to allow this specific traffic while blocking everything else. 

For instance, you would permit traffic from the HR subnet to the IP address of the payroll application server on specific ports, say TCP port 443 for HTTPS.

Also, firewalls in VXLAN environments need to understand VXLAN headers. Traditional firewalls might fall short here. So, next-gen firewalls that support VXLAN are essential. 

Next-gen firewalls not only inspect the inner packets but also understand the outer VXLAN headers. For example, if your firewall doesn't recognize VXLAN, it might miss threats encapsulated within the overlay network.

Firewalls also help in segmenting traffic between different VTEPs (VXLAN Tunnel Endpoints). Let's say you have VTEP1 and VTEP2, each handling different sets of workloads. If VTEP1 is compromised, you need to prevent the attacker from moving laterally to VTEP2. A firewall can enforce this by allowing only legitimate, pre-defined traffic between the VTEPs.

Also consider using distributed firewalls for better scalability and performance. In a large VXLAN deployment with many segments, a central firewall can become a bottleneck. 

Distributed firewalls run on the hypervisors or network devices themselves, inspecting traffic close to the source. This setup is more efficient and scales with your network growth.

Lastly, don't forget about logging and monitoring. Firewalls in VXLAN networks should log traffic flows and security events. These logs help detect anomalies and troubleshoot issues. 

For instance, if someone in the HR segment tries to access unauthorized resources in the engineering segment, the firewall log will show this attempt, allowing you to take action.