Overlay networks have become a cornerstone of modern software-defined deployments, providing flexibility and scalability in how data is routed and managed across disparate networks. These overlays operate on top of underlay networks, which are the physical or virtual networks that provide basic connectivity. Understanding the distinction between underlay and overlay networks, as well as the implications of architecture choices—such as mesh and hub & spoke—is crucial for optimizing performance, cost, and manageability.
Underlay vs. Overlay Networks
Before diving into architecture choices, it's important to distinguish between underlay and overlay networks:
Underlay Networks: These are the foundational physical or virtual networks that provide the basic infrastructure for data transmission. Underlays are typically the routers, switches, and physical connections that make up a network. The underlay is responsible for traditional IP routing and switching.
Overlay Networks: Overlays are virtual networks built on top of the underlay. They abstract the underlying infrastructure, allowing for more advanced features like multi-tenancy, segmentation, and enhanced security. Overlay networks encapsulate the original data packets into new packets that can traverse the underlay. Examples of overlays include GRE (Generic Routing Encapsulation), IPSEC, VPN (especially SSL-based variants), VXLAN (Virtual Extensible LAN), and Wireguard.
Architecture Choices | Mesh vs. Hub & Spoke
When deploying an overlay network, one of the most critical decisions is choosing between a mesh and a hub-and-spoke architecture. Each has its own set of benefits and challenges, particularly in terms of complexity, cost, and resource requirements.
Mesh Architecture
In a mesh architecture, each node in the network is connected to every other node, creating a web of connections. This design is often recommended to avoid "hairpinning," where data has to travel through a central hub before reaching its destination, which can introduce latency.
Complexity: The complexity of a mesh network increases exponentially with the number of sites (or nodes). The formula for calculating the number of tunnels in a full mesh network is N*(N-1)/2, where N is the number of nodes. For example, a network with 100 sites would require managing 4,950 tunnels. As you add more nodes, the management overhead becomes unsustainable, especially when you consider the possibility of multiple underlays per site.
Costs: The resource-intensive nature of a mesh architecture means higher costs in terms of both deployment and ongoing management. The complexity of policy-based routing, queuing, and managing link degradation or flapping further adds to the burden.
Use Case: While mesh architectures might be ideal for small, tightly-knit networks where low latency is critical, they quickly become impractical at scale.
Hub & Spoke Architecture
Hub & spoke architecture, on the other hand, connects all nodes (spokes) to a central node (hub). This design significantly reduces the number of tunnels needed, making it easier to manage and more cost-effective.
Simplicity: In a hub & spoke architecture, the number of tunnels is linear rather than exponential, reducing both complexity and cost. The architecture is straightforward, with a clear path for data from the spoke to the hub and then to its destination.
Scalability: Hub & spoke architectures are more scalable, with the ability to support a large number of nodes without the exponential increase in tunnel count. For example, this architecture can support branch networking at scale with up to a million nodes, making it suitable for large deployments.
Hairpinning Myth: The common argument against hub & spoke is the potential for hairpinning, where data must pass through the hub even if the final destination is closer to the source. However, in practice, the impact of hairpinning is negligible, especially in regions like South Africa, where there are a few key concentration points (e.g., Johannesburg, Cape Town, and Durban). The hairpin effect is more of a theoretical concern than a practical one, especially when compared to the complexity and cost of managing a full mesh network.
Overlay Protocols & Their Role in Architecture
Overlay protocols play a key role in determining the performance and flexibility of the chosen architecture. Here’s how some common protocols fit into the hub-and-spoke versus mesh debate:
GRE, IPSEC, VPN (SSL-based): These protocols are widely used in both hub-and-spoke and mesh architectures, providing secure, encapsulated tunnels for data transmission. However, they can add complexity in a mesh setup due to the large number of tunnels. IPSEC is by far the biggest pain of all!
VXLAN: Ideal for data center interconnects, VXLAN can be used effectively in a hub-and-spoke architecture to extend Layer 2 networks over Layer 3 underlays. This protocol is well-suited for environments that require high scalability.
Wireguard: A lightweight VPN protocol that offers high performance and simplicity. Wireguard’s ease of use makes it a strong candidate for both hub-and-spoke and small-scale mesh networks.
Wrap | The Case for Hub & Spoke
While both mesh and hub-and-spoke architectures have their place, the latter offers a more practical and cost-effective solution for most deployments. The simplicity, scalability, and manageability of hub-and-spoke make it the preferred choice for large-scale SD-WAN and overlay network deployments.
Mesh architectures, while theoretically appealing, often introduce more complexity than they solve, especially in terms of tunnel management and policy enforcement. In most cases, the benefits of avoiding hairpinning are outweighed by the operational challenges.
Ultimately, the choice of architecture should be guided by the specific needs of the deployment, but for most use cases, hub-and-spoke provides a balanced, scalable, and manageable approach to overlay network implementations.
Ronald Bartels ensures that Internet inhabiting things are connected reliably online at Nepean Networks - the leading specialized SD-WAN provider in South Africa. Learn more about the best SD-WAN provider in the world! 👉 Contact Nepean
