Data Center Networks

Published

July 1, 2026

Modified

July 6, 2026

Networks

CLOS is a non-blocking, multistage switching architecture and the standard for modern data center networks.

Charles Clos formalized the concept in 1953 for telephone circuit-switching networks. His design combined multiple small crossbar switches in sequence to reduce cost while maintaining full connectivity between any input and output port. Modern data centers adapted the theory into the folded CLOS topology known as spine-and-leaf.

Traditional three-tier data center designs — core, distribution, access — created bottlenecks at the core and unpredictable latency due to variable hop counts. Every traffic flow in a CLOS fabric traverses exactly three stages, leaf to spine to leaf, regardless of fabric size. The fixed hop count delivers consistent latency and equal-cost multipath forwarding.

Google adopted CLOS topologies with the Jupiter project around 2005, scaling to building-scale fabrics with over 1 Pbps of bisection bandwidth. Yahoo, Microsoft, and Meta followed, making CLOS the de facto architecture for cloud data centers.

When to Use

CLOS is not the right choice for every deployment. The topology’s full-mesh cabling consumes switch ports rapidly at small scale, making simpler designs more economical below certain thresholds.

Hosts	Topology	Notes
<50	Stacked L2 switches	Two or three switches in a stack handle the workload without needing a fabric
50–200	Dual-homed leaf or ring	Provides redundancy while minimizing port waste — each leaf only needs two uplinks instead of one per spine
200+	CLOS (spine-and-leaf)	Fixed-hop latency, ECMP distribution, and linear scaling outweigh the cabling cost once east-west traffic dominates

In a dual-homed leaf design, servers connect to two leaf switches for redundancy (typically via MCLAG), but each leaf connects to only two spines instead of every spine. The spines may be chained or partially meshed. This saves switch ports compared to full-mesh CLOS because each leaf uses exactly two uplinks regardless of fabric size. The tradeoff is fewer ECMP paths and potentially variable latency if the spines are not fully meshed. It fits mid-scale deployments where port density on the leaves is more valuable than guaranteed three-hop performance.

A ring topology chains leaf switches in a loop with no dedicated spine tier. Each leaf connects to exactly two neighbors, and traffic flows clockwise or counter-clockwise depending on path cost. It provides basic redundancy — surviving a single link or switch failure — but has variable hop counts, with a worst-case diameter of half the ring. Rings work for small campuses or lab environments where budget matters more than predictable latency.

Traffic Patterns

Workload characteristics determine whether CLOS delivers value or wastes resources.

East-west dominant — CLOS excels. Workloads that communicate heavily across nodes (databases, distributed storage, ML training) benefit from full bisection bandwidth and consistent latency.
North-south dominant — If most traffic exits the fabric toward users or upstream networks, a traditional hierarchical design may suffice since the core bottleneck is less likely to be saturated.
Collective operations — HPC workloads with frequent all-reduce or broadcast patterns may prefer Dragonfly topology, which trades some bisection bandwidth for shorter diameter and lower worst-case latency at scale.
RDMA / RoCE — Standard ECMP can cause head-of-line blocking for lossless Ethernet flows. HPC fabrics running RDMA often layer adaptive routing over CLOS to avoid static hash collisions.

For mixed environments — compute plus general infrastructure — a hybrid approach works: CLOS for the shared fabric, with a dedicated low-latency segment (InfiniBand or RoCE with adaptive routing) for compute nodes that require it.

Performance

A properly sized CLOS fabric provides full bisection bandwidth. Split the endpoints into two equal halves, and every endpoint in one half communicates with the other at full line speed simultaneously. The fabric carries all that traffic without congestion — unlike hierarchical designs where the core becomes a bottleneck.

Each leaf connects to every spine using equal-cost paths, enabling ECMP to distribute traffic across all spine links simultaneously. With N spines, a leaf-to-leaf flow has N parallel paths, multiplying available throughput linearly with each added spine.

Some designs intentionally apply oversubscription — provisioning fewer spine-facing ports than server-facing ports — to reduce cost. Oversubscription lowers capital expenditure but risks queue buildup, packet drops, and increased jitter under heavy east-west loads. It remains acceptable when application profiles operate well below port capacity or when traffic patterns do not overlap.

Layers

Leaf switches sit at the fabric edge and connect directly to servers, storage, or other endpoints. They typically feature high-density server-facing ports (e.g., 48 × 25G) with fewer uplink ports of higher speed (e.g., 6 × 100G). Each leaf is fully meshed to every spine switch, ensuring no single path becomes a bottleneck.

Spine switches form the transit backbone. They connect exclusively to leaf switches — never to end devices — and forward traffic between leaves without applying policies or performing host-facing functions. In a pure CLOS design, spines are dumb transit nodes that keep intelligence at the edge. To any connected server, the whole network behaves like one giant switch with equal access to every other endpoint and identical latency, hiding all internal complexity.

Super-spine switches extend the topology beyond three stages. When a fabric grows too large for a single spine tier to handle, super-spines add two more stages (leaf → spine → super-spine → spine → leaf), creating a five-stage CLOS architecture. This recursive scaling supports tens of thousands of endpoints while maintaining non-blocking characteristics, though it increases hop count and control-plane complexity.

Modularity

A modular CLOS fabric scales horizontally by adding units rather than upgrading individual chassis. Add leaf switches for more server capacity; add spine switches for more east-west bandwidth. The full-mesh connectivity preserves itself automatically, keeping performance predictable as the network grows.

This modularity has direct implications for cabling. Every leaf must connect to every spine, so adding a single leaf switch requires running one cable to each existing spine. Adding a spine means cabling it to every existing leaf. In a fabric with 20 leaves and 6 spines, each new leaf needs 6 uplink cables, and each new spine needs 20 downlink cables.

Cable count scales predictably: total cables equal the number of leaves multiplied by the number of spines. A 20 × 6 fabric uses 120 cables. This regularity simplifies cable labeling conventions and data center rack layout — every switch in a given tier has identical cabling patterns. Provision switch port capacity and rack space with headroom for future expansion.

Deploy 4 leaves and 2 spines today, expecting to grow to 10 leaves and 6 spines within two years. The initial fabric uses only 8 cables (4 × 2), but the target requires 60 (10 × 6). Choose spine switches with enough uplink ports for all 10 planned leaves, not just the 4 in use today. Reserve rack space where the additional leaves will go, close to the spines. When expansion time comes, stage the new switches and connect them to the spare spine ports — a matter of hours, not a facility project.

Underlay

The underlay provides IP reachability between every switch in the fabric. It is the foundation on which overlay networks like VXLAN depend.

BGP (Border Gateway Protocol) is the routing protocol of choice for large-scale data center fabrics¹. Hyperscalers adapted BGP from its original role in inter-domain internet routing into a simpler, more scalable alternative to traditional interior gateway protocols like OSPF and IS-IS.

Rationale

Traditional link-state protocols flood topology changes across an entire area regardless of failure scope. In a Clos fabric with dozens of parallel paths, this generates unnecessary control-plane chatter. BGP takes a different approach: each router calculates and advertises only its best path, naturally suppressing redundant updates. A link failure is absorbed locally as the affected speaker installs an alternate path, without propagating the event beyond its peers.

BGP also relies on TCP for transport, eliminating the need for custom adjacency formation, neighbor maintenance, and flow-control state machines that link-state protocols require. This reduces implementation complexity and makes peering sessions easier to troubleshoot.

When combined with a folded Clos topology, eBGP’s built-in AS_PATH loop detection prevents unwanted long routes. Every BGP update carries an AS_PATH attribute—a list of autonomous systems the route has traversed. A router rejects any route whose AS_PATH already contains its own ASN, since that indicates the route would loop back through a network that already knows about it.

AS_PATH Loop Detection Example

Consider a fabric with spines sharing ASN 65001, Leaf A with ASN 65101, and Leaf B with ASN 65102. Leaf A advertises its server prefix 10.0.1.0/24 to Spine 1. The spine prepends its ASN, so the route now carries AS_PATH [65001, 65101]. Spine 1 forwards this update to Leaf B, which installs it—traffic for 10.0.1.0/24 flows from Leaf B to Spine 1 to Leaf A, the correct three-hop path. If Leaf B then tried to readvertise that same prefix back to Spine 2, the AS_PATH would be [65102, 65001, 65101]. But when Spine 2 evaluates the update, it sees its own ASN (65001) already in the path and drops it. The route never circulates between spines or takes a longer path like Leaf B → Spine 2 → Spine 1 → Leaf A. This eliminates “path hunting,” where BGP would otherwise explore suboptimal multi-hop routes across the fabric.

Allocation

The standard ASN scheme for a spine-and-leaf fabric uses two tiers:

Assign a single ASN to all spine switches, forming one logical autonomous system
Assign a unique ASN to each leaf switch, making every leaf its own autonomous system

With this scheme, a leaf advertises server-facing prefixes to all spines. A spine receives these routes but never re-advertises them to another spine, because the spine ASN already appears in the AS_PATH. The result is clean east-west forwarding: leaf to spine to leaf, with no unintended transit through multiple spine or leaf hops.

Multi AS

This approach is known as the Multi-AS scheme and is the recommended allocation model in RFC 7938². It relies on private use ASNs reserved by RFC 6996³, which fall into two ranges:

16-bit: 64512–65534 (1,023 available ASNs)
32-bit: 4200000000–4294967294 (~947 million available ASNs)

A typical 16-bit allocation assigns a base number to the spine tier and increments sequentially for each leaf:

Spines: 65001
Leaf 1: 65101, Leaf 2: 65102, Leaf 3: 65103, and so on

The gap between spine and leaf ranges makes configuration errors easier to spot—a leaf accidentally set to 65001 immediately triggers loop detection across the entire spine tier.

Overlay

The underlay provides IP reachability between switches. The overlay provides tenant isolation, multi-tenancy, and Layer 2 connectivity across the fabric without requiring a spanning-tree-based Layer 2 domain underneath.

Benefits

Without an overlay, every VLAN must route through the entire underlay. In a multi-tenant environment with hundreds of VLANs, unknown unicast and broadcast traffic floods across every switch port carrying that VLAN. The approach does not scale and creates security concerns between tenants.

The overlay introduces additional indirection, but the tradeoffs justify the complexity in most data center environments:

Multi-tenancy. VXLAN VNIs provide hard isolation between tenants. Traffic from one VNI cannot leak into another because the underlay only sees outer IP addresses belonging to the infrastructure. A compromised host in one tenant segment cannot reach another tenant’s segment without explicit cross-VNI routing policies.

Network agility. Adding or removing a tenant segment requires configuring a VNI on the relevant leaf switches — no underlay changes, no spanning-tree recalculation, no impact on existing segments. Operations teams can provision overlays independently of the physical fabric.

VM mobility. When a virtual machine migrates between hosts attached to different leaves, it retains its IP and MAC addresses because the VXLAN segment spans the fabric. The BGP EVPN control plane automatically updates MAC-to-VTEP mappings so traffic follows the migrated workload without manual reconfiguration.

Underlay simplicity. The underlay only needs to route VTEP IP addresses — it does not need to know about tenant VLANs, broadcast domains, or application topologies. This separation of concerns allows the underlay and overlay teams to evolve their respective layers independently.

VXLAN

VXLAN (Virtual Extensible LAN) implements the overlay by encapsulating tenant Layer 2 frames inside Layer 3 UDP packets, tunneling them across the IP underlay⁴. A VXLAN network appears as a flat Layer 2 segment to connected hosts, even though the underlying transport is a routed Layer 3 fabric with ECMP, shortest-path forwarding, and no spanning tree.

Encapsulation

Each VXLAN packet has two layers of addressing. The inner frame is the original Ethernet frame from the source host, preserved byte-for-byte including its destination MAC address. The outer header adds new addressing for transport across the underlay.

The outer header consists of:

Outer UDP header with destination port 4789 (IANA-assigned for VXLAN)
Outer IP header with source and destination addresses of the tunnel endpoints
Outer Ethernet header for the first hop toward the remote tunnel endpoint
VXLAN header containing the VNI (Virtual Network Identifier) — a 24-bit field identifying the tenant segment

The 24-bit VNI supports up to 16 million segments (2^24 = 16,777,216), compared to the 4,094 VLAN IDs available in traditional 8-bit VLAN tagging. This eliminates VLAN exhaustion in large multi-tenant deployments.

Encapsulation Walkthrough

A server sends an Ethernet frame destined for another server on the same VXLAN segment but attached to a different leaf. The source leaf — acting as the ingress VTEP — encapsulates the frame by prepending the VXLAN header (with the VNI), outer IP header (source = local VTEP IP, destination = remote VTEP IP), outer UDP header, and outer Ethernet header. The resulting packet enters the underlay as a normal IP packet, where the BGP-routed fabric forwards it using ECMP across spine switches to the egress leaf. The egress VTEP strips the outer headers, revealing the original frame, which is then delivered to the destination server. To the servers, the process is transparent — they communicate over what appears to be a single Layer 2 broadcast domain.

Because the outer header uses standard IP and UDP, the underlay can apply ECMP across all available paths. Unlike Layer 2 hashing — which traditionally limits load balancing to MAC or VLAN fields — the outer IP header provides five-tuple information (source IP, destination IP, source port, destination port, protocol) for richer hash distribution across parallel links.

VTEPs

A VTEP (VXLAN Tunnel Endpoint) terminates VXLAN tunnels. In a SONiC deployment, each leaf switch runs a VTEP that performs both encapsulation (for egress traffic leaving the local segment) and decapsulation (for ingress traffic arriving from the underlay).

The VTEP maintains a MAC-to-VTEP mapping table: for each known host MAC address, it records which remote VTEP IP address that host is reachable through. When forwarding a frame, the VTEP looks up the destination MAC, finds the corresponding remote VTEP IP, and sets it as the outer destination address before handing the packet to the underlay for IP routing.

Two methods populate this table:

ARP flooding — when the VTEP does not know the destination MAC, it floods the ARP request to all VTEPs sharing the same VNI. The replying VTEP registers its IP as the location for that MAC. Simple but generates unnecessary traffic as the fabric scales.
BGP EVPN — VTEPs learn MAC locations through BGP route advertisements instead of flooding, eliminating unknown-unicast storms and enabling active-active gateway designs.

BGP EVPN

BGP EVPN (Ethernet VPN) extends BGP to carry Layer 2 and Layer 3 forwarding information for VXLAN segments, replacing flooding with explicit control-plane signaling⁵. Defined in RFC 7432, it has become the standard control plane for large-scale overlay networks.

EVPN defines several route types, each serving a distinct function:

Type 1 (ES — Ethernet Segment) — advertises which VTEPs belong to a multi-homed Ethernet segment, enabling redundancy between leaves without relying on MCLAG protocols
Type 2 (MAC IP Advertisement) — carries MAC address to VTEP IP mappings, replacing ARP flooding with explicit BGP routes
Type 3 (Inclusive Multicast Route) — advertises VTEP IPs for each VNI, building the multicast group for flood-and-learn traffic without requiring IGMP
Type 5 (IP Prefix Advertisement) — advertises Layer 3 prefixes reachable through the VXLAN overlay, enabling integrated routing and bridging

With Type 2 routes, a VTEP that learns a host’s MAC address advertises it via BGP to all other VTEPs in the same VNI. When any VTEP needs to forward traffic to that MAC, it performs a BGP lookup instead of flooding — converting unknown unicast into known unicast across the entire overlay.

Type 5 routes enable Layer 3 gateway offload. Instead of running a distributed Layer 2 gateway and relying on ARP, the VTEP advertises the tenant’s IP prefix directly into EVPN. Remote VTEPs can then perform local proxy ARP on behalf of the remote gateway, meaning return traffic from the server goes directly to the local VTEP rather than hairpinning across the overlay to a centralized gateway.

Attachment

How servers connect to the fabric edge determines redundancy, bandwidth, and control-plane behavior. The choice depends on workload criticality, operating system capabilities, and whether Layer 2 or Layer 3 attachment is preferred.

Single Attach

The simplest option: a server connects to one leaf switch with one or more cables. There is no switch-level redundancy—if the leaf fails, the server loses network connectivity entirely.

Despite the lack of redundancy, single-homing is common in practice. Many cloud-scale deployments use it because it eliminates control-plane complexity at the edge. Without dual-homing, there is no need for MCLAG synchronization, split-brain detection, or inter-switch link protocols. Each leaf operates independently, making failure domains easier to reason about.

Typical use cases include:

Development and test environments where uptime requirements are relaxed
Bare-metal clusters that rely on application-level redundancy rather than network redundancy
Budget-constrained deployments where reducing switch port count matters
Temporary or transitional workloads such as batch processing nodes

A single-homed server can still aggregate multiple physical links to its leaf using LACP (Link Aggregation Control Protocol, IEEE 802.3ad). This provides link-level redundancy—if one cable fails, traffic shifts to remaining member links—and increases throughput by hashing flows across the bundle. However, the entire bundle terminates on a single switch, so a leaf failure remains a single point of failure.

Dual Attach

The server connects to two leaf switches, typically with one cable per leaf. This provides both link-level and device-level redundancy: if one leaf fails, the other continues forwarding traffic.

Two approaches handle the active forwarding path.

MCLAG

MCLAG (Multi-Chassis Link AGGregation, also called cross-chassis LACP or virtual chassis) makes two leaf switches appear as a single logical device to the connected server. The server runs standard LACP across all its cables, unaware that they terminate on separate switches.

The two leaves synchronize forwarding state over an ISL (Inter-Switch Link), a dedicated high-speed connection carrying control information and, in some implementations, data traffic during asymmetric failures. Key behaviors include:

Active-active forwarding — both links forward simultaneously, maximizing available bandwidth through LACP load balancing
Split-brain protection — if the ISL fails, the leaves must agree on which one retains forwarding authority to prevent duplicate MAC learning and blackholing
MAC synchronization — learned MAC addresses replicate between peers so either leaf can forward without an initial learning delay after failover

MCLAG works well when the server OS does not support BGP. It preserves a Layer 2 attachment model while adding device-level redundancy. The tradeoff is operational complexity: ISL provisioning, synchronization latency tuning, and split-brain recovery procedures require careful configuration.

In SONiC, MCLAG is supported through the swss-mclag container, which manages peer communication, domain synchronization, and lag interface distribution.

ECMP

An alternative to MCLAG is attaching the server at Layer 3 and running BGP from the host operating system itself. The server establishes eBGP sessions to each leaf, advertising its own prefix. The leaves install equal-cost routes and distribute incoming traffic across both paths using ECMP.

This approach requires the server’s OS to support BGP—commonly achieved with FRRouting, Bird, or a kernel-integrated daemon. Modern Linux distributions include FRR as a package, making this feasible on most bare-metal hosts. For virtual machines, the virtual switch (e.g., Open vSwitch) can run BGP on behalf of the guest.

Advantages over MCLAG:

No ISL required — the leaves do not need a direct link between them, reducing cabling and eliminating a shared dependency
No split-brain risk — BGP’s distributed nature means each leaf independently calculates best paths
Scales beyond two leaves — adding a third or fourth leaf simply means establishing another BGP session, whereas MCLAG is inherently a two-device construct

The tradeoff is that the server must participate in the control plane. Its IP address becomes a routed prefix rather than a bridged MAC, which affects ARP behavior and may require changes to how monitoring tools discover hosts.

Host BGP Example

A server with interfaces eth0 and eth1 connected to Leaf A and Leaf B respectively runs FRR. It configures eBGP sessions to both leaves’ loopback addresses, advertising its prefix 10.10.1.0/32. Both leaves receive the route with the same AS_PATH length and MED, installing two equal-cost entries. Incoming traffic hash-distributes across both leaves. If Leaf A fails, the BGP session drops, the leaf withdraws the route, and all traffic shifts to Leaf B—typically within seconds.

Multi Attach

Some high-performance workloads benefit from connecting to more than two leaves. This pattern appears in AI/ML clusters where GPU servers have 8–16 network interfaces, each needing independent uplinks to avoid any single leaf becoming a bottleneck.

With BGP-based attachment, multi-homing is straightforward: the server runs BGP to N leaves, and the underlay distributes traffic across all N paths. With MCLAG, however, most implementations limit the domain to two peers. Scaling beyond two devices requires either vendor-specific extensions (such as Cisco vPC+ or Arista Virtual Switching Framework domains) or moving to a pure Layer 3 model.

Multi-homing also interacts with overlay design. In a VXLAN fabric, the VTEP sits on the leaf switch. If a server connects to four leaves, each leaf acts as an ingress VTEP for that server’s traffic. The BGP EVPN control plane must correctly advertise the server’s MAC from whichever leaf last saw it, and handle rapid MAC mobility if the server’s traffic pattern shifts between interfaces.

Virtual Switches

Virtualized environments add another layer: the hypervisor’s virtual switch mediates between VMs and physical NICs. Common virtual switches include Open vSwitch (OVS), Linux Bridge, and vendor-specific implementations.

The virtual switch handles several responsibilities relevant to fabric attachment:

VXLAN tunnel termination — in many designs, the hypervisor’s VTEP encapsulates VM traffic before it reaches the leaf, shifting overlay processing out of the hardware switch
Port security — limiting which MAC addresses can transmit on a given interface, preventing rogue MAC flooding from VMs
Traffic shaping — enforcing rate limits per VM to protect shared physical links from noisy neighbors

When the virtual switch terminates VXLAN tunnels, the physical leaf sees only decapsulated inner frames. This model—sometimes called host-based VXLAN—is common in VMware NSX and OpenStack Neutron deployments. The tradeoff is CPU overhead on the hypervisor for encapsulation and decapsulation, though modern NICs with hardware offload (SR-IOV, SmartNICs) mitigate this cost.

Footnotes

RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers
https://datatracker.ietf.org/doc/html/rfc7938 ↩︎
RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers
https://datatracker.ietf.org/doc/html/rfc7938 ↩︎
RFC 6996 — Autonomous System (AS) Reservation for Private Use
https://datatracker.ietf.org/doc/html/rfc6996 ↩︎
RFC 7348 — Virtual Extensible LAN (VXLAN)
https://datatracker.ietf.org/doc/html/rfc7348 ↩︎
RFC 7432 — BGP Ethernet VPN
https://datatracker.ietf.org/doc/html/rfc7432 ↩︎

--- title: Data Center Networks date: 2026/07/01 date-modified: 2026/07/06 --- # Networks CLOS is a **non-blocking, multistage switching architecture** and the standard for modern data center networks. **Charles Clos** formalized the concept in 1953 for telephone circuit-switching networks. His design combined multiple small crossbar switches in sequence to reduce cost while maintaining full connectivity between any input and output port. Modern data centers adapted the theory into the **folded CLOS** topology known as **spine-and-leaf**. Traditional three-tier data center designs — core, distribution, access — created bottlenecks at the core and unpredictable latency due to variable hop counts. Every traffic flow in a CLOS fabric traverses exactly **three stages**, leaf to spine to leaf, regardless of fabric size. The fixed hop count delivers consistent latency and equal-cost multipath forwarding. Google adopted CLOS topologies with the **Jupiter** project around 2005, scaling to building-scale fabrics with over 1 Pbps of bisection bandwidth. Yahoo, Microsoft, and Meta followed, making CLOS the de facto architecture for cloud data centers. ## When to Use CLOS is not the right choice for every deployment. The topology's full-mesh cabling consumes switch ports rapidly at small scale, making simpler designs more economical below certain thresholds. | Hosts | Topology | Notes | | --- | --- | --- | | <50 | Stacked L2 switches | Two or three switches in a stack handle the workload without needing a fabric | | 50–200 | Dual-homed leaf or ring | Provides redundancy while minimizing port waste — each leaf only needs two uplinks instead of one per spine | | 200+ | CLOS (spine-and-leaf) | Fixed-hop latency, ECMP distribution, and linear scaling outweigh the cabling cost once east-west traffic dominates | : {tbl-colwidths="[10,25,65]"} In a **dual-homed leaf** design, servers connect to two leaf switches for redundancy (typically via MCLAG), but each leaf connects to only **two spines** instead of every spine. The spines may be chained or partially meshed. This saves switch ports compared to full-mesh CLOS because each leaf uses exactly two uplinks regardless of fabric size. The tradeoff is fewer ECMP paths and potentially variable latency if the spines are not fully meshed. It fits mid-scale deployments where port density on the leaves is more valuable than guaranteed three-hop performance. A **ring topology** chains leaf switches in a loop with no dedicated spine tier. Each leaf connects to exactly two neighbors, and traffic flows clockwise or counter-clockwise depending on path cost. It provides basic redundancy — surviving a single link or switch failure — but has variable hop counts, with a worst-case diameter of half the ring. Rings work for small campuses or lab environments where budget matters more than predictable latency. ## Traffic Patterns Workload characteristics determine whether CLOS delivers value or wastes resources. * **East-west dominant** — CLOS excels. Workloads that communicate heavily across nodes (databases, distributed storage, ML training) benefit from full bisection bandwidth and consistent latency. * **North-south dominant** — If most traffic exits the fabric toward users or upstream networks, a traditional hierarchical design may suffice since the core bottleneck is less likely to be saturated. * **Collective operations** — HPC workloads with frequent all-reduce or broadcast patterns may prefer **Dragonfly** topology, which trades some bisection bandwidth for shorter diameter and lower worst-case latency at scale. * **RDMA / RoCE** — Standard ECMP can cause head-of-line blocking for lossless Ethernet flows. HPC fabrics running RDMA often layer **adaptive routing** over CLOS to avoid static hash collisions. For mixed environments — compute plus general infrastructure — a hybrid approach works: CLOS for the shared fabric, with a dedicated low-latency segment (InfiniBand or RoCE with adaptive routing) for compute nodes that require it. ## Performance A properly sized CLOS fabric provides **full bisection bandwidth**. Split the endpoints into two equal halves, and every endpoint in one half communicates with the other at full line speed simultaneously. The fabric carries all that traffic without congestion — unlike hierarchical designs where the core becomes a bottleneck. Each leaf connects to every spine using equal-cost paths, enabling **ECMP** to distribute traffic across all spine links simultaneously. With N spines, a leaf-to-leaf flow has N parallel paths, multiplying available throughput linearly with each added spine. Some designs intentionally apply **oversubscription** — provisioning fewer spine-facing ports than server-facing ports — to reduce cost. Oversubscription lowers capital expenditure but risks queue buildup, packet drops, and increased jitter under heavy east-west loads. It remains acceptable when application profiles operate well below port capacity or when traffic patterns do not overlap. ## Layers **Leaf** switches sit at the fabric edge and connect directly to servers, storage, or other endpoints. They typically feature high-density server-facing ports (e.g., 48 × 25G) with fewer uplink ports of higher speed (e.g., 6 × 100G). Each leaf is fully meshed to every spine switch, ensuring no single path becomes a bottleneck. **Spine** switches form the transit backbone. They connect exclusively to leaf switches — never to end devices — and forward traffic between leaves without applying policies or performing host-facing functions. In a pure CLOS design, spines are dumb transit nodes that keep intelligence at the edge. To any connected server, the whole network behaves like one giant switch with equal access to every other endpoint and identical latency, hiding all internal complexity. **Super-spine** switches extend the topology beyond three stages. When a fabric grows too large for a single spine tier to handle, super-spines add two more stages (leaf → spine → super-spine → spine → leaf), creating a five-stage CLOS architecture. This recursive scaling supports tens of thousands of endpoints while maintaining non-blocking characteristics, though it increases hop count and control-plane complexity. ## Modularity A modular CLOS fabric scales horizontally by adding units rather than upgrading individual chassis. Add leaf switches for more server capacity; add spine switches for more east-west bandwidth. The full-mesh connectivity preserves itself automatically, keeping performance predictable as the network grows. This modularity has direct implications for **cabling**. Every leaf must connect to every spine, so adding a single leaf switch requires running one cable to each existing spine. Adding a spine means cabling it to every existing leaf. In a fabric with 20 leaves and 6 spines, each new leaf needs 6 uplink cables, and each new spine needs 20 downlink cables. Cable count scales predictably: total cables equal the number of leaves multiplied by the number of spines. A 20 × 6 fabric uses 120 cables. This regularity simplifies cable labeling conventions and data center rack layout — every switch in a given tier has identical cabling patterns. Provision switch port capacity and rack space with headroom for future expansion. Deploy **4 leaves** and **2 spines** today, expecting to grow to **10 leaves** and **6 spines** within two years. The initial fabric uses only 8 cables (4 × 2), but the target requires 60 (10 × 6). Choose spine switches with enough uplink ports for all 10 planned leaves, not just the 4 in use today. Reserve rack space where the additional leaves will go, close to the spines. When expansion time comes, stage the new switches and connect them to the spare spine ports — a matter of hours, not a facility project. # Underlay The underlay provides IP reachability between every switch in the fabric. It is the foundation on which overlay networks like VXLAN depend. **BGP** (Border Gateway Protocol) is the routing protocol of choice for large-scale data center fabrics[^k2p9x]. Hyperscalers adapted BGP from its original role in inter-domain internet routing into a simpler, more scalable alternative to traditional interior gateway protocols like OSPF and IS-IS. ## Rationale Traditional link-state protocols flood topology changes across an entire area regardless of failure scope. In a Clos fabric with dozens of parallel paths, this generates unnecessary control-plane chatter. BGP takes a different approach: each router calculates and advertises only its **best path**, naturally suppressing redundant updates. A link failure is absorbed locally as the affected speaker installs an alternate path, without propagating the event beyond its peers. BGP also relies on **TCP** for transport, eliminating the need for custom adjacency formation, neighbor maintenance, and flow-control state machines that link-state protocols require. This reduces implementation complexity and makes peering sessions easier to troubleshoot. When combined with a folded Clos topology, eBGP's built-in **AS_PATH loop detection** prevents unwanted long routes. Every BGP update carries an **AS_PATH** attribute—a list of autonomous systems the route has traversed. A router rejects any route whose AS_PATH already contains its own ASN, since that indicates the route would loop back through a network that already knows about it. ::: {.callout-note title="AS_PATH Loop Detection Example" collapse="true"} Consider a fabric with spines sharing ASN 65001, Leaf A with ASN 65101, and Leaf B with ASN 65102. Leaf A advertises its server prefix `10.0.1.0/24` to Spine 1. The spine prepends its ASN, so the route now carries AS_PATH `[65001, 65101]`. Spine 1 forwards this update to Leaf B, which installs it—traffic for `10.0.1.0/24` flows from Leaf B to Spine 1 to Leaf A, the correct three-hop path. If Leaf B then tried to readvertise that same prefix back to Spine 2, the AS_PATH would be `[65102, 65001, 65101]`. But when Spine 2 evaluates the update, it sees its own ASN (65001) already in the path and drops it. The route never circulates between spines or takes a longer path like Leaf B → Spine 2 → Spine 1 → Leaf A. This eliminates "path hunting," where BGP would otherwise explore suboptimal multi-hop routes across the fabric. ::: ## Allocation The standard ASN scheme for a spine-and-leaf fabric uses two tiers: * Assign a single **ASN** to all spine switches, forming one logical autonomous system * Assign a unique **ASN** to each leaf switch, making every leaf its own autonomous system With this scheme, a leaf advertises server-facing prefixes to all spines. A spine receives these routes but never re-advertises them to another spine, because the spine ASN already appears in the AS_PATH. The result is clean east-west forwarding: leaf to spine to leaf, with no unintended transit through multiple spine or leaf hops. ## Multi AS This approach is known as the **Multi-AS** scheme and is the recommended allocation model in RFC 7938[^k2p9x]. It relies on **private use ASNs** reserved by RFC 6996[^m7w3q], which fall into two ranges: * **16-bit**: `64512`–`65534` (1,023 available ASNs) * **32-bit**: `4200000000`–`4294967294` (~947 million available ASNs) A typical 16-bit allocation assigns a base number to the spine tier and increments sequentially for each leaf: * Spines: `65001` * Leaf 1: `65101`, Leaf 2: `65102`, Leaf 3: `65103`, and so on The gap between spine and leaf ranges makes configuration errors easier to spot—a leaf accidentally set to `65001` immediately triggers loop detection across the entire spine tier. # Overlay The underlay provides IP reachability between switches. The **overlay** provides tenant isolation, multi-tenancy, and Layer 2 connectivity across the fabric without requiring a spanning-tree-based Layer 2 domain underneath. ## Benefits Without an overlay, every VLAN must route through the entire underlay. In a multi-tenant environment with hundreds of VLANs, unknown unicast and broadcast traffic floods across every switch port carrying that VLAN. The approach does not scale and creates security concerns between tenants. The overlay introduces additional indirection, but the tradeoffs justify the complexity in most data center environments: **Multi-tenancy.** VXLAN VNIs provide hard isolation between tenants. Traffic from one VNI cannot leak into another because the underlay only sees outer IP addresses belonging to the infrastructure. A compromised host in one tenant segment cannot reach another tenant's segment without explicit cross-VNI routing policies. **Network agility.** Adding or removing a tenant segment requires configuring a VNI on the relevant leaf switches — no underlay changes, no spanning-tree recalculation, no impact on existing segments. Operations teams can provision overlays independently of the physical fabric. **VM mobility.** When a virtual machine migrates between hosts attached to different leaves, it retains its IP and MAC addresses because the VXLAN segment spans the fabric. The BGP EVPN control plane automatically updates MAC-to-VTEP mappings so traffic follows the migrated workload without manual reconfiguration. **Underlay simplicity.** The underlay only needs to route VTEP IP addresses — it does not need to know about tenant VLANs, broadcast domains, or application topologies. This separation of concerns allows the underlay and overlay teams to evolve their respective layers independently. ## VXLAN **VXLAN** (**V**irtual **E**xtensible **L**AN) implements the overlay by encapsulating tenant Layer 2 frames inside Layer 3 UDP packets, tunneling them across the IP underlay[^n4j8r]. A VXLAN network appears as a flat Layer 2 segment to connected hosts, even though the underlying transport is a routed Layer 3 fabric with ECMP, shortest-path forwarding, and no spanning tree. ### Encapsulation Each VXLAN packet has two layers of addressing. The **inner frame** is the original Ethernet frame from the source host, preserved byte-for-byte including its destination MAC address. The **outer header** adds new addressing for transport across the underlay. The outer header consists of: * Outer **UDP** header with destination port **4789** (IANA-assigned for VXLAN) * Outer **IP** header with source and destination addresses of the tunnel endpoints * Outer **Ethernet** header for the first hop toward the remote tunnel endpoint * **VXLAN header** containing the **VNI** (**V**irtual **N**etwork **I**dentifier) — a 24-bit field identifying the tenant segment The 24-bit VNI supports up to **16 million** segments (`2^24 = 16,777,216`), compared to the 4,094 VLAN IDs available in traditional 8-bit VLAN tagging. This eliminates VLAN exhaustion in large multi-tenant deployments. ::: {.callout-note title="Encapsulation Walkthrough" collapse="true"} A server sends an Ethernet frame destined for another server on the same VXLAN segment but attached to a different leaf. The source leaf — acting as the **ingress VTEP** — encapsulates the frame by prepending the VXLAN header (with the VNI), outer IP header (source = local VTEP IP, destination = remote VTEP IP), outer UDP header, and outer Ethernet header. The resulting packet enters the underlay as a normal IP packet, where the BGP-routed fabric forwards it using ECMP across spine switches to the egress leaf. The **egress VTEP** strips the outer headers, revealing the original frame, which is then delivered to the destination server. To the servers, the process is transparent — they communicate over what appears to be a single Layer 2 broadcast domain. ::: Because the outer header uses standard IP and UDP, the underlay can apply **ECMP** across all available paths. Unlike Layer 2 hashing — which traditionally limits load balancing to MAC or VLAN fields — the outer IP header provides five-tuple information (source IP, destination IP, source port, destination port, protocol) for richer hash distribution across parallel links. ### VTEPs A **VTEP** (**V**XLAN **T**unnel **E**ndpoint) terminates VXLAN tunnels. In a SONiC deployment, each leaf switch runs a VTEP that performs both encapsulation (for egress traffic leaving the local segment) and decapsulation (for ingress traffic arriving from the underlay). The VTEP maintains a **MAC-to-VTEP mapping table**: for each known host MAC address, it records which remote VTEP IP address that host is reachable through. When forwarding a frame, the VTEP looks up the destination MAC, finds the corresponding remote VTEP IP, and sets it as the outer destination address before handing the packet to the underlay for IP routing. Two methods populate this table: * **ARP flooding** — when the VTEP does not know the destination MAC, it floods the ARP request to all VTEPs sharing the same VNI. The replying VTEP registers its IP as the location for that MAC. Simple but generates unnecessary traffic as the fabric scales. * **BGP EVPN** — VTEPs learn MAC locations through BGP route advertisements instead of flooding, eliminating unknown-unicast storms and enabling active-active gateway designs. ## BGP EVPN **BGP EVPN** (**E**thernet **VPN**) extends BGP to carry Layer 2 and Layer 3 forwarding information for VXLAN segments, replacing flooding with explicit control-plane signaling[^t5v1z]. Defined in RFC 7432, it has become the standard control plane for large-scale overlay networks. EVPN defines several **route types**, each serving a distinct function: * **Type 1 (ES — Ethernet Segment)** — advertises which VTEPs belong to a multi-homed Ethernet segment, enabling redundancy between leaves without relying on MCLAG protocols * **Type 2 (MAC IP Advertisement)** — carries MAC address to VTEP IP mappings, replacing ARP flooding with explicit BGP routes * **Type 3 (Inclusive Multicast Route)** — advertises VTEP IPs for each VNI, building the multicast group for flood-and-learn traffic without requiring IGMP * **Type 5 (IP Prefix Advertisement)** — advertises Layer 3 prefixes reachable through the VXLAN overlay, enabling integrated routing and bridging With Type 2 routes, a VTEP that learns a host's MAC address advertises it via BGP to all other VTEPs in the same VNI. When any VTEP needs to forward traffic to that MAC, it performs a BGP lookup instead of flooding — converting unknown unicast into known unicast across the entire overlay. Type 5 routes enable **Layer 3 gateway offload**. Instead of running a distributed Layer 2 gateway and relying on ARP, the VTEP advertises the tenant's IP prefix directly into EVPN. Remote VTEPs can then perform **local proxy ARP** on behalf of the remote gateway, meaning return traffic from the server goes directly to the local VTEP rather than hairpinning across the overlay to a centralized gateway. # Attachment How servers connect to the fabric edge determines redundancy, bandwidth, and control-plane behavior. The choice depends on workload criticality, operating system capabilities, and whether Layer 2 or Layer 3 attachment is preferred. ## Single Attach The simplest option: a server connects to **one leaf switch** with one or more cables. There is no switch-level redundancy—if the leaf fails, the server loses network connectivity entirely. Despite the lack of redundancy, single-homing is common in practice. Many cloud-scale deployments use it because it eliminates control-plane complexity at the edge. Without dual-homing, there is no need for MCLAG synchronization, split-brain detection, or inter-switch link protocols. Each leaf operates independently, making failure domains easier to reason about. Typical use cases include: * **Development and test environments** where uptime requirements are relaxed * **Bare-metal clusters** that rely on application-level redundancy rather than network redundancy * **Budget-constrained deployments** where reducing switch port count matters * **Temporary or transitional workloads** such as batch processing nodes A single-homed server can still aggregate multiple physical links to its leaf using **LACP** (Link Aggregation Control Protocol, IEEE 802.3ad). This provides link-level redundancy—if one cable fails, traffic shifts to remaining member links—and increases throughput by hashing flows across the bundle. However, the entire bundle terminates on a single switch, so a leaf failure remains a single point of failure. ## Dual Attach The server connects to **two leaf switches**, typically with one cable per leaf. This provides both link-level and device-level redundancy: if one leaf fails, the other continues forwarding traffic. Two approaches handle the active forwarding path. ### MCLAG **MCLAG** (**M**ulti-**C**hassis **L**ink **AGG**regation, also called cross-chassis LACP or virtual chassis) makes two leaf switches appear as a single logical device to the connected server. The server runs standard LACP across all its cables, unaware that they terminate on separate switches. The two leaves synchronize forwarding state over an **ISL** (**I**nter-**S**witch **L**ink), a dedicated high-speed connection carrying control information and, in some implementations, data traffic during asymmetric failures. Key behaviors include: * **Active-active forwarding** — both links forward simultaneously, maximizing available bandwidth through LACP load balancing * **Split-brain protection** — if the ISL fails, the leaves must agree on which one retains forwarding authority to prevent duplicate MAC learning and blackholing * **MAC synchronization** — learned MAC addresses replicate between peers so either leaf can forward without an initial learning delay after failover MCLAG works well when the server OS does not support BGP. It preserves a Layer 2 attachment model while adding device-level redundancy. The tradeoff is operational complexity: ISL provisioning, synchronization latency tuning, and split-brain recovery procedures require careful configuration. In SONiC, MCLAG is supported through the **swss-mclag** container, which manages peer communication, domain synchronization, and lag interface distribution. ### ECMP An alternative to MCLAG is attaching the server at **Layer 3** and running **BGP** from the host operating system itself. The server establishes eBGP sessions to each leaf, advertising its own prefix. The leaves install equal-cost routes and distribute incoming traffic across both paths using ECMP. This approach requires the server's OS to support BGP—commonly achieved with **FRRouting**, **Bird**, or a kernel-integrated daemon. Modern Linux distributions include FRR as a package, making this feasible on most bare-metal hosts. For virtual machines, the virtual switch (e.g., Open vSwitch) can run BGP on behalf of the guest. Advantages over MCLAG: * **No ISL required** — the leaves do not need a direct link between them, reducing cabling and eliminating a shared dependency * **No split-brain risk** — BGP's distributed nature means each leaf independently calculates best paths * **Scales beyond two leaves** — adding a third or fourth leaf simply means establishing another BGP session, whereas MCLAG is inherently a two-device construct The tradeoff is that the server must participate in the control plane. Its IP address becomes a routed prefix rather than a bridged MAC, which affects ARP behavior and may require changes to how monitoring tools discover hosts. ::: {.callout-note title="Host BGP Example" collapse="true"} A server with interfaces `eth0` and `eth1` connected to Leaf A and Leaf B respectively runs FRR. It configures eBGP sessions to both leaves' loopback addresses, advertising its prefix `10.10.1.0/32`. Both leaves receive the route with the same AS_PATH length and MED, installing two equal-cost entries. Incoming traffic hash-distributes across both leaves. If Leaf A fails, the BGP session drops, the leaf withdraws the route, and all traffic shifts to Leaf B—typically within seconds. ::: ## Multi Attach Some high-performance workloads benefit from connecting to **more than two leaves**. This pattern appears in AI/ML clusters where GPU servers have 8–16 network interfaces, each needing independent uplinks to avoid any single leaf becoming a bottleneck. With BGP-based attachment, multi-homing is straightforward: the server runs BGP to N leaves, and the underlay distributes traffic across all N paths. With MCLAG, however, most implementations limit the domain to two peers. Scaling beyond two devices requires either vendor-specific extensions (such as Cisco vPC+ or Arista Virtual Switching Framework domains) or moving to a pure Layer 3 model. Multi-homing also interacts with **overlay design**. In a VXLAN fabric, the VTEP sits on the leaf switch. If a server connects to four leaves, each leaf acts as an ingress VTEP for that server's traffic. The BGP EVPN control plane must correctly advertise the server's MAC from whichever leaf last saw it, and handle rapid MAC mobility if the server's traffic pattern shifts between interfaces. ## Virtual Switches Virtualized environments add another layer: the hypervisor's **virtual switch** mediates between VMs and physical NICs. Common virtual switches include **Open vSwitch** (OVS), **Linux Bridge**, and vendor-specific implementations. The virtual switch handles several responsibilities relevant to fabric attachment: * **VXLAN tunnel termination** — in many designs, the hypervisor's VTEP encapsulates VM traffic before it reaches the leaf, shifting overlay processing out of the hardware switch * **Port security** — limiting which MAC addresses can transmit on a given interface, preventing rogue MAC flooding from VMs * **Traffic shaping** — enforcing rate limits per VM to protect shared physical links from noisy neighbors When the virtual switch terminates VXLAN tunnels, the physical leaf sees only decapsulated inner frames. This model—sometimes called **host-based VXLAN**—is common in VMware NSX and OpenStack Neutron deployments. The tradeoff is CPU overhead on the hypervisor for encapsulation and decapsulation, though modern NICs with hardware offload (SR-IOV, SmartNICs) mitigate this cost. [^k2p9x]: RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers <https://datatracker.ietf.org/doc/html/rfc7938> [^m7w3q]: RFC 6996 — Autonomous System (AS) Reservation for Private Use <https://datatracker.ietf.org/doc/html/rfc6996> [^n4j8r]: RFC 7348 — Virtual Extensible LAN (VXLAN) <https://datatracker.ietf.org/doc/html/rfc7348> [^t5v1z]: RFC 7432 — BGP Ethernet VPN <https://datatracker.ietf.org/doc/html/rfc7432>