Skip to main content

ECMP Load Balancing

Equal-Cost Multi-Path (ECMP) routing distributes traffic across multiple paths that share the same routing cost. On NetActuate's network, ECMP allows you to spread traffic across multiple VMs or bare metal servers at the same location, providing both load distribution and redundancy.

ECMP Basics​

When a router has multiple equal-cost routes to the same destination prefix, it can forward packets across all of those paths simultaneously rather than selecting a single best path. The router uses a hash of packet header fields (typically source IP, destination IP, protocol, and port numbers) to assign each flow to a specific path. This ensures that packets belonging to the same flow always follow the same path, preserving packet ordering.

Key characteristics of ECMP:

  • Per-flow distribution -- packets in the same flow stay on the same path
  • Deterministic hashing -- the same 5-tuple always maps to the same path
  • Automatic failover -- if a path goes down, flows are redistributed across remaining paths
  • No application changes required -- ECMP operates at the network layer

ECMP with Anycast​

Combining ECMP with anycast gives you two layers of traffic distribution:

  1. Geographic distribution (anycast) -- routes users to the nearest location based on BGP path selection
  2. Local distribution (ECMP) -- distributes traffic across multiple servers within each location

This combination is ideal for latency-sensitive, high-availability workloads like DNS, CDN edge nodes, and API gateways.

ECMP with Virtual Machines​

To use ECMP with VMs on NetActuate:

  1. Deploy multiple VMs at the same location.
  2. Configure each VM with a BGP session announcing the same prefix.
  3. Each VM establishes a BGP peering with the NetActuate router.
  4. The router sees multiple equal-cost paths to the prefix and enables ECMP forwarding.

Traffic is distributed across VMs based on the flow hash. If one VM goes down or its BGP session drops, traffic is automatically redistributed to the remaining VMs.

Scaling considerations​

  • You can add or remove VMs without disrupting existing flows on other VMs.
  • The hash distribution is not perfectly even with small numbers of paths. With 2 paths, you get roughly 50/50 distribution. With 3 or more, variance decreases.
  • Each VM should be able to handle the full traffic load independently, since failover shifts all flows from a failed VM to the remaining VMs.

ECMP with Bare Metal​

ECMP works the same way with bare metal servers. Each server establishes a BGP session and announces the same prefix. The NetActuate router distributes traffic across all active sessions.

Bare metal ECMP is common for:

  • High-throughput workloads that need dedicated hardware
  • Latency-critical applications where hypervisor overhead is unacceptable
  • Mixed deployments where bare metal handles primary traffic and VMs handle overflow

Failover with BFD​

Bidirectional Forwarding Detection (BFD) accelerates failover by detecting path failures in sub-second timeframes, compared to the default BGP hold timer of 90 seconds.

With BFD enabled:

  • The router detects a failed path within milliseconds (typically 300ms with 3x 100ms intervals).
  • The BGP session is torn down immediately upon BFD failure.
  • ECMP paths are recalculated and traffic is redistributed to surviving paths.

Without BFD, failover depends on the BGP hold timer. If a VM crashes without gracefully closing the BGP session, the router waits for the hold timer to expire (default 90 seconds) before removing the path.

To enable BFD on your BGP session, add the following to your BIRD2 configuration:

protocol bfd {
    interface "*" {
        min rx interval 100 ms;
        min tx interval 100 ms;
        idle tx interval 300 ms;
        multiplier 3;
    };
}

protocol bgp netactuate {
    ...
    bfd on;
    ...
}

Note: BFD must be enabled on both sides of the session. Contact NetActuate support to confirm BFD is enabled on the router side for your sessions.

Distribution Behavior​

ECMP distribution is based on a hash function applied to packet headers. The specific behavior depends on the router's hash algorithm:

  • 5-tuple hash (most common) -- hashes on source IP, destination IP, protocol, source port, destination port
  • 3-tuple hash -- hashes on source IP, destination IP, and protocol

For TCP and UDP traffic, the 5-tuple hash provides good distribution because each client connection typically uses a unique source port. For protocols without ports (such as ICMP), distribution is based on IP addresses only, which may result in less even distribution.

What ECMP does not do​

  • ECMP does not provide session persistence or sticky sessions. If a path is added or removed, some flows may be reassigned to different paths.
  • ECMP does not perform health checks at the application layer. It relies on BGP session state (and optionally BFD) to determine path availability.
  • ECMP does not guarantee equal traffic volume on each path. It guarantees equal-cost routing; the actual traffic distribution depends on the hash distribution of flows.

Next Steps​

Need Help?​

Contact support@netactuate.com or open a support ticket from the portal.