kube-egress-gateway

kube-egress-gateway provides a scalable and cost-efficient way to configure fixed source IP for Kubernetes pod egress traffic on Azure. kube-egress-gateway components run in kubernetes clusters, either managed (Azure Kubernetes Service, AKS) or unmanaged, utilize one or more dedicated kubernetes nodes as pod egress gateways and route pod outbound traffic to gateway via wireguard tunnel.

Compared with existing methods, for example, creating dedicated kubernetes nodes with NAT gateway or instance level public ip address and only scheduling pods with such requirement on these nodes, kube-egress-gateway provides a more cost-efficient method as pods requiring different egress IPs can share the same gateway and can be scheduled on any regular worker node.

Design

• Design doc provides details about how kube-egress-gateway works.

Installation

• Follow Installation guide to configure your Kubernetes cluster and install kube-egress-gateway components.

Usage

Operating system requirements

• Only Linux (Ubuntu or Azure Linux) based nodes are supported. Other Linux distributions have not been tested but may work. Windows node or pod support is not available at this time.

Deploy a Static Egress Gateway

Public IP Mode (Default)

To deploy a static egress gateway with public IP, you need to create a StaticGatewayConfiguration CR:

apiVersion: egressgateway.kubernetes.azure.com/v1alpha1
kind: StaticGatewayConfiguration
metadata:
  name: myStaticEgressGateway
  namespace: myNamespace
spec:
  gatewayVmssProfile:
    vmssResourceGroup: myResourceGroup
    vmssName: myGatewayVMSS
    publicIpPrefixSize: 31
  provisionPublicIps: true
  publicIpPrefixId: /subscriptions/mySubscriptionID/resourcegroups/myResourceGroup/providers/Microsoft.Network/publicipprefixes/myPIPPrefix
  defaultRoute: staticEgressGateway
  excludeCidrs:
    - 10.244.0.0/16
    - 10.245.0.0/16

Private IP Mode (Preview - AKS 1.34+)

For private IP egress (requires irtual Machines node pools):

apiVersion: egressgateway.kubernetes.azure.com/v1alpha1
kind: StaticGatewayConfiguration
metadata:
  name: <gateway-config-name>
  namespace: <namespace>
spec:
  gatewayNodepoolName: <nodepool-name>
  provisionPublicIps: <true|false>
  publicIpPrefixSize: </28-/31>
  excludeCidrs:            # (Optional) CIDRs to exclude from routing through gateway
    - 10.0.0.0/8
    - 172.16.0.0/12
    - 169.254.169.254/32

Note: Private IP mode requires User-Defined Routes or ExpressRoute for internet connectivity.

Configuration Reference

StaticGatewayConfiguration is a namespaced resource, meaning a static egress gateway can only be used by pods in the same namespace. There are two required configurations:

• gatewayNodepoolName: Name of the Azure VirtualMachineScaleSet (VMSS) or Virtual Machines (VMs) to be used as gateway nodepool.
• publicIpPrefixSize: Length of the public IP prefix to be installed on the gateway nodepool as egress (only applicable when provisionPublicIps: true). In above example, 31 means a /31 pip prefix, which contains 2 public IPs, will be installed. Note that gateway nodepool instance count cannot exceed this size. The gateway nodepool can only have 1 or 2 instances if public IP prefix size is 31. Likewise, at most 4 instances are allowed if public IP prefix size is 30. Otherwise, kube-egress-gateway operator will report error. At the time of writing, Azure only supports prefix sizes /28-/31. For private IP mode, this field determines the maximum number of gateway nodes by pre-allocating that many private IPs.
• provisionPublicIps: Determines the egress mode:
  • true (default): Public IP mode - A public IP prefix will be associated with the gateway nodepool secondary IPConfiguration. Egress traffic uses public IPs directly to reach the internet.
  • false: Private IP mode - Gateway nodes use private IP addresses from the cluster's VNet subnet. Requires proper network routing (User-Defined Routes, Azure Firewall, or ExpressRoute) for outbound connectivity. Gateway nodepool must use VM-based nodes for stable private IP assignment.

Three optional configurations:

• publicIpPrefixId: BYO public IP prefix is supported. Users can provide Azure resource ID of their own public IP prefix in this field. Make sure kube-egress-gateway operator has access to the prefix. If not provided and provisionPublicIps is set to true, a system generated prefix will be provisioned.
• defaultRoute: Enum, either staticEgressGateway or azureNetworking. Set it to be staticEgressGateway if traffic by default should be routed to the egress gateway or azureNetworking if traffic should be routed to pods' eth0 by default like regular pods. Default value is staticEgressGateway.
• excludeCidrs: List of destination network CIDRs that should bypass the default route and flow via the other network interface. That is, if defaultRoute is staticEgressGateway, cidrs set in excludeCidrs will be routed via pod's eth0 interface. For example, traffic within the cluster like pod-pod traffic and pod-service traffic should not be routed to the egress gateway and can be set here. On the other hand, if defaultRoute is azureNetworking, then only cidrs set in excludeCidrs will be routed to the egress gateway.

kube-egress-gateway reconcilers manage the setup and resources and report the egress IP information in StaticGatewayConfiguration status:

apiVersion: egressgateway.kubernetes.azure.com/v1alpha1
kind: StaticGatewayConfiguration
metadata:
  name: myStaticEgressGateway
  namespace: myNamespace
spec:
  ...
status:
  egressIpPrefix: 1.2.3.4/31 # example public IP prefix output (public IP mode), this will be pods' egress IPNet

When provisionPublicIps: false (private IP mode), egressIpPrefix will be a comma-separated list of private IPs configured on the corresponding gateway nodepool instance secondary ipConfigurations, e.g. 10.0.1.8,10.0.1.9.

Deploy a Pod using Static Egress Gateway

Constructing a pod to use a static egress gateway is simple: just add pod annotation kubernetes.azure.com/static-gateway-configuration: <StaticGatewayConfiguration name>. Only name is required here because kube-egress-gateway CNI plugin always assume the gateway is in the same namespace as the pod. Note that existing pods must be recreated to enable egress gateway because CNI plugin can only take effect when pod is being created. See sample pod here.

Troubleshooting

Refer to troubleshooting guide and known issues.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.