VPC CNI for k8s

  • ENI(Elastic Network Interface)
    ENI could be considered as a network card for an EC2 instance. A single EC2 instance can have multiple ENIs attached, one primary plus several secondaries.
    An ENI itself can hold multiple IP addresses.
    When an ENI is created and attached to a EC2 node, it reserves a bunch of IPv4 addresses from the Subnet IP address poll.

  • AWS VPC CNI
    CNI consists of a specification and libraries for writing plugins to configure network interfaces in Linux containers
    AWS VPC CNI include 2 components, CNI plugin and a long-running node-Local IP Address Management(IPAM) daemon, adding a pod on every node that keeps track of all ENIs and IPs attached to the instance.

    1. CNI plugin
      CNI plugin wire up host’s and pod’s network statck when called. it executes commands from kubelet to add/delete network interfaces for Pods or containers on the host.
  1. IPAM
    For IPAM, it maintains a local warm-pool of available IP addresses on the host(node) and assigns an IP address to a Pod when it gets scheduled on the node
    kubelet –> CNI Plugin –>(gRPC)–> IPAM(avaliable Warm Pool) 
    WARM_ENI_TARGET
WARM_IP_TARGET
MINIMUM_IP_TARGET

    If the worker nodes are running in small subnets with a limited number of IPs available, using WARM_IP_TARGET together with MINIMUM_IP_TARGET is an option. MINIMUM_IP_TARGET is the “floor” of how many IPs to keep around on each node. If you plan to run around 10 pods, the advice is to set MINIMUM_IP_TARGET slightly higher than that, like 12, and have WARM_IP_TARGET=2.

  • ENI Capacity
    for a single EC2, there are two max numbers related to ENI:
    1. max num of ENIs per EC2 instance
    2. max num of private IPv4 addresses per ENI
  • Maintenance of Warm Pool for Local IP Address
    1. Whenever the num of available IP address goes below a configured min threshold, IPAM will
      a. create a new ENI and attach it to instance
      b. allocate all available IP address on this new ENI
      c. once these IP addresses become available on the instance, add these IP addressed to warm-pool
  • options
    WARM_IP_TARGET
    https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md

refer