Kubernetes Networking with EKS

Kubernetes Pod Networking
Traditional Docker networking allows all containers on a host to reach one another over private IP addresses on a virtual bridge on each host. Port forwarding can also be configured on the host to provide external access to containers on an as needed basis. This native Docker model places a significant burden on the deployment of applications to containers.
Kubernetes, in the other hand, has a more complex networking implementation with the purpose of simplify the process of converting applications to run in a container environment. 

Kubernetes Container Network Interface (CNI)
The CoreOS team created a pluggable networking interface for containers, called CNI which was adopted in 2016 by the Kubernetes team. It now forms the basis for the kubernetes networking model. 

Fundamental to Kubernetes is the concept of Pods as a group of related containers. Pods share the same lifecycle, are installed together on the same node, and can talk to one another in the same ways as they could when installed onto a single VM.
In terms of networking this means that: 

• Pods in a cluster share a single IP address.
• Containers in a cluster see same IP for themselves as other pods/nodes in the network use to reach that container.
• Without network policies that forbid such communication, all pods can communicate with one another freely, without Network Address Translation (NAT). 
  

The result is a flat network space where networking between containers on a Pod occurs over localhost, where all Pods and nodes can communicate with one another without special work.

Network Policy
For security reasons a fully accessible flat network is often not desirable. Kubernetes offers a solution to this problem with network policies that can be used to restrict network access between Pods or between collections of Pods in a Kubernetes namespace. 

Kubernetes clusters can be built with a wide variety of CNI plugins, not all of which implement Network Policies. In order to limit the impact of an intrusion it is generally wise to implement at least some network policy rules to prevent cross service escalation. One of the most common ways to do this is to limit teams to a particular Kubernetes namespace and use policies to isolate network traffic along namespace boundaries. 

Beyond the basics of Pod connectivity, Kubernetes offers additional constructs that help create flexible distributed systems but are generally independent of your choice of Pod networking technologies.  

AWS EKS and Kubernetes Networking
The whole point of EKS is making Kubernetes operation on AWS Amazon to be easy. For that, when it comes to networking, EKS brings a built-in VPC CNI integration originally without a network policy engine. But EKS has now implemented hooks into VPC CNI that allows users to install the Calico policy engine on top of the pre-installed VPC CNI plugin. 

Advantages of VPC CNI
Amazon native networking provides several significant advantages over some of the more traditional overlay network solutions for Kubernetes: 

• Raw AWS network performance.
• Integration of tools familiar to AWS developers and admins, like AWS VPC flow logs and Security Groups, allowing users with existing VPC networks and networking best practices to carry those over directly to Kubernetes.
• The ability to enforce network policy decisions at the Kubernetes layer if you install Calico. 

If the infrastructure team has significant experience with AWS networking, and/or the application deployed is sensitive to network performance all of this makes VPC CNI very attractive. 

There are two important details to keep in mind about the VPC CNI. First is that no overlay network is used for pods therefore, the pod has IPs in the same CIDR the clusters nodes have. Second that for providing IPs from the VPC range AWS attaches additional ENIs (Elastic Network Interfaces) to thew cluster nodes. Keep in mind that an ENI can have multiple IPs attached to it. One IP is called the primary and the other are called secondary IPs. 

Disadvantages of VPC CNI
On the other hand, there are a couple of limitations that may be significant. There are three primary reasons why you might instead choose an overlay network instead of a AWS VPC CNI. 

• Pod density limitations.
• Need for encryption on the network.
• Multicast requirements. 

VPC CNI Pod Density Limitations
First, as we said above, the VPC CNI plugin is designed to use/abuse ENI interfaces to get each pod in your cluster its own IP address from Amazon VPC directly. This means that we will be network limited in the number of pods that we can run on any given worker node in the cluster. 

In order to understand these limits, we will need to dig into how VPC CNI works.
When we attach ENIs to a given EC2 instance (and a cluster node is that) the primary IP for each ENI is used for cluster communication purposes. New pods are assigned to one of the secondary IPs for that ENI. VPC CNI has a custom daemonset that manages the assignment of IPs to pods ( a daemonset is just a pod that has the particularity of running on each nodes of the cluster but only one instance is permitted). Because ENI and IP allocation requests can take time, this l-ipam daemon creates a warm pool of ENIs and IPs on each node and uses one of the available IPs for each new pod as it is assigned. This yields the following formula for maximum pod density for any given instance:
ENIs * (IPs_per_ENI -1 )
Each instance type in Amazon has unique limitations in the number of ENIs and IPs per ENI allowed. For example, an m5.large worker node allows 25 pod IP addresses per node at an approximate cost of $2.66/month per pod.
Stepping up to an m5.xlarge allows for a theoretical maximum of 55 pods, for a monthly cost of $2.62, making the m5.large the more cost effective node choice by a small amount for clusters bound by IP address limitations. 

Additional Overhead Considerations
And if that set of calculations is not enough, there are a few other factors to consider. Kubernetes clusters generally run a set of services on each node. VPC CNI itself uses an l-ipam daemonset, and if you want Kubernetes network policies, calico requires another. Furthermore, production clusters generally also have daemonsets for metrics collection, log aggregation, and other cluster-wide services. Each of these uses an IP address per node.
So now the formula is:
(ENIs * (IPs_per_ENI -1 ) -1*DaemonSets
This makes some of the cheaper instances on Amazon completely unusable because there are no IP addresses left for application pods.
On the other hand, Kubernetes itself has a supported limit of 100 pods per node, making some of the larger instances with lots of available addresses less attractive. However, the pod per node limit is configurable. 

Encryption and Overlay Networks
If there is a requirement for encrypted transport within the network, this can be provided by a CNI-overlay networking solution. VPC provides network-level isolation via multiple subnets and custom routing rules. Traffic can be further isolated using security groups but given the current VPC CNI limitations that all ENI’s on a node be launched with the same subnet, and security group, the need to encrypt traffic at the pod level is still a real concern for many.
Weave Net overlay network from Weaveworks is one of the most popular options and is one of the few overlays network plugins that offers pod network encryption. 

Multicast Networks on AWS
Another requirement that can simplify networking choices is multicast support. AWS VPC does not support multicast, and again Weave Net is one of the only overlay networks that provides multicast support. 

Cloud Portability
The fourth advantage often cited for using an overlay network on Amazon is cloud portability. Hybrid cloud, disaster recovery, and other requirements often push users away from custom cloud vendor solutions and towards open solutions.
However, in this case the level of lock-in is quite low. The CNI layer provides a level of abstraction on top of the underlying network, and it is possible to deploy the same workloads using the same deployment configurations with different CNI backends. However, as you can see from the above discussion, the VPC CNI plugin on Amazon does impose some complications and restrictions and requires a good bit of Amazon VPC knowledge, which is not portable. 

Overlay Networking Options
There are a number of different overlay networking possibilities on Amazon, and Weaveworks is the sponsor of Weave Net, one of the more popular options. As mentioned earlier Weave Net is one of the few if not the only option if we need either encryption or multicast support and is a solid community supported overlay networking option. 

And if there is a need to avoid the overhead of an L2 solution (like Weave Net, Flannel, etc.) we can use Calico networking, which uses BGP to program routing tables and avoids packet encapsulation. This allows for lower overhead, but at some cost in terms of IP addresses, since it assigns a block of addresses to each node in the cluster. It has algorithms for reducing the address fragmentation this causes, but it nonetheless uses more IP’s than traditional overlay approaches which generally only consume the IP addresses they actually use. 

Conclusions
So, what is the right solution for a AWS cluster? Here we have some guidelines from Mark Ramm-Christensen for helping us to select a solution.
Do you have workloads that are particularly sensitive to network performance?
If so, the VPC CNI plugin will provide the best networking performance.
Do you have lightweight workloads that will push the pod density limits of vpc-cni?
If so, you should look at an overlay network provider.
Are you going to push up against the IP address limits of VPC?
If so, an overlay network makes sense.
Do you need multicast or encryption?
If so, Weave Net is likely your best choice. 

REFERENCES
Mark Ramm-Christensen. AWS and Kubernetes Networking Options and Trade-offs. DZone.com. Feb. 05, 19
Kubernetes Documentation. Kubernetes.io/docs
AWS EKS Documentations. docs.aws.amazon.com/eks/latest/userguide
Weaveworks Weave Net. weave.works/docs/net/latest/kubernetes/kube-addon/#eks