Tuesday 28 January 2020

How to fix Insufficient pods issue when deploying to Amazon EKS?

Recently I tried to deploy a nodeJS application to Amazon EKS and found that the deployment was not ready and the pod was stuck at Pending state indefinitely. I had no idea what's happening. In order to troubleshoot, I increased kubectl output verbosity to ``--v=9``. For more: [Kubectl output verbosity and debugging](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-output-verbosity-and-debugging). To get more details, I ran ``kubectl describe pod`` and got the below message: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 17s (x14 over 18m) default-scheduler 0/1 nodes are available: 1 Insufficient pods. ``` I was wondering the maximum number of pods per each node, so I ran ``` kubectl get nodes -o yaml | grep pods ``` It returned ``` 4 ``` Then, I wanted to know how many pods which were currently running. ``` kubectl get pods --all-namespaces | grep Running | wc -l ``` I also got ``` 4 ``` From [Architecting Kubernetes clusters — choosing a worker node size](https://learnk8s.io/kubernetes-node-size), we know the below info: ``` On Amazon Elastic Kubernetes Service (EKS), the maximum number of pods per node depends on the node type and ranges from 4 to 737. On Google Kubernetes Engine (GKE), the limit is 100 pods per node, regardless of the type of node. On Azure Kubernetes Service (AKS), the default limit is 30 pods per node but it can be increased up to 250. ``` As more pods need to be provisioned, the maximum number of pods are not enough in this case. As I was just working on a demonstration, so I chose a small node type ``t2.micro`` to minimise the cost. How to increase this value? This is calculated from [AWS ENI documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI). The formula is ``` N * (M-1) + 2 ``` where ``` N is the number of Elastic Network Interfaces (ENI) of the instance type M is the number of IP addresses of a single ENI ``` So I was using ``t2.micro``, the calculation is ``2 * (2-1) + 2 = 4``. AWS also provides a mapping file [here](https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt) so that you don't need to calculate yourself. The final solution is to change the node type from ``t2.micro`` to ``t3.small`` in my case.

No comments:

Post a Comment

A Fun Problem - Math

# Problem Statement JATC's math teacher always gives the class some interesting math problems so that they don't get bored. Today t...