This is a bible for certified Kubernetes application developers. It covers the features and how to design and build applications to run on Kubernetes.
Updates on 22 Feb 2020
- Updated version to 1.17.3-00 for kubelet, kubeadm and kubectl
- Updated kube-flannel as
extensions/v1beta1 was removed
Creating a cluster
# Setup Docker & Kubernetes repositories
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
# Install Docker & Kubernetes
sudo apt-get update
sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu kubelet=1.17.3-00 kubeadm=1.17.3-00 kubectl=1.17.3-00
sudo apt-mark hold docker-ce kubelet kubeadm kubectl
# Enable iptables bridge call
echo "net.bridge.bridge-nf-call-iptables=1" | sudo tee -a /etc/sysctl.conf
# Make that change take effect immediately
sudo sysctl -p
Setting up the Kube master server
# Initalize the cluster
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
# Setup local kube config
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# Install Flannel networking
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Setting up the Kube node server
# The command is part of output of kubeadm init running on your master
sudo kubeadm join $controller_private_ip:6443 --token $token --discovery-token-ca-cert-hash $hash
Working with Kubernetes Objects
Kubernetes Objects are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of your cluster.
Ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/
List all the object types
kubectl api-resources -o name
# Some examples
Pod
Node
Service
ServiceAccount
...
Example of a Kubernetes Object : deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
Use kubectl apply
kubectl apply -f deployment.yml --record
Working with Pods
A Pod is the basic execution unit of a Kubernetes application–the smallest and simplest unit in the Kubernetes object model that you create or deploy. A Pod represents processes running on your Cluster.
Ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/
Example of pod definition: my-pod.yml
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600']
# Create a pod from the definition file in yaml format
kubectl create -f my-pod.yml
# Edit a pod and re-apply a pod definition
kubectl apply -f my-pod.yml
# Another way to edit a pod (Changes will be taken effect immediately)
kubectl edit pod my-pod
# Delete a pod
kubectl delete pod my-pod
Working with Namespace
Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces.
Ref： https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
By default, a Kubernetes cluster will instantiate a default namespace.
NAME STATUS AGE
default Active 13m
Create a namespace
kubectl create ns development
Spcify a namespace in metadata.namespace attribute
apiVersion: v1
kind: Pod
metadata:
name: my-pod
namespace: my-namespace
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', 'echo Hello World! && sleep 3600']
kubectl create -f my-namespace.yml
Get a list of the namespaces in the cluster
kubectl get namespaces
NAME STATUS AGE
development Active 5m
default Active 1d
kube-system Active 1d
kube-public Active 1d
Specify a namespace when using kubectl get
kubectl get pods -n my-namespace
Specify a namespace when using kubectl describe
kubectl describe pod my-pod -n my-namespace
Working with ConfigMaps
ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable.
Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/
Consuming ConfigMap as Environment Variables
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config-map
data:
myKey: myValue
anotherKey: anotherValue
apiVersion: v1
kind: Pod
metadata:
name: my-configmap-pod
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', "echo $(MY_VAR) && sleep 3600"]
env:
- name: MY_VAR
valueFrom:
configMapKeyRef:
name: my-config-map
key: myKey
Consuming ConfigMap as Mounted Volume
apiVersion: v1
kind: Pod
metadata:
name: my-configmap-volume-pod
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', "echo $(cat /etc/config/myKey) && sleep 3600"]
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: my-config-map
Working with SecurityContexts
A security context defines privilege and access control settings for a Pod or Container.
Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
apiVersion: v1
kind: Pod
metadata:
name: my-securitycontext-pod
spec:
securityContext:
runAsUser: 2001
fsGroup: 3001
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', "cat /message/message.txt && sleep 3600"]
volumeMounts:
- name: message-volume
mountPath: /message
volumes:
- name: message-volume
hostPath:
path: /etc/message
Working with Resource Requests
When you specify a Pod, you can optionally specify how much CPU and memory (RAM) each Container needs. When Containers have resource requests specified, the scheduler can make better decisions about which nodes to place Pods on. And when Containers have their limits specified, contention for resources on a node can be handled in a specified manner. For more details about the difference between requests and limits, see Resource QoS.
Ref: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container
apiVersion: v1
kind: Pod
metadata:
name: my-resource-pod
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600']
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
Working with Secrets
Kubernetes secret objects let you store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a Pod definition or in a container image.
Ref: https://kubernetes.io/docs/concepts/configuration/secret/
Create a secret using a yaml definition
apiVersion: v1
kind: Secret
metadata:
name: my-secret
stringData:
username: YWRtaW4=
password: MWYyZDFlMmU2N2Rm
For best practice, delete the definition after you create it
kubectl apply -f my-secret.yml
rm my-secret.yml
Pass the sensitive data to containers as an environment variable
apiVersion: v1
kind: Pod
metadata:
name: my-secret-pod
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', "echo Hello, Kubernetes! && sleep 3600"]
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password
Working with ServiceAccounts
A service account provides an identity for processes that run in a Pod.
Ref:
Create a ServiceAccount
kubectl create serviceaccount my-serviceaccount
apiVersion: v1
kind: Pod
metadata:
name: my-serviceaccount-pod
spec:
serviceAccountName: my-serviceaccount
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', "echo Hello, Kubernetes! && sleep 3600"]
Working with Multi-Container Pods
Multi-container pods provide an opportunity to enhance containers with helper containers that provide additional functionality
Ref:
apiVersion: v1
kind: Pod
metadata:
name: multi-container-pod
spec:
containers:
- name: nginx
image: nginx:1.15.8
ports:
- containerPort: 80
- name: busybox-sidecar
image: busybox
command: ['sh', '-c', 'while true; do sleep 30; done;']
Working with Probes
A Probe is a diagnostic performed periodically by the kubelet on a Container.
livenessProbe: Indicates whether the Container is running. If the liveness probe fails, the kubelet kills the Container, and the Container is subjected to its restart policy. If a Container does not provide a liveness probe, the default state is Success.
apiVersion: v1
kind: Pod
metadata:
labels:
test: liveness
name: liveness-exec
spec:
containers:
- name: liveness
image: k8s.gcr.io/busybox
args:
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
livenessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe: Indicates whether the Container is ready to service requests. If the readiness probe fails, the endpoints controller removes the Pod’s IP address from the endpoints of all Services that match the Pod. The default state of readiness before the initial delay is Failure. If a Container does not provide a readiness probe, the default state is Success.
apiVersion: v1
kind: Pod
metadata:
labels:
test: liveness
name: liveness-exec
spec:
containers:
- name: liveness
image: k8s.gcr.io/busybox
args:
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
startupProbe: Indicates whether the application within the Container is started. All other probes are disabled if a startup probe is provided, until it succeeds. If the startup probe fails, the kubelet kills the Container, and the Container is subjected to its restart policy. If a Container does not provide a startup probe, the default state is Success.
apiVersion: v1
kind: Pod
metadata:
labels:
test: liveness
name: liveness-exec
spec:
containers:
- name: liveness
image: k8s.gcr.io/busybox
args:
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
startupProbe:
httpGet:
path: /healthz
port: liveness-port
failureThreshold: 30
periodSeconds: 10
Ref:
Working with Container Logging
Ref: https://kubernetes.io/docs/concepts/cluster-administration/logging/
Get the container's logs
kubectl logs <pod name="">
Get the specfic container's logs in a multi-container pod
kubectl logs <pod name=""> -c <container name="">
kubectl logs <pod name=""> > <log name="">
Working with Monitoring Applications
Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/
# Get resource usage for all pods in the default namespace
kubectl top pods
# Get resource usage for a single pod
kubectl top pod <pod name="">
# Get resource usage for all pods in a specific namespace
kubectl top pods -n kube-system
# Get resource usage for nodes
kubectl top nodes
Working with Labels, Selectors and Annotations
Ref:
apiVersion: v1
kind: Pod
metadata:
name: my-production-label-pod
labels:
app: my-app
environment: production
spec:
containers:
- name: nginx
image: nginx
apiVersion: v1
kind: Pod
metadata:
name: my-development-label-pod
labels:
app: my-app
environment: development
spec:
containers:
- name: nginx
image: nginx
# show labels for each pods
kubectl get pods --show-labels
# get pods using equality-based selectors
kubectl get pods -l environment=production
kubectl get pods -l environment=development
# get pods using inequality-based selectors
kubectl get pods -l environment!=production
# get pods using set-based selectors
kubectl get pods -l 'environment in (development,production)'
# get pods using chained multiple selectors with a comma-delimited list
kubectl get pods -l app=my-app,environment=production
# to view existing labels and annotations
kubectl describe pod <pod name="">
Working with Deployment
A Deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments.
Ref: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
kubectl get deployments
kubectl get deployment <deployment name="">
kubectl describe deployment <deployment name="">
kubectl edit deployment <deployment name="">
kubectl delete deployment <deployment name="">
Working with Rolling Update
apiVersion: apps/v1
kind: Deployment
metadata:
name: rolling-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.1
ports:
- containerPort: 80
# Perform a rolling update
kubectl set image deployment/rolling-deployment nginx=nginx:1.7.9 --record
# Explore the rollout history of the deployment
kubectl rollout history deployment/rolling-deployment
kubectl rollout history deployment/rolling-deployment --revision=2
# Roll back to the previous revision
kubectl rollout undo deployment/rolling-deployment
# Roll back to a specific earlier revision by providing the revision number
kubectl rollout undo deployment/rolling-deployment --to-revision=1
Proportional scaling by setting maxSurge and maxUnavailable in the deployment spec
spec:
strategy:
rollingUpdate:
maxSurge: 3
maxUnavailable: 2
Working with Jobs and CronJobs
Ref:
This job calculates the first 2000 digits of pi
apiVersion: batch/v1
kind: Job
metadata:
name: pi
spec:
template:
spec:
containers:
- name: pi
image: perl
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
restartPolicy: Never
backoffLimit: 4
Check the status of Jobs. It may take several seconds to finish.
kubectl get jobs
By running kubectl get pods, you should see a new pod with STATUS Completed
kubectl get pods
Check the output
kubectl logs <pod_name>
A cronjob example
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: hello
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox
args:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
restartPolicy: OnFailure
List and check the status of CronJobs
kubectl get cronjobs
Working with Services
Ref:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
# Get services
kubectl get svc
# Get endpoints
kubectl get endpoints my-service
Working with Network Policies
A network policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints.
Ref: https://kubernetes.io/docs/concepts/services-networking/network-policies/
In order to use NetworkPolicies in the cluster, we need to have a network plugin called canal
wget -O canal.yaml https://docs.projectcalico.org/v3.5/getting-started/kubernetes/installation/hosted/canal/canal.yaml
kubectl apply -f canal.yaml
Create a sample nginx pod
apiVersion: v1
kind: Pod
metadata:
name: network-policy-secure-pod
labels:
app: secure-app
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
Create a client pod used to test network access to the nginx pod
apiVersion: v1
kind: Pod
metadata:
name: network-policy-client-pod
spec:
containers:
- name: busybox
image: radial/busyboxplus:curl
command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
Get the cluster ip address of the nginx pod
kubectl get pod network-policy-secure-pod -o wide
By executing the below command, you should see the network is accessible
kubectl exec network-policy-client-pod -- curl <cluster_ip_address>
Create a network policy to restricts all access to the secure pod except pods with allow-access: "true" label
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: my-network-policy
spec:
podSelector:
matchLabels:
app: secure-app
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
allow-access: "true"
ports:
- protocol: TCP
port: 80
egress:
- to:
- podSelector:
matchLabels:
allow-access: "true"
ports:
- protocol: TCP
port: 80
Apply before network policy and execute the curl command again, you should see the network is not accessible
kubectl exec network-policy-client-pod -- curl <cluster_ip_address>
Get more info about NetworkPolicy in the cluster
kubectl get networkpolicies
kubectl describe networkpolicy my-network-policy
Edit the client pod
kubectl edit pod network-policy-client-pod
under metadata, add
labels:
allow-access: "true"
Re-run the curl command again, the secure pod is accessible from the client pod
Working with Volumes
On-disk files in a Container are ephemeral, which presents some problems for non-trivial applications when running in Containers. First, when a Container crashes, kubelet will restart it, but the files will be lost - the Container starts with a clean state. Second, when running Containers together in a Pod it is often necessary to share files between those Containers. The Kubernetes Volume abstraction solves both of these problems.
Ref: https://kubernetes.io/docs/concepts/storage/volumes/
apiVersion: v1
kind: Pod
metadata:
name: volume-pod
spec:
containers:
- image: busybox
name: busybox
command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
volumeMounts:
- mountPath: /tmp/storage
name: my-volume
volumes:
- name: my-volume
emptyDir: {}
Working with PersistentVolumes and PersistentVolumeClaims
PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) provide a way to easily consume storage resources, especially in the context of a complex production environment that uses multiple storage solutions.
Ref:
kind: PersistentVolume
apiVersion: v1
metadata:
name: my-pv
spec:
storageClassName: local-storage
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 512Mi
kubectl get pv
kubectl get pvc
kind: Pod
apiVersion: v1
metadata:
name: my-pvc-pod
spec:
containers:
- name: busybox
image: busybox
command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
volumeMounts:
- mountPath: "/mnt/storage"
name: my-storage
volumes:
- name: my-storage
persistentVolumeClaim:
claimName: my-pvc
