How to deploy a k8s cluster on VmWare ESX

k8s, vmware, ansible, metalLB, prometheus, grafana, alertmanager, metrics-server, harbor, clair, notary

If you’re short on resources in your home lab and simply can’t deploy an OpenStack private cloud to play around with, this tutorial will walk you through setting up a highly-available k8s cluster on VMware ESX vms. The principles used to deploy them very much resemble those used to deploy OpenStack instances so you’ll be able to quickly and effectively walk through the very first step in this tutorial.

The k8s cluster we’ll be deploying will have 3x master and 3x worker nodes and will use metalLB for bare-metal load balancing, prometheus, grafana and alertmanager for monitoring, metrics and reporting and harbor with clair and notary for image repository, signing and scanning.

We will be:

  • checking out and customising ansible-deploy-vmware-vm
  • creating vms from templates
  • setting up:
    • the k8s cluster via kubespray
    • metalLB
    • prometheus, grafana and alertmanager
    • metrics-server
    • harbor
    • notary and clair

What we won’t be doing is:

  • setting up VMwareESX (see here)
  • creating a VMware template (see here)

INFO: the VMware templates were made after an Ubuntu 18.04 vm.


STEP 1 — deploy multiple VMWare virtual machines from a template

Deploy the vm’s via Ansible. First and foremost, checkout ansible-deploy-vmware-vm and cd to it.

Second, we need to tell ansible how to connect to our VMware ESX cluster. Edit or create the answerfile.yml and fill in the self explanatory blanks:

Define the vms we want to deploy. Sample vms-to-deploy:

Define the playbook. Sample deploy-k8s-vms-prod.yml:

Let’s deploy the vms:


STEP 2

Deploy the k8s cluster on top of the created vms.

sample hosts-prod.yaml:

sample inventory.ini (make sure the inventory file contains the vm’s proper name (ie. the ones defined under “ansible-deploy-vmware-vm/vms-to-deploy”))

Run the playbook to create the cluster:


STEP 3

MetalLB is a bare-metal load balancer for k8s that makes your current network extend into your k8s cluster.

Connect to one of the master nodes over ssh and setup your environment:

Deploy metalLB by performing the following on one of the master nodes:

MetalLB remains idle until configured. As such, we need to define an IP range the k8s cluster can use and that is outside of any DHCP pool.

Check the status of the pods with:


STEP 4

Install prometheus, grafana and alertmanager:

In order to access the dashboards via the LoadBalancer IPs, we need to change a few service types from “ClusterIP” to “LoadBalancer”:

In order to access the web interface of these services, use the following ports:

– grafana -> 3000 (default usr/pass is admin/admin)
– prometheus -> 9090
– alertmanager -> 9093

STEP 5

Installing the metrics-server or how to get “kubectl top nodes” and “kubectl top pods” to work.


STEP 6

Installing Harbor with clair and notary support.

ssh to your harbor vm and install docker and docker-compose:

Logout and login and check docker:

Time to generate SSL certificated:

Download and install Harbor:

Edit harbor.yml and change a few options:

Finally, install Harbor, enabling clair and notary:

Configure the docker daemon on each of your worker nodes and

Login to a k8s master node and create a secret object for harbor:

Login to the harbor web-interface and create a new repository called “private”.

To deploy images to Harbor, we need to pull them, tag them and push them. From the harbor machine (or any other that has the certificates in place as above), perform the following:

Deploy the kuard app on k8s. ssh to a node where you have access to the cluster and create kuard-deployment.yaml:

Apply it.


STEP 7

Signing docker images with Notary. Install it first:

If needed, copy the “ca.crt” to the client machine you’re working from.

Check if you can connect to the harbor server:

You should get something like this:

Now pull an image from docker hub and tag it but don’t push it just yet.

Let’s enable the Docker Content Trust and then push the image. Please note that when first pushing a signed image, you will be asked to create a password.

Ok, so now the Docker image is pushed in our Registry server and it is signed by the Notary server. We can verify

Test that you can pull from docker hub and harbor:

For using a completely private repository, leave the “DOCKER_CONTENT_TRUST_SERVER” and the “DOCKER_CONTENT_TRUST” environment variables set, on all the k8s cluster machines.

STEP 8

Clair is an open source project for the static analysis of vulnerabilities in application containers. To enable it go to the Harbor web-interface and click on “vulnerability” -> “edit” and select the scan frequency. Save and also click “scan now” if it’s your first time doing this.

How to pass the CKA and CKAD exam

Last week, I passed the Certified Kubernetes Administrator (CKA) and Certified Kubernetes Application Developer (CKAD) exam. I had very little knowledge and experience with Kubernetes but after spending only 3 weeks preparing, I managed to successfully pass both exams.

Below, I will detail the process I took in going from zero to hero.


1) Take an online course
It goes without saying that if you have close to zero knowledge on the subject at hand, some guidance to kick things off doesn’t hurt. I can wholeheartedly recommend the LinuxAcademy CKA, CKAD and Kubernetes the Hard Way courses. I found them to be very well structured and well worth the money. Take your time and make sure you know the material well. Redo the hands-on labs until you can recite them by heart. The $49,99 you pay per month is more than enough to help you pass both exams.


2) Take free practice exams
Once you’re done with the courses and know them like the back of your hand, step outside of your comfort zone and challenge yourself further. I’ve managed to find a few other practice exams that helped me a lot.

There’s a short one from the Kubernetes App Developer (CKAD) course from matthewpalmer.net that has a challenge which I found to be quite interesting.

Another free resource I found was the last point of the the Kubernetes certification course over at kodekloud.com. It’s a medium sized exercise that will walk you through many different aspects of k8s. I really enjoyed this one so I can highly recommend it.


3) Do free exercises
Now we get to the sauce. I found https://github.com/dgkanatsios/CKAD-exercises to be an amazing resource in checking your overall knowledge. They are very good exercises to go through, whether you are taking the CKA or the CKAD exam.
Additionally, https://github.com/wiewioras/k8s-labs is one amazing piece of exercise that I recommend to everyone.


4) Prep notes and tips
See https://github.com/twajr/ckad-prep-notes. Make sure to check the kubectl commands from the “core concepts”. They will save you a lot of time when creating resources and time is of the essence. With 24 questions and 3 hours for the CKA, that gives you 7.5 minutes per question. The CKAD is slightly more demanding with 19 questions and 2 hours, giving you an average of 6.3 minutes per question. Both exams are designed in such a way that you don’t have the luxury of spending time checking the docs; you really need to know what you’re doing.


In summary
– take a course
– do as many practice exams as you can
– do as many exercises as you can
– be VERY fast in creating resources
– know as much as possible about k8s


And finally, here are my certs:


Last words:
Kubernetes is a new and cool technology. My take is that it will become the new cloud platform. It is without a shadow of a doubt, the most exciting technology I have seen in the last 10 years. I believe it will become a standard.