Vault is a cloud native secrets engine that can protect your secrets and allow applications to retrieve them giving you a central place to store secrets. This video by Armon Dadger Hashicorp CTO gives a really good oversight Vault.
To run a HA Vault cluster in Kubernetes requires quite a few steps so I wanted to document how to approach it. Disclaimer this is by far from a guide on how to deploy a production grade vault cluster. It is aimed at giving you a way to deploy a HA cluster as starting point or if you are link me and want to deploy a Vault cluster on your home Kubernetes cluster for fun!
Step 1- Create Vault Namespace
The official vault helm chart which we are going to use to install Vault sets up Vault in a vault namespace. In order to get our cluster ready we need to create the vault namespace using the following command:
kubectl create namespace vault
Step 2 – Generate Certificates
In order to run a Vault cluster you use a TLS certificate for vault to use in order for it to expose its https endpoints. There are a few approaches we can take but the easiest is to generate our own root CA and then use that root CA to sign a certificate. We can then add the root CA to the trust store of the machine we want to use to access the cluster so that it will trust the vault endpoints.
To generate the root CA first we need to create a new private key:
openssl genrsa -out rootCAKey.pem 4096
Create the certificate signing request with this key:
openssl req -x509 -sha256 -new -nodes -key rootCAKey.pem -days 10000 -out rootCACert.pem
Note I’m using an expiry of around 30 years for the root CA.
If you are doing this for real then you would probably want to look at setting up an intermediary but as this is just for fun we are going to sign the server certificate directly with the root CA. Lets generate a private key for the server:
openssl genrsa -out vault.key.pem 4096
For the vault server certificate, we will want to configure a few subject alternative names in order for everything to work correctly. Firstly, we can come up with a domain that we will later alias to point to our vault cluster, I’m going to use
vault.local for that. Next, we need to create subject n subject alternative names in the format of
vault-X.vault-internal where n is the number of vault nodes we want to run and x is the node number. For example, if we run 3 nodes then we would need to create a certificate with the 3 addresses vault-0.vault-internal, vault-1.vault-internal and vault-2.vault-internal. These internal endpoints are used by the vault nodes to talk to one another.
In order to create the subject alternative names we need to create a config file like the one below:
req_extensions = v3_req
distinguished_name = dn
prompt = no
CN = vault.local
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
DNS.1 = vault.svc.cluster.local
DNS.2 = vault-0.vault-internal
DNS.3 = vault-1.vault-internal
DNS.4 = vault-2.vault-internal
DNS.5 = vault.local
With this config file in place we can now create the CSR for our vault server certificate:
openssl req -new -key vault.key.pem -sha256 -out vault.csr -config cert.config
To generate the server certificate we just need to sign the CSR we just generated with the root CA:
openssl x509 -req -sha256 -in vault.csr -CA rootCACert.pem -CAkey rootCAKey.pem -CAcreateserial -out rbaServerCert.pem -days 365 -extfile cert.config -extensions 'v3_req'
We have now created our two certificates one for the root CA and one for the vault server itself.
Step 3 – Upload Certificates
In order for Vault to be able to use the certificates, we have generated we need to load them into the cluster as kubernetes secrets. We need to only upload the certificate for the root CA so that we can set Vault to trust any certificate that is signed by this root. This is needed so that the Vault instances trust each other and can talk over TLS to each other.
To upload the root CA cert we can use an opaque secret:
kubectl --namespace='vault' create secret opaque rootCA ./rootCACert.pem
For the server certificate we need to upload both the certificate and the private key in order for our Vault container to use that certificate to host its TLS endpoint. To do this we can use the command:
kubectl --namespace='vault' create secret tls tls-server --cert ./vault.cert.pem --key ./vault.key.pem
This command will upload the server certificate using the Kubernetes inbuilt tls secret type which will create a secret with the key and certificate as data under tls.crt and tls.key.
Step 4 – Set up Local Storage Volumes
In order to run a Vault cluster we need to choose a storage backend. The solution with the least moving parts is to configure vault to use an inbuilt raft storage backend. To make this work we have to create persistent volumes for each vault container to use. To solve this we are going to create a persistent volume on disk on each node, we can use node affinity to pin the volume so that it is always on the same node.
To set this up we first have to configure local storage by creating the local storage class:
kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: local-storage reclaimPolicy: Delete provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer
With the file above created, we can create the storage class using the
kubectl apply -f storageclass.yaml command.
Next, we have to create the volumes themselves. I have 3 worker nodes in my cluster and am going to run a 3 node vault cluster so I am going to create 3 persistent volumes by using the code:
- key: kubernetes.io/hostname
In the above code you will need to change the
worker01 to the name of your worker node. This is the line that makes the volume only available on that node. You will have to copy and paste this file for as many nodes as you are going to run, altering the name and the node affinity clause each time. Then use the
kubectl apply -f <filename> to create the volumes on the cluster.
Before the volumes will work you have to make sure the path you set physically exists on disk, so be sure to create that on each of your worker nodes.
Step 5 – Install Vault
With all of the above in place we are finally ready to install Vault! We are going to use helm to install Vault, so the first thing we need to do is add the hashicorp repo to helm:
helm repo add hashicorp https://helm.releases.hashicorp.com
Next we need to create the following
This values.yaml configures 3 vault nodes and mounts the certificates using the secrets we created earlier. I have tuned down the memory requirements of each node as the VMs I’m running are pretty small.
With all of this in place we can get Vault installed:
helm install vault hashicorp/vault --namespace vault -f values.yaml
Once vault is installed we can check its running by listing the pods:
kubectl get pods -n vault
This should print out the 3 vault pods
vault-2. To get the cluster up and running we need to initialise one of the vaults and then save the unseal keys and root token and add those to the other vaults.
To initialise one of the vaults first open a terminal on the container using:
kubectl exec -it --stdin=true --tty=true vault-0 /bin/ash
Now we need to turn off TLS verification as we are accessing vault on localhost, do this by setting up VAULT_SKIP_VERIFY environment variable:
Now we can initialise vault using:
vault operator init
Make sure you save the unseal keys and root token somewhere safe and then unseal this vault by running
vault operator unseal 3 times and providing 3 of the unseal keys. Then ssh onto the other two vault containers using the kubectl exec command given above but changing the pod name to
vault-2, export the VAULT_SKIP_VERIFY variable and then run
vault operator unseal 3 times to unseal both of the other vaults.
Step 6 – Configuring Your Machine To Use Vault
To finish our setup we need to configure our machine to be able to trust our new Vault cluster. To do that first load your root CA certificate into your machine’s trust store. How to do this will vary based on which operating system you are on. For Mac you can follow these instructions.
I have Metal LB setup as a load balancer on my cluster as explained in this post. Metal LB creates a virtual IP address that will load balance across the cluster and route to vault. I have added an entry to my
/etc/hosts file with
vault.local pointing at the virtual IP address for the Metal LB IP that points to vault.
Next, install vault on your machine, instructions to do this can be found on the vault install page. With all of that configured we can set the vault address for the cli to use by running:
Now we can successfully use our new vault HA cluster, test it out by running
vault status you should see the status of the cluster.