Vault Auto-unseal using Transit Secret Engine on Kubernetes

Theoretical introduction

  • Intialzie Vault
  • Unseal Vault

Unsealing has to happen every time Vault starts. This is because Vault starts with sealed state in which it can’t read storage because it doesn’t know how to decrypt it.

Initialzing happens once when the server started with new backend. During initialization Vault generates bunch of keys:

  • unseal keys
  • encryption keys
  • root token

As you can easily guess, unseal keys generated during initizlization are used for unsealing the Vault.

We can distinguish three options for unsealing the Vault:

  • Manual unsealing
  • Auto-unseal
  • Transit Unseal — de facto one of Auto-unseal option

Manual unsealing

Auto-unseal

Services and devices supported with Auto-unseal you can find in official docs.

Transit Auto-unseal

Transit Auto-unseal setup

# namespace for Vault central
kubectl create ns vault

# namespace for Vault with Transit Auto-unseal
kubectl create ns vault-a

Let’s start with installation of Vault Central. Save below Helm chart values to install Vault in HA mode with default manual unsealing.

server:
affinity: ""
ha:
enabled: true
replicas: 2
raft:
enabled: true
# change namespace to Vault central
kns vault

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault -f vault-central-helm-values.yml

With the below initialization we split the root key into 4 shares (unseal keys). We can also set how many keys are required to reconstruct the root key, which is then used to decrypt the Vault’s encryption key.

kubectl exec vault-0 -- vault operator init \
-key-shares=4 \
-key-threshold=2 \
-format=json > vault-central-keys.json

Once is applied Vault generates unseal keys encrypted with base64 and hex and root token responsible for authentication against Vault. We will use it later.

{
"unseal_keys_b64": [
"4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T",
"miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK",
"EyVw9nQH/T+3zsa4HbPJ2s15l6B5MizMKQlKqs9taFzX",
"zc7eU9MEvy9AaV4FPSQe7Jla2LcqSjS8KNPFDlQs0Rcg"
],
"unseal_keys_hex": [
"e169b9058b0d6a5fb331bb1bddec0d6e2eb32ed288397cf72fe3455fb8f0d3fdd3",
"9a26ac837d4598f26ac7d2eb9cf69512e1badfd7ef8c0a99177829e1994ac3ec0a",
"132570f67407fd3fb7cec6b81db3c9dacd7997a079322ccc29094aaacf6d685cd7",
"cdcede53d304bf2f40695e053d241eec995ad8b72a4a34bc28d3c50e542cd11720"
],
"unseal_shares": 4,
"unseal_threshold": 2,
"recovery_keys_b64": [],
"recovery_keys_hex": [],
"recovery_keys_shares": 0,
"recovery_keys_threshold": 0,
"root_token": "hvs.NbXRWfYNI4PmA860aBlC4onU"

Let’s unseal the first of two Vault instances in the central cluster. Pass two of four unseal_keys_b64 to the Vault to unseal them according to key-threshold.

kubectl exec vault-0 -- vault operator unseal 4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T
kubectl exec vault-0 -- vault operator unseal miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK

This same we have to do with the second instance, but before that, we must connect the second Vault to Raft storage cluster.

kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
kubectl exec vault-1 -- vault operator unseal 4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T
kubectl exec vault-1 -- vault operator unseal miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK

Time to create Transit Secret Engin. This component will generate root key that we will use to Auto-unseal other Vaults.

# separate window
kubectl port-forward vault-0 -n vault 8200:8200

# set Vault address to use locally Vault CLI
export VAULT_ADDR=http://127.0.0.1:8200

# use 'root_token' generated during Vault initialization
vault login

# create transit secret
vault secrets enable transit
vault write -f transit/keys/autounseal

Save the below Vault policy that we will attach to Auto-unseal token.

path "transit/encrypt/autounseal" {
capabilities = [ "update" ]
}

path "transit/decrypt/autounseal" {
capabilities = [ "update" ]
}
# create policy with the above definition
vault policy write autounseal autounseal-policy.hcl

# create token for Auto-unsealing
$ vault token create -orphan -policy=autounseal -period=24h

Key Value
--- -----
token hvs.CAESIP_A7TaC9kt4yUeqg5_bJNiOJElb4UbA01xoV9Rk4ei6Gh4KHGh2cy5zVXpaa3A1MG9uOEZrNXN2a3J0TGl0cHU
token_accessor wkTM4nsF0ehkRvIuBD9cedHC
token_duration 24h
token_renewable true
token_policies ["autounseal" "default"]
identity_policies []
policies ["autounseal" "default"]

Finally, we have created periodic orphan token which we will use for Auto-unsealing. Orphan means that created token doesn’t have a parent token so can’t be revoked together with ancestor. What is important to note, transit Auto-unseal token is renewed automatically by default.

Now it’s time to prepare Helm chart with the second Vault installation. It will be Vault with transit Auto-unsealing configuration. Check below Helm values file. Provide Vault central address and of course generated in previous step token (root key).

server:
standalone:
enabled: true
config: |
disable_mlock = true
ui=true

storage "file" {
path = "/vault/data"
}

listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = "true"
}

seal "transit" {
address = "http://vault.vault:8200"
token = "hvs.CAESIP_A7TaC9kt4yUeqg5_bJNiOJElb4UbA01xoV9Rk4ei6Gh4KHGh2cy5zVXpaa3A1MG9uOEZrNXN2a3J0TGl0cHU"
disable_renewal = "false"
key_name = "autounseal"
mount_path = "transit/"
tls_skip_verify = "true"
}
# change namespace to Vault Auto-unseal
kns vault-a

helm install vault hashicorp/vault -f vault-auto-unseal-helm-values.yml

The last step is to initialize Vault.

kubectl exec -it vault-0 -- vault operator init

Recovery Key 1: FFMLznSZq9wh/0CJwKLJWKkI9BrK/hjF6ySDYl9a19Ie
Recovery Key 2: qRfrdpkuEcXsF+dFh1Geru8VHkiL/hWUW+vY25twlwT1
Recovery Key 3: dX8sed7Dv8kI8kfFuYWDeQlagoikEVBpV5lZqH4ORnEh
Recovery Key 4: TCCplv+KvZHEOlICQU6eb67hGccufiqcZGkSiGlpQPkx
Recovery Key 5: ictL+c9czgMO+ME8qoTcGpgsvymEcORN7MkrpDE28x4a

Initial Root Token: hvs.6umGyyta9xrjq0q7Cv09Hr8X

Success! Vault is initialized

… and check its status to verify Sealed status.

kubectl exec -it vault-a -- vault status

Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.12.0
Build Date 2022-10-10T18:14:33Z
Storage Type file
Cluster Name vault-cluster-7a11a0ae
Cluster ID a883d977-e70a-6367-3148-9c7a2c246897
HA Enabled false

Each Vault initialization with configured to Auto-Unseal generates Recovery keys instead of Unseal Keys. Recovery keys can’t be used for unsealing Vault. These keys perform only authorization functions, which allows, for example, generates a new root token.

Final Thoughts

--

--

DevOps & Serverless Enthusiast. AWS, GCP, and K8S certified. Home page: https://abartosik.dev

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store