Secrets Management with External Secrets, Argo CD and GitOps

Introduction

The previous tutorial’s introduction discussed the benefits of managing secrets with GitOps via Argo CD and the Argo CD Vault Plugin. This guide explores using an alternative secrets management tool, External Secrets Operator¹ (ESO). Although the implementation of these tools differ, both essential solve the same problem. How to manage secrets via GitOps.

How does External Secrets Operator differ from Argo CD Vault Plugin?

Prerequisites

For this guide you’ll need the following:

• A secret management system such as HashiCorp Vault, AWS Secrets Manager or Google Cloud Secret Manager (ESO supports many more systems)
• Beginner to intermediate knowledge of Argo CD, Kubernetes & Git/GitHub
• Terraform ≥ v0.15

How External Secrets Operator (ESO) Works

Here’s a high level overview of how ESO works with Argo CD to create secrets in a Kubernetes cluster:

1. Argo CD syncs the ExternalSecret manifest from a git repo and applies it to create the ExternalSecret resource in the Kubernetes cluster.
2. The ESO Controller inspects the configuration of the synced ExternalSecret to determine which SecretStore to use.
3. The Controller then queries that SecretStore for the credentials required to access the external API (Secrets Manager e.g. Vault).
4. ESO fetches the secret(s) from the secrets manager at the path named in the ExternalSecret.
5. Finally, ESO creates/syncs the actual secret inside the cluster.

Simply put, the External Secrets Operator uses an ExternalSecret resource as a template to define the creation and maintenance of a Kubernetes Secret.

Install External Secrets Operator on a Kubernetes Cluster

Install the External Secrets Operator either via Helm (using the helm command) or via Argo CD as an App:

Helm

Add the External Secretes repo:

helm repo add external-secrets https://charts.external-secrets.io

Install the External Secrets Operator:

helm install external-secrets \
   external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true

Argo CD (my preference)

If you haven’t already, download and install the Argo CD CLI² then login to your Argo CD installation using the argocd command:

argocd login argocd.local
Username: admin
Password:
'admin:login' logged in successfully
Context 'argocd.local' updated

Create the external-secrets app using the following argocd command:

argocd app create external-secrets \
 --repo https://charts.external-secrets.io \
 --helm-chart external-secrets \
 --revision 0.5.9 \
 --dest-namespace external-secrets \
 --dest-server https://kubernetes.default.svc

And then sync it to complete the installation on the cluster:

argocd app sync external-secrets

Create a secret containing your Secret Manager’s credentials

Before creating a SecretStore, it’s best to create the secret containing the credentials the SecretStore will use to authenticate with your Secrets Manager. In the case of this guide that’s HashiCorp’s Vault using the token-based authentication method.

Run the following command to store your Vault token in a file named vault_token.txt:

echo -n 'VAULT_TOKEN' > ./vault_token.txt

Then use kubectl to create a secret named vault-token containing the credentials stored in the vault_token.txt file:

kubectl -n external-secrets create secret generic vault-token \
 --from-file=token=./vault_token.txt

Create a ClusterSecretStore

ESO offers two types of resources for defining how secret managers are accessed, SecretStore and ClusterSecretStore. The SecretStore resource is namespaced and can only be referenced by ExternalSecrets that reside in the the same namespace as the SecretStore.

The ClusterSecretStore however is a cluster scoped SecretStore that can be referenced by all ExternalSecrets from all namespaces.

This guide uses a CluserSecretStore to demonstrate the creation of secrets in any namespace:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.local"
      path: "secret"
      version: "v2"
      auth:
        # points to a secret that contains a vault token
        # https://www.vaultproject.io/docs/auth/token
        tokenSecretRef:
          name: "vault-token"
          key: "token"
          namespace: external-secrets

This ClusterSeceretStore resource defines the url of the Vault server (line 8). It also references the name (vault-token, line 15) and namespace (external-secrets, line 17) of the previously created secret that contains the static token required to authenticate against Vault.

Create a Secret in Vault

Create an example secret in Vault using the following command:

vault kv put secret/demo-secrets tokenA=too_many_secrets tokenB=not_enough_secrets

Key                Value
---                -----
created_time       2022-08-18T23:48:59.493381453Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Create an ExternalSecret

Create a git repository containing the following example ExternalSecret manifest:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: demo-secret
  namespace: default
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: demo-secret
  data:
  - secretKey: token
    remoteRef:
      key: secret/demo-secrets
      property: tokenA

Sync this ExternalSecret to the cluster by creating an app named demo-secret in Argo CD that uses the git repository as a source for the manifest:

argocd app create demo-secret --repo https://github.com/colinwilson/argocd-micro-example-apps.git --path external-secrets/ExternalSecret --dest-namespace default --dest-server https://kubernetes.default.svc --sync-policy automated --self-heal

If successful you can view the demo-secret app in the Argo CD UI:

Click on app to see further details:

You can confirm that ESO has successfully generated the secret from the ExternalSecret manifest by running the kubectl command below:

kubectl -n default get secret demo-secret -o jsonpath="{.data.token}" | base64 -d; echo

too_many_secrets

Automatic Updates

References

• External Secrets Operator Documentation
• Argo CD Documentation

Footnotes