Skills
skills/ionfury/homelab/secrets

secrets

SKILL.md

Secrets Management

Comprehensive guide to secret management in the homelab Kubernetes platform. Three mechanisms exist for provisioning secrets, each serving a distinct purpose in the lifecycle of credentials.

Architecture Overview

┌──────────────────────────────────────────────────────────────────────┐
│                     Secrets Data Flow                                │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  1. secret-generator (in-cluster, ephemeral)                        │
│     Secret with annotation ──► controller generates random value    │
│     Lost on cluster rebuild, auto-regenerated                       │
│                                                                      │
│  2. ExternalSecret (from AWS SSM, persistent)                       │
│     ExternalSecret CR ──► ESO pulls from SSM ──► creates Secret     │
│     Survives cluster rebuilds (data lives in AWS)                   │
│                                                                      │
│  3. app-secrets module (Terragrunt + SSM, persistent)               │
│     Terragrunt generates random ──► stores in SSM ──► ExternalSecret│
│     Best of both: generated + persistent                            │
│                                                                      │
│  4. kubernetes-replicator (cross-namespace)                         │
│     Source Secret (annotated) ──► replica Secret in target namespace │
│     Keeps shared credentials in sync across namespaces              │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Decision Tree

App needs a secret?
│
├─ Can it be randomly generated? (password, API key, token)
│   │
│   ├─ Does it need to survive cluster rebuilds?
│   │   │
│   │   ├─ YES (e.g., encryption key seed, LDAP key)
│   │   │   └─ Use app-secrets Terragrunt module + ExternalSecret
│   │   │      (See: LLDAP pattern below)
│   │   │
│   │   └─ NO (e.g., session secret, internal API key)
│   │       └─ Use secret-generator annotation
│   │          (Simplest option, auto-regenerates)
│   │
│   └─ Is it a database credential?
│       └─ Use secret-generator with type: kubernetes.io/basic-auth
│          (See: Database Credentials section)
│
├─ Must match an external value? (OAuth, cloud API, webhook URL)
│   └─ Use ExternalSecret → AWS SSM
│      User must populate SSM parameter manually or via Terragrunt
│
├─ Shared across namespaces? (DB superuser, Dragonfly password)
│   └─ Use kubernetes-replicator annotations
│      Source secret in origin namespace → replicas in consumer namespaces
│
└─ Unclear?
    └─ AskUserQuestion: "Can this secret be randomly generated,
       or must it match a specific external value?"

Mechanism 1: secret-generator (Ephemeral, In-Cluster)

The mittwald/kubernetes-secret-generator controller watches for annotated Secrets and auto-populates them with random values. Preferred for secrets that do not need to persist across cluster rebuilds.

Basic Pattern

apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
  annotations:
    secret-generator.v1.mittwald.de/autogenerate: "password,api-key"
    secret-generator.v1.mittwald.de/encoding: hex
    secret-generator.v1.mittwald.de/length: "32"
data: {}

Annotation Reference

Annotation Required Values Description
autogenerate Yes Comma-separated key names Keys to generate random values for
encoding No hex, base64, base32, raw Encoding for generated values (default: base64)
length No Integer string (e.g., "32") Length of generated value (default: "40")

Database Credentials Pattern

For applications using the shared CNPG cluster, create a kubernetes.io/basic-auth Secret with a fixed username and auto-generated password:

# kubernetes/clusters/live/config/<app>/db-credentials.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: <app>-db-credentials
  namespace: <app-namespace>
  annotations:
    secret-generator.v1.mittwald.de/autogenerate: password
    secret-generator.v1.mittwald.de/encoding: hex
    secret-generator.v1.mittwald.de/length: "32"
type: kubernetes.io/basic-auth
stringData:
  username: <app>

Real examples:

  • kubernetes/clusters/live/config/authelia-prereqs/authelia-db-credentials.yaml
  • kubernetes/clusters/live/config/authelia-prereqs/lldap-db-credentials.yaml
  • kubernetes/clusters/live/config/zipline/zipline-db-credentials.yaml

Application Secret Pattern

For non-auth secrets (encryption keys, session tokens):

# kubernetes/clusters/live/config/<app>/secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: <app>-secret
  namespace: <app-namespace>
  annotations:
    secret-generator.v1.mittwald.de/autogenerate: CORE_SECRET
    secret-generator.v1.mittwald.de/length: "32"
    secret-generator.v1.mittwald.de/encoding: base64
data: {}

Real example: kubernetes/clusters/live/config/zipline/secret.yaml

Platform-Level Secrets

Shared platform secrets that need cross-namespace replication combine both secret-generator and replicator annotations:

# kubernetes/platform/config/database/superuser-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: cnpg-platform-superuser
  annotations:
    secret-generator.v1.mittwald.de/autogenerate: password
    secret-generator.v1.mittwald.de/length: "32"
    secret-generator.v1.mittwald.de/encoding: base64
    replicator.v1.mittwald.de/replication-allowed: "true"
    replicator.v1.mittwald.de/replication-allowed-namespaces: "zipline,authelia"
type: kubernetes.io/basic-auth
stringData:
  username: postgres

Mechanism 2: ExternalSecret (Persistent, from AWS SSM)

Use External Secrets Operator (ESO) when secrets must come from outside the cluster or persist across cluster rebuilds. ESO pulls values from AWS SSM Parameter Store via the aws-ssm ClusterSecretStore.

Infrastructure

The ClusterSecretStore is defined at: kubernetes/platform/config/secrets/cluster-secret-store.yaml

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-ssm
spec:
  provider:
    aws:
      service: ParameterStore
      region: us-east-2

SSM Path Convention

/homelab/kubernetes/${cluster_name}/<app-or-secret-name>
  • Cluster-specific: /homelab/kubernetes/live/cloudflare-api-token
  • Shared across clusters: /homelab/kubernetes/shared/istio-mesh-ca
  • App-secrets module: /homelab/kubernetes/live/lldap-secrets (JSON with multiple keys)
  • Test: /homelab/kubernetes/test-secret

Basic ExternalSecret

# yaml-language-server: $schema=https://kubernetes-schemas.pages.dev/external-secrets.io/externalsecret_v1.json
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: <app>-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-ssm
  target:
    name: <app>-credentials
  data:
    - secretKey: api-token
      remoteRef:
        key: /homelab/kubernetes/${cluster_name}/<app>/api-token

Multi-Key JSON Pattern

When a single SSM parameter stores multiple keys as JSON (created by the app-secrets module), extract individual properties:

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: lldap-secrets
  namespace: authelia
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-ssm
  target:
    name: lldap-secrets
  data:
    - secretKey: LLDAP_KEY_SEED
      remoteRef:
        key: /homelab/kubernetes/live/lldap-secrets
        property: LLDAP_KEY_SEED
    - secretKey: LLDAP_JWT_SECRET
      remoteRef:
        key: /homelab/kubernetes/live/lldap-secrets
        property: LLDAP_JWT_SECRET
    - secretKey: LLDAP_LDAP_USER_PASS
      remoteRef:
        key: /homelab/kubernetes/live/lldap-secrets
        property: LLDAP_LDAP_USER_PASS

Real example: kubernetes/clusters/live/config/authelia-prereqs/lldap-secrets.yaml

Templated ExternalSecret

For secrets that need transformation (e.g., generating a config file from credentials):

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: hardware-monitoring-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-ssm
  target:
    name: hardware-monitoring-credentials
    template:
      engineVersion: v2
      data:
        ipmi-config.yml: |
          modules:
            default:
              user: "{{ .ipmiUsername }}"
              pass: "{{ .ipmiPassword }}"
  data:
    - secretKey: ipmiUsername
      remoteRef:
        key: /homelab/kubernetes/${cluster_name}/ipmi-username
    - secretKey: ipmiPassword
      remoteRef:
        key: /homelab/kubernetes/${cluster_name}/ipmi-password

Real example: kubernetes/platform/config/monitoring/hardware-monitoring-secrets.yaml

ExternalSecret Placement

Scope Location Example
Platform-wide (all clusters) kubernetes/platform/config/<subsystem>/ cloudflare-api-token, longhorn-s3-backup
Cluster-specific kubernetes/clusters/<cluster>/config/<app>/ lldap-secrets

Mechanism 3: app-secrets Terragrunt Module (Generated + Persistent)

For secrets that must be randomly generated AND survive cluster rebuilds. The app-secrets module generates random values with OpenTofu and stores them as a JSON SecureString in AWS SSM.

Module Location

infrastructure/modules/app-secrets/

How It Works

  1. Terragrunt unit defines the secret names and generation parameters
  2. OpenTofu generates random passwords and stores as JSON in SSM
  3. Local backup is written for disaster recovery
  4. ExternalSecret in Kubernetes pulls individual keys from the JSON parameter

Step 1: Create a Terragrunt Unit

# infrastructure/units/<app>-secrets/terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "../../../.././/modules/app-secrets"
}

inputs = {
  name = "<app>"

  secrets = {
    SECRET_KEY_1 = { length = 32, special = false }
    SECRET_KEY_2 = { length = 32, special = false }
  }

  ssm_parameter_path = "/homelab/kubernetes/live/<app>-secrets"

  local_backup_path = pathexpand("~/.secrets/homelab/<app>-secrets.json")
}

Real example: infrastructure/units/lldap-secrets/terragrunt.hcl

Step 2: Add Unit to Stack

Add the unit to the relevant stack in infrastructure/stacks/<stack>/terragrunt.stack.hcl:

unit "<app>_secrets" { source = "../../units/<app>-secrets" }

Step 3: Apply and Create ExternalSecret

task tg:apply-<stack>  # Apply with human approval

Then create the ExternalSecret in Kubernetes using the multi-key JSON pattern (see above).

Module Behavior

  • lifecycle.prevent_destroy = true protects the SSM parameter from accidental deletion
  • lifecycle.ignore_changes = [value] prevents re-generating secrets on subsequent applies
  • Local backup at ~/.secrets/homelab/<app>-secrets.json for disaster recovery

Mechanism 4: kubernetes-replicator (Cross-Namespace)

The mittwald/kubernetes-replicator copies Secrets from one namespace to another. Used when a shared resource (database, cache) generates a secret that consumer namespaces need.

Source Secret Annotations

Add to the source Secret (in the originating namespace):

annotations:
  replicator.v1.mittwald.de/replication-allowed: "true"
  replicator.v1.mittwald.de/replication-allowed-namespaces: "app1,app2"

Replica Secret

Create an empty Secret in the target namespace that references the source:

---
apiVersion: v1
kind: Secret
metadata:
  name: <source-secret-name>
  namespace: <target-namespace>
  annotations:
    replicator.v1.mittwald.de/replicate-from: <source-namespace>/<source-secret-name>
data: {}

Common Replication Patterns

Source Source Namespace Consumers Purpose
cnpg-platform-superuser database zipline, authelia Shared DB superuser
dragonfly-password database immich, authelia Shared cache password
immich-database-app database immich Dedicated DB app credentials
heartbeat-ping-url kube-system monitoring Health check URL

Adding a New Replication

  1. Source side - add/update replication annotations:

    replicator.v1.mittwald.de/replication-allowed: "true"
replicator.v1.mittwald.de/replication-allowed-namespaces: "existing-ns,new-ns"

  2. Consumer side - create replica Secret:

    ---
apiVersion: v1
kind: Secret
metadata:
  name: <source-secret-name>
  namespace: <consumer-namespace>
  annotations:
    replicator.v1.mittwald.de/replicate-from: <source-ns>/<source-secret-name>
data: {}

  3. Add both files to their respective kustomization.yaml

Three-Tier Secret Pattern (Exemplar: Authelia)

The kubernetes/clusters/live/config/authelia-prereqs/ directory demonstrates the complete secret pattern for an application that needs all three types:

File Mechanism Purpose
lldap-secrets.yaml ExternalSecret (from app-secrets module) Persistent LLDAP encryption keys
authelia-db-credentials.yaml secret-generator Ephemeral DB password
lldap-db-credentials.yaml secret-generator Ephemeral DB password
cnpg-superuser-replica.yaml kubernetes-replicator Replicated from database namespace
dragonfly-secret-replication.yaml kubernetes-replicator Replicated from database namespace

Debugging

ExternalSecret Not Syncing

# Check ExternalSecret status
KUBECONFIG=~/.kube/<cluster>.yaml kubectl get externalsecret -A

# Describe for detailed error
KUBECONFIG=~/.kube/<cluster>.yaml kubectl describe externalsecret <name> -n <namespace>

# Check ClusterSecretStore health
KUBECONFIG=~/.kube/<cluster>.yaml kubectl get clustersecretstore aws-ssm

# Check ESO operator logs
KUBECONFIG=~/.kube/<cluster>.yaml kubectl logs -n kube-system -l app.kubernetes.io/name=external-secrets --tail=50

Common Failure Causes

Symptom Cause Fix
SecretSyncedError SSM parameter does not exist Create parameter: aws ssm put-parameter --name <path> --type SecureString --value <json>
SecretSyncedError with property error JSON key missing Verify SSM parameter JSON has expected keys
ClusterSecretStore not ready AWS credentials invalid Check external-secrets-access-key in kube-system
Secret exists but empty Replicator source not annotated Add replication-allowed annotations to source
Stale secret value refreshInterval too long Default is 1h; reduce if needed

Verify SSM Parameter Exists

aws ssm get-parameter --name "/homelab/kubernetes/<cluster>/<secret>" --with-decryption

PrometheusRule Alerts

ExternalSecret health is monitored by alerts defined in: kubernetes/platform/config/monitoring/external-secrets-alerts.yaml

Alert Condition Severity
ExternalSecretSyncFailure Sync errors increasing over 5m critical
ExternalSecretNotReady Not ready for 10m+ warning
ClusterSecretStoreUnhealthy Store not ready for 5m critical

Cross-References

Document Relevance
kubernetes/platform/CLAUDE.md Secrets management overview, SSM parameters for bootstrap
kubernetes/platform/config/CLAUDE.md Config subsystem organization
deploy-app skill Secrets decision tree for new deployments
cnpg-database skill Database credential chain
terragrunt skill Infrastructure operations for app-secrets module
Weekly Installs
24
Repository
ionfury/homelab
GitHub Stars
22
First Seen
Feb 25, 2026
Security Audits
Gen Agent Trust HubPass
SocketPass
SnykPass
Installed on
cline24
github-copilot24
codex24
kimi-cli24
gemini-cli24
cursor24