Encryption Key Rotation

Lakehousecat encrypts sensitive data at rest using three independent keys. Each protects a different part of the system, and each is rotated a different way. This page explains what each key protects, and walks through rotating the one you manage directly — ENCRYPT_KEY.

Before you start: back up your instance, including its secrets. A backup that doesn't capture the encryption keys themselves cannot be used to recover encrypted data — see Backup.

Keep the old key with your pre-rotation backups

A backup taken before a rotation contains data encrypted with the old key. Restoring it after the rotation only works if the old key is set again as ENCRYPT_KEYS_LEGACY. Store the old key together with every backup that predates the rotation, and don't destroy it while those backups might still be needed. The order is: back up (including secrets), rotate, then take a new backup.

The three keys

Key	Protects	Who rotates it
ENCRYPT_KEY	LHC content: charts, dashboards, datasources, custom models, sessions, user profiles, job definitions	You, via the lhc CLI — this page
Airflow Fernet key	Airflow connections and variables in the Airflow metadata database	You, via Airflow's own key-rotation procedure
Superset secret key	Superset's stored database connections	Your cluster administrator — see below

On instances created from 2026-08-26 onward, ENCRYPT_KEY and the Airflow Fernet key are independent values — rotating one never requires rotating the other.

Rotating ENCRYPT_KEY

ENCRYPT_KEY is read by five services — lhc, audio, analytics, semantic and llm — each from its own Kubernetes secret:

Deployment	Secret
lhc	lhc-secret
audio	lhc-audio-secret
analytics	lhc-analytics-secret
semantic	lhc-semantic-secret
llm	lhc-llm-secret

The Operator creates these five secrets from lhc-backend-secret when the instance is provisioned. After that, the five secrets are the source of truth for ENCRYPT_KEY: an Operator update (a change to the Lakehousecat resource, a forced update, or an upgrade) rebuilds the other values in them but keeps the ENCRYPT_KEY and ENCRYPT_KEYS_LEGACY you set. Patching lhc-backend-secret alone therefore rotates nothing. Still set the new key there as well (step 1), so the instance status stays clean: while the five secrets and lhc-backend-secret disagree, the Lakehousecat resource reports the condition EncryptKeyDriftDetected with reason BackendSecretStale. Reason ServiceSecretsSplit means the five secrets disagree with each other, which you have to fix before any pod restarts. The Airflow Fernet key also lives in lhc-backend-secret and is a separate value.

Rotating requires cluster access (to update the secrets and restart pods) and Admin permissions in Lakehousecat (to run the CLI command that rewrites the data).

1. Stage the new key, keep the old one readable

Generate a new key. It must be a Fernet key: 32 random bytes in URL-safe base64, 44 characters. A plain openssl rand -base64 32 can produce + and /, which the services reject, so use:

openssl rand 32 | base64 | tr '+/' '-_'

In each of the five secrets, set ENCRYPT_KEY to the new key and add the old value as ENCRYPT_KEYS_LEGACY (comma-separated if you ever need more than one). This lets the services read data written with either key, while writes use the new key. The secrets are regular, mutable Kubernetes secrets — a kubectl patch (or your usual GitOps/Helm values change) works directly. <new-key-base64> and <old-key-base64> are the keys base64-encoded once more, because that's how Kubernetes stores secret data (printf %s '<key>' | base64):

for s in lhc-secret lhc-audio-secret lhc-analytics-secret lhc-semantic-secret lhc-llm-secret; do
  kubectl -n <namespace> patch secret "$s" --type='merge' \
    -p '{"data":{"ENCRYPT_KEY":"<new-key-base64>","ENCRYPT_KEYS_LEGACY":"<old-key-base64>"}}'
done

Set the new key in lhc-backend-secret too, so the drift condition clears once all six agree:

kubectl -n <namespace> patch secret lhc-backend-secret --type='merge' \
  -p '{"data":{"ENCRYPT_KEY":"<new-key-base64>"}}'

Then restart all five deployments, one after another, so every service loads both values:

for d in lhc audio analytics semantic llm; do
  kubectl -n <namespace> rollout restart deployment/"$d"
  kubectl -n <namespace> rollout status deployment/"$d"
done

Confirm the new key arrived: primary_key_fingerprint in the status output below must show the fingerprint of the new key.

2. Rewrite the data

lhc instance rotate-encryption-key

This runs as a dry run by default — it reports how many rows in each table still carry the old key's fingerprint without writing anything:

{ "run_id": 3 }

lhc instance rotate-encryption-key-status 3

{
  "run_id": 3,
  "status": "completed",
  "mode": "dry_run",
  "primary_key_fingerprint": "01b51894",
  "tables": {
    "chart": { "scanned": 42, "pending": 42, "rewritten": 0 },
    "datasource": { "scanned": 5, "pending": 5, "rewritten": 0 }
  }
}

Once the counts look right, run it for real:

lhc instance rotate-encryption-key --apply

Poll rotate-encryption-key-status again until status is completed. The operation is interruptible and safe to re-run — rows already rewritten are skipped, so a second --apply after a partial run only picks up where the first left off. If a row was written with a key that isn't in ENCRYPT_KEYS_LEGACY, the run stops immediately with status: "failed" and names the affected row, rather than silently leaving it unrotated.

3. Remove the fallback

Once status: "completed" shows pending: 0 for every table, remove ENCRYPT_KEYS_LEGACY from all five secrets and restart all five deployments (lhc, audio, analytics, semantic, llm) once more:

for s in lhc-secret lhc-audio-secret lhc-analytics-secret lhc-semantic-secret lhc-llm-secret; do
  kubectl -n <namespace> patch secret "$s" --type='json' \
    -p '[{"op":"remove","path":"/data/ENCRYPT_KEYS_LEGACY"}]'
done
for d in lhc audio analytics semantic llm; do
  kubectl -n <namespace> rollout restart deployment/"$d"
  kubectl -n <namespace> rollout status deployment/"$d"
done

Leaving the old key configured after rotation defeats the rotation — it remains a valid way to read the data. Keep a copy of the old key with your pre-rotation backups (see the warning at the top).

Verify

Open a chat with history, browse a chart or dashboard, and confirm a datasource connection still works. Also trigger a job — job definitions are encrypted and run through Airflow. These paths exercise the encrypted content directly, so a rotation problem shows up here first.

Airflow Fernet key

The Airflow Fernet key encrypts the connections and variables in Airflow's metadata database. It lives in lhc-backend-secret as fernet-key, separate from ENCRYPT_KEY, and that is its only source: every Airflow component reads it from there. The Operator creates it once and never rewrites it.

Rotating it is the standard Airflow procedure. Avoid rotating while a job is running — see Operations for how to safely pause a running job first.

1. Generate a new key the same way as for ENCRYPT_KEY. Set fernet-key to the new key followed by the old one, comma-separated, so Airflow can still read everything encrypted with the old key:

   kubectl -n <namespace> patch secret lhc-backend-secret --type='merge' \
     -p '{"data":{"fernet-key":"<base64 of new-key,old-key>"}}'

2. Restart the Airflow deployments (airflow-api-server, airflow-dag-processor, airflow-scheduler, airflow-triggerer, airflow-worker) and rewrite the stored values with the new key:

   kubectl -n <namespace> exec deploy/airflow-scheduler -- airflow rotate-fernet-key

3. Set fernet-key to the new key alone and restart the Airflow deployments once more.

Superset secret key

Superset's secret key protects stored database connection details (passwords, SSH tunnel credentials, and OAuth2 tokens — not the connection string itself, which is stored with the password masked). Rotating it is more involved than ENCRYPT_KEY, so contact Lakehousecat support before rotating this key. Getting the steps out of order can leave Superset unable to reach its data sources.

How the secret is stored depends on the age of your instance:

• Instances created from 2026-08-26 onward: lhc-superset-secret is a regular, mutable secret, and restarted Superset pods pick up a new value immediately.
• Older instances: the secret is immutable and can only be replaced by delete-and-recreate. On a managed Kubernetes node pool (GKE, EKS, AKS) this is currently not supported: some pods can keep serving the old value indefinitely, and there's no way to force a refresh without node-level access these platforms don't grant.

Superset's built-in re-encryption command is known to report success without doing anything, so don't rely on it.

Cached Superset sessions renew themselves

Lakehousecat caches its Superset sessions for up to 2 hours. After a rotation, the first request per session is rejected with 422 Signature verification failed, Lakehousecat signs in again and repeats the request. Charts and embedded dashboards keep working without clearing any cache.

What this page doesn't cover

The Kubernetes Operator's own encryption key (used for the handshake between your instance and the Lakehousecat portal) is managed entirely by the Operator and never mounted into any Lakehousecat service — there is nothing to rotate here. JWT signing secrets and the API key salt have their own, independent lifecycles and aren't covered by this page either.