Backup
Responsibilities Overview
Backup and recovery in a Lakehousecat deployment involves two distinct layers of responsibility:
| Layer | Responsible party | Scope |
|---|---|---|
| Application-level database backups | Lakehousecat (via Job Definitions) | PostgreSQL and ClickHouse dumps stored in SeaweedFS |
| Persistent Volume backups | Cluster / infrastructure administrator | PVCs for PostgreSQL, ClickHouse, SeaweedFS, Valkey |
| Kubernetes Secrets backup | Cluster / infrastructure administrator | Namespace secrets including encryption keys |
Application-Level Backups (Built-in Job Definitions)
Lakehousecat ships with built-in Job Definitions for regular database backups. These run as Airflow-based jobs inside the cluster and store backups in the internal SeaweedFS instance:
| Job Definition | What it backs up | Destination |
|---|---|---|
BACKUP_POSTGRES | Full dump of the PostgreSQL instance — includes all application databases (lhc, lhc_jobs, lhc_visualization, lhc_staging, Airflow metadata) | SeaweedFS (lhc-backend bucket) |
BACKUP_CLICKHOUSE | Full dump of the ClickHouse instance — includes all loaded data source tables and semantic views | SeaweedFS (lhc-backend bucket) |
These jobs are pre-deployed as locked Job Definitions and can be scheduled via the Operations section. Navigate to Operations → Job Definitions to configure the schedule (e.g., daily at 2:00 AM).
These backups are stored inside SeaweedFS, which itself runs on a Persistent Volume provisioned by the Kubernetes cluster. The backup data is only as durable as the underlying storage. If the SeaweedFS PVC is lost, the backups are lost with it.
A cluster-level backup strategy for the SeaweedFS PVC — or replication to external object storage — is required for true off-site durability.
Persistent Volume Backup (Cluster Administrator Responsibility)
The Lakehousecat Operator operates with namespace-scoped RBAC permissions and cannot manage storage classes, Persistent Volumes, or cluster-level infrastructure. The Operator provisions PVCs based on the Helm chart configuration, but the storage class, volume provisioner, and backup mechanism are provided by the Kubernetes cluster.
The following components use Persistent Volumes that must be backed up independently:
| Component | Default PVC size | Notes |
|---|---|---|
| PostgreSQL (primary) | 50 GiB | Stores all application and Airflow metadata |
| ClickHouse | 20 GiB | Stores all loaded data source data |
| SeaweedFS | 50 GiB | Stores files, logs, application backups, and user documents |
| Valkey | 8 GiB | Session and queue state — not backed up by application jobs |
Cluster administrators must formulate and implement a PVC backup strategy using the tools available in their environment — for example:
- Velero (cluster-agnostic PVC snapshots)
- AWS EBS Snapshots / EKS Backup
- Google Cloud Persistent Disk Snapshots / GKE Backup
- Azure Disk Snapshots / AKS Backup
- On-premise: storage class native snapshots or volume replication
The backup cadence and retention policy are determined by the cluster administrator based on business requirements.
Kubernetes Secrets Backup — Critical
All sensitive data in Lakehousecat — across the backend API, Airflow, and Superset — is encrypted using a symmetric encryption key. This key is stored as a Kubernetes Secret in the application namespace.
If the namespace is deleted, or if the secrets are lost, encrypted data in the PostgreSQL and ClickHouse backups cannot be decrypted, even if the backup files themselves are intact.
The following secrets are critical and must be backed up independently of the PVCs:
| Secret name | Contents |
|---|---|
lhc-backend-secret | Central backend configuration including the application encryption key |
lhc-encryption-key | Symmetric encryption key used to encrypt sensitive data at rest |
lhc-api-key | Internal API key used for service-to-service authentication |
lhc-jwt-secret | JWT signing secret for user session tokens |
lhc-postgresql-password-secret | PostgreSQL passwords (admin, user, replication) |
lhc-clickhouse-users-secret | ClickHouse user passwords |
lhc-seaweedfs-root-secret | SeaweedFS root credentials |
lhc-airflow-api-server-secret | Airflow webserver secret key |
lhc-license-secret | Instance license token and subscription info |
Recommended approach: Export all secrets from the namespace to an encrypted external store before any destructive cluster operations, and on a regular schedule alongside database backups.
# Export all secrets from the namespace (store output securely and encrypted)
kubectl get secrets -n <namespace> -o yaml > lhc-secrets-backup.yaml
Store this file in a secure, access-controlled location — it contains all credentials and encryption keys required to restore the application.
What Is Not Backed Up
| Component | Reason |
|---|---|
| Valkey | In-memory session and queue state. Valkey is ephemeral by design. Loss of Valkey state requires users to re-login; no data loss occurs in persistent stores. |
| SeaweedFS storage class / PVC | Outside the Operator's scope — cluster administrator responsibility |
Backup Checklist
For a complete and restorable backup, the following must all be covered:
-
BACKUP_POSTGRESJob Definition scheduled and running successfully -
BACKUP_CLICKHOUSEJob Definition scheduled and running successfully - SeaweedFS PVC backed up at the cluster level (snapshots or replication)
- PostgreSQL PVC backed up at the cluster level
- ClickHouse PVC backed up at the cluster level
- Namespace secrets exported and stored securely off-cluster
- Backup restore procedure tested in a non-production environment