PeerDB Enterprise on EKS: Helm Repository Move, Temporal Schema Bootstrap, and Catalog Secret Keys

Deploying PeerDB Enterprise on EKS as a Helm-managed ArgoCD application hit three blockers the documentation does not mention: a 404 on the old Helm repository URL, Temporal crash-looping because no schema migration runs against an external PostgreSQL backend, and a catalog secret that requires six keys instead of one. This post documents all three.

---

Architecture

RDS PostgreSQL (nonprod + prod)
        |  CDC via PeerDB (WAL logical replication)
        v
  PeerDB (namespace: peerdb)
        |  S3 staging → ClickHouse native protocol
        v
ClickHouse on EKS (namespace: clickhouse)
        |  HTTPS (port 8443)
        v
https://<analytics-endpoint>
# REVIEW: redacted — confirm

PeerDB runs as a set of microservices on EKS:

Component	Role
peerdb-server	API server for peer/mirror management
flow-worker	Executes CDC flows (WAL reading, S3 staging, CH loading)
flow-snapshot-worker	Handles initial table snapshots
flow-api	Internal API for flow coordination
Temporal	Workflow orchestration engine (subchart)
PeerDB UI	Web interface for mirror management

All components need a catalog database — a PostgreSQL database where PeerDB stores peer definitions, mirror configurations, and flow state. Temporal needs its own databases for workflow history and visibility queries.

---

Helm Chart Discovery: The 404 Problem

The PeerDB documentation references a Helm repository at https://peerdbio.github.io/helm-charts. Adding it fails:

$ helm repo add peerdb https://peerdbio.github.io/helm-charts
Error: looks like "https://peerdbio.github.io/helm-charts" is not a valid chart repository:
  404 Not Found

ClickHouse Inc. acquired PeerDB in mid-2024. The acquisition moved the Helm charts to a new enterprise repository. The old URL returns 404 with no redirect.

The working repository:

helm repo add peerdb https://peerdb-io.github.io/peerdb-enterprise
helm repo update
helm search repo peerdb

NAME            CHART VERSION   APP VERSION
peerdb/peerdb   0.9.12          v0.36.12

The chart name changed from the community peerdb to the enterprise peerdb/peerdb. The chart structure also changed — the enterprise chart includes Temporal as a subchart dependency rather than expecting a pre-installed Temporal instance.

---

RDS Catalog Setup

PeerDB needs four database objects on the catalog PostgreSQL instance. I used RDS nonprod as the catalog backend — PeerDB is not a production data path; it reads from production RDS but stores its own metadata on nonprod.

-- Connect to RDS nonprod as the master user
-- kubectl run pg-tmp --rm -i --image=postgres:15 --restart=Never \
--   --env="PGPASSWORD=<master_pass>" -- \
--   psql -h <rds-nonprod-endpoint> \
--   -U postgres -d postgres

-- 1. Catalog user and database
CREATE USER peerdb_catalog WITH LOGIN PASSWORD '<catalog_pass>';
CREATE DATABASE peerdb_catalog_db;
GRANT ALL ON DATABASE peerdb_catalog_db TO peerdb_catalog;

-- 2. Temporal user and databases
CREATE USER "temporal-pg-user" WITH LOGIN PASSWORD '<temporal_pass>';
CREATE DATABASE temporal;
CREATE DATABASE temporal_visibility;
GRANT ALL ON DATABASE temporal TO "temporal-pg-user";
GRANT ALL ON DATABASE temporal_visibility TO "temporal-pg-user";

RDS restriction: CREATE DATABASE ... OWNER <user> fails because the master user is not a superuser. The workaround is CREATE DATABASE followed by GRANT ALL ON DATABASE.

---

Kubernetes Secrets: The Missing Keys Gotcha

PeerDB requires four Kubernetes secrets. The documentation suggests the catalog secret needs only a password key. In practice, when catalog.existingSecret is set, the chart templates extract six keys from the secret:

Secret	Required Keys
peerdb-catalog-secret	host, port, user, password, dbname, uri
peerdb-temporal-secret	password
peerdb-ui-secret	UI_PEERDB_PASSWORD, UI_NEXTAUTH_SECRET
peerdb-server-secret	SERVER_PEERDB_PASSWORD

Creating the catalog secret with only password produces this error in the peerdb-server pod:

Error: template: peerdb/templates/deployment-peerdb-server.yaml:
  map has no entry for key "host"

The complete catalog secret:

kubectl create secret generic peerdb-catalog-secret -n peerdb \
  --from-literal=host=<rds-nonprod-endpoint> \
  --from-literal=port=5432 \
  --from-literal=user=peerdb_catalog \
  --from-literal=password='<catalog_pass>' \
  --from-literal=dbname=peerdb_catalog_db \
  --from-literal=uri='postgresql://peerdb_catalog:<catalog_pass>@<rds-nonprod-endpoint>:5432/peerdb_catalog_db?sslmode=require'

All six keys must be present. The uri key contains the full connection string with SSL mode — PeerDB uses this for certain internal connections that bypass the individual host/port/user fields.

---

Temporal Schema Bootstrap

The PeerDB Helm chart deploys Temporal as a subchart (temporal-deploy). Temporal’s server pods start and attempt to connect to the temporal and temporal_visibility databases. The Temporal subchart does not run schema migrations automatically when using an external PostgreSQL backend.

The Temporal server pods crash-loop with:

unable to connect to temporal database: unable to initialize schema:
  relation "namespaces" does not exist

Temporal expects its schema tables to already exist. The schema must be applied manually using the temporalio/admin-tools container:

# Run the admin-tools pod with RDS TLS flags
kubectl run temporal-schema --rm -i --image=temporalio/admin-tools:1.25.2 \
  --restart=Never -n peerdb -- bash -c '
export SQL_PLUGIN=postgres12
export SQL_HOST=<rds-nonprod-endpoint>
export SQL_PORT=5432
export SQL_USER=temporal-pg-user
export SQL_PASSWORD=<temporal_pass>
export SQL_TLS=true
export SQL_TLS_DISABLE_HOST_VERIFICATION=true

# Initialize temporal database schema
export SQL_DATABASE=temporal
temporal-sql-tool setup-schema -v 0.0
temporal-sql-tool update --schema-dir /etc/temporal/schema/postgresql/v12/temporal/versioned

# Initialize temporal_visibility database schema
export SQL_DATABASE=temporal_visibility
temporal-sql-tool setup-schema -v 0.0
temporal-sql-tool update --schema-dir /etc/temporal/schema/postgresql/v12/visibility/versioned
'

Two environment variables are critical for RDS connectivity:

• SQL_TLS=true — RDS requires TLS connections.
• SQL_TLS_DISABLE_HOST_VERIFICATION=true — The RDS certificate’s CN does not match the hostname used in the connection string. Without disabling host verification, the TLS handshake fails with a certificate mismatch error.

After the schema is applied, the Temporal server pods stop crash-looping and reach Running state within one reconciliation cycle.

---

Helm Values Deep Dive

The PeerDB values file (platform/peerdb/values-peerdb.yaml) configures the external catalog, Temporal subchart, and flow-worker resources:

Catalog Configuration

catalog:
  pgHost: <rds-nonprod-endpoint>
  pgPort: "5432"
  pgUser: peerdb_catalog
  pgDatabase: peerdb_catalog_db
  pgAdminDatabase: postgres
  deploy:
    enabled: false       # Do NOT deploy an in-cluster PostgreSQL
  existingSecret: peerdb-catalog-secret

Setting catalog.deploy.enabled: false tells the chart to use an external PostgreSQL instance instead of deploying one via a CNPG (CloudNativePG) subchart. This is the correct setting for RDS-backed deployments.

Temporal Release Name

temporal:
  releaseName: peerdb-temporal-deploy
  host: peerdb-temporal-deploy-frontend.peerdb.svc.cluster.local
  deploy:
    enabled: true

The releaseName controls the Kubernetes service names generated by the Temporal subchart. Setting it to peerdb-temporal-deploy produces a frontend service at peerdb-temporal-deploy-frontend.peerdb.svc.cluster.local. The temporal.host field must match this generated name exactly. A mismatch — for example, using peerdb as the release name, which generates peerdb-frontend — causes PeerDB’s flow-worker to fail to connect to Temporal with a gRPC dial error.

Temporal Subchart: RDS Backend

temporal-deploy:
  server:
    config:
      persistence:
        default:
          driver: "sql"
          sql:
            driver: "postgres12"
            host: <rds-nonprod-endpoint>
            port: "5432"
            database: temporal
            user: temporal-pg-user
            existingSecret: peerdb-temporal-secret
            tls:
              enabled: true
              disableHostVerification: true
        visibility:
          driver: "sql"
          sql:
            driver: "postgres12"
            host: <rds-nonprod-endpoint>
            port: "5432"
            database: temporal_visibility
            user: temporal-pg-user
            existingSecret: peerdb-temporal-secret
            tls:
              enabled: true
              disableHostVerification: true

Both default (workflow history) and visibility (search/list queries) persistence backends point to RDS nonprod with TLS enabled and host verification disabled — matching the flags used during schema bootstrap.

Flow-Worker: S3 Staging

flowWorker:
  lowCost: true
  replicaCount: 1
  extraEnv:
    - name: PEERDB_CLICKHOUSE_AWS_S3_BUCKET_NAME
      value: <client>-peerdb-staging
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: peerdb-aws-s3-secret
          key: AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: peerdb-aws-s3-secret
          key: AWS_SECRET_ACCESS_KEY
    - name: AWS_REGION
      value: us-east-2
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
      ephemeral-storage: "8Gi"
    limits:
      cpu: "2"
      memory: "4Gi"
      ephemeral-storage: "16Gi"

PeerDB writes data to S3 as an intermediate staging layer before loading into ClickHouse. The flow-worker reads WAL changes, batches them, writes Parquet files to the S3 bucket, then tells ClickHouse to ingest from S3 using the s3() table function. This avoids direct streaming into ClickHouse and enables efficient bulk loading.

The lowCost: true flag tells PeerDB to optimize for cost over throughput — appropriate for a platform with moderate data volume where replication lag of a few seconds is acceptable.

The ephemeral storage request (8Gi) accounts for temporary Parquet file assembly before S3 upload. Under-provisioning ephemeral storage causes pod eviction during large initial copies.

---

ArgoCD Multi-Source Application

The PeerDB ArgoCD app uses a multi-source configuration: the Helm chart comes from the PeerDB repository, while the values file comes from the infra git repository.

# platform/argocd/apps/peerdb.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: peerdb
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://peerdb-io.github.io/peerdb-enterprise
      chart: peerdb
      targetRevision: "0.*"
      helm:
        releaseName: peerdb
        valueFiles:
          - $values/platform/peerdb/values-peerdb.yaml
    - repoURL: https://github.com/<client>-infra.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: peerdb
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: -1
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

The $values ref in the Helm valueFiles path tells ArgoCD to resolve the path relative to the second source (the infra repo). This pattern keeps values files in version control while pulling the chart from the upstream Helm repository — changes to values trigger ArgoCD sync without chart version changes.

---

Results

After resolving all three issues, PeerDB reaches a fully operational state:

$ kubectl get pods -n peerdb
NAME                                    READY   STATUS    RESTARTS
peerdb-flow-api-6c8b4d7f9-x2k1m        1/1     Running   0
peerdb-flow-snapshot-worker-7d9f-k8n2p  1/1     Running   0
peerdb-flow-worker-5b8c6d7e4-j3m4n      1/1     Running   0
peerdb-server-8f9a7b6c5-p5q6r           1/1     Running   0
peerdb-ui-4d5e6f7a8-r7s8t               1/1     Running   0
peerdb-temporal-deploy-frontend-...      1/1     Running   0
peerdb-temporal-deploy-history-...       1/1     Running   0
peerdb-temporal-deploy-matching-...      1/1     Running   0
peerdb-temporal-deploy-worker-...        1/1     Running   0

The PeerDB UI is accessible via port-forward:

kubectl port-forward svc/peerdb-ui 3000:3000 -n peerdb
# Open http://localhost:3000
# Password: stored in peerdb-ui-secret (UI_PEERDB_PASSWORD key)

PeerDB SQL interface (for creating peers and mirrors programmatically):

kubectl port-forward svc/peerdb-server 9900:9900 -n peerdb
psql -h localhost -p 9900 -U peerdb -d peerdb
# Password: stored in peerdb-server-secret (SERVER_PEERDB_PASSWORD key)

The system is ready for peer registration and CDC mirror creation (covered in the next post).

---

Production Rules

Verify Helm repository URLs before debugging chart installation failures. The PeerDB community repository (peerdbio.github.io/helm-charts) returns 404 with no redirect after the ClickHouse acquisition. The current enterprise repository is peerdb-io.github.io/peerdb-enterprise. Acquisitions move repos without deprecation notices — always check helm search repo output before assuming a chart problem.

When deploying Temporal against an external PostgreSQL backend, run schema migrations manually before starting server pods. The Temporal subchart deploys server components but does not run temporal-sql-tool. The databases exist but contain no tables — the server crash-loops on relation "namespaces" does not exist until schema is applied via the temporalio/admin-tools container.

When catalog.existingSecret is set, the secret requires six keys: host, port, user, password, dbname, and uri. A secret with only password fails at Helm template rendering with map has no entry for key "host". The uri key must contain a full connection string including sslmode.

RDS requires both SQL_TLS=true and SQL_TLS_DISABLE_HOST_VERIFICATION=true for Temporal connectivity. AWS RDS certificates carry a CN that does not match the RDS endpoint hostname. Both the Temporal subchart persistence config and the admin-tools schema bootstrap job need both flags — omitting either one fails the TLS handshake.

temporal.releaseName must match temporal.host exactly. The release name determines Kubernetes service names for all Temporal components. Using peerdb as the release name generates peerdb-frontend; using peerdb-temporal-deploy generates peerdb-temporal-deploy-frontend. A mismatch causes PeerDB’s flow-worker to fail all gRPC connections to Temporal.