Blog › Field Notes › Keycloak SMTP via ExternalSecrets and Fineract Production Gateway Wiring
Platform #keycloak#smtp#argocd#external-secrets#gateway-api#traefik#fineract#gitops

Keycloak SMTP via ExternalSecrets and Fineract Production Gateway Wiring

Wired Keycloak SMTP through ExternalSecrets and AWS Secrets Manager across dev and prod, then diagnosed three ArgoCD project, destination, and Gateway listener gaps blocking Fineract production routing.

· Gideon Warui
ON THIS PAGE

Two distinct problems to work through: configuring Keycloak to send transactional emails across both dev and production, and getting Fineract production properly registered and routable in ArgoCD. Both required changes across multiple layers: ConfigMaps, ExternalSecrets, AWS Secrets Manager, ArgoCD Applications, ArgoCD Projects, and Traefik Gateway listeners.

Problem 1: Keycloak Cannot Send Emails

Keycloak was running in both dev (identity namespace) and prod (prod-identity namespace) but had no SMTP configuration. Users could not receive password reset emails, verification links, or any other transactional email from the identity provider.

The Setup

The Keycloak deployment loads environment variables from two sources via envFrom:

  1. keycloak-config (ConfigMap) — non-secret configuration
  2. keycloak-secrets (Secret) — credentials managed by External Secrets Operator, pulling from AWS Secrets Manager

The SMTP provider is Migadu, which requires SSL on port 465.

The Solution

Nine environment variables were needed. Eight are non-secret and go in the ConfigMap. One (the password) is a secret and must go through the ExternalSecret pipeline.

Adding SMTP Configuration to ConfigMaps

Both services/dev/keycloak/configmap.yaml and services/prod/keycloak/configmap.yaml received the same block:

apiVersion: v1
kind: ConfigMap
metadata:
  name: keycloak-config
  namespace: identity  # prod-identity for production
data:
  # ... existing vars ...

  # SMTP Email Configuration (Migadu)
  KC_SPI_EMAIL_SMTP_HOST: smtp.migadu.com
  KC_SPI_EMAIL_SMTP_PORT: "465"
  KC_SPI_EMAIL_SMTP_FROM: noreply@<client>.com
  KC_SPI_EMAIL_SMTP_FROM_DISPLAY_NAME: <client>
  KC_SPI_EMAIL_SMTP_AUTH: "true"
  KC_SPI_EMAIL_SMTP_SSL: "true"
  KC_SPI_EMAIL_SMTP_STARTTLS: "false"
  KC_SPI_EMAIL_SMTP_USER: noreply@<client>.com

Numeric and boolean values must be quoted in ConfigMaps. "465", "true", and "false" are strings — Kubernetes ConfigMap values are always strings, and Keycloak reads them as such.

Adding the Password to ExternalSecrets

The password cannot live in a ConfigMap. It flows through External Secrets Operator:

AWS Secrets Manager -> ExternalSecret -> Kubernetes Secret -> Pod env var

A new entry was added to both services/dev/keycloak/external-secret.yaml and services/prod/keycloak/external-secret.yaml:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: keycloak-secrets
  namespace: identity  # prod-identity for production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: keycloak-secrets
    creationPolicy: Owner
  data:
    # ... existing secret mappings ...
    - secretKey: KC_SPI_EMAIL_SMTP_PASSWORD
      remoteRef:
        key: <client>-shared/keycloak       # <client>-shared/keycloak-prod for production
        property: KC_SPI_EMAIL_SMTP_PASSWORD

Updating AWS Secrets Manager

The password needed to be added to the JSON blob stored in AWS Secrets Manager. The secret is a flat JSON object with key-value pairs. Since jq was not available on the machine, I used Python:

import json, subprocess

secret_id = '<client>-shared/keycloak'
result = subprocess.run(
    ['aws', 'secretsmanager', 'get-secret-value',
     '--secret-id', secret_id,
     '--query', 'SecretString', '--output', 'text'],
    capture_output=True, text=True
)
data = json.loads(result.stdout.strip())
data['KC_SPI_EMAIL_SMTP_PASSWORD'] = 'the-password'
new_secret = json.dumps(data)

subprocess.run(
    ['aws', 'secretsmanager', 'put-secret-value',
     '--secret-id', secret_id,
     '--secret-string', new_secret],
    capture_output=True, text=True
)

This pattern — get, merge, put — preserves existing keys in the secret. The same was done for <client>-shared/keycloak-prod.

Verification

After ArgoCD synced and pods were restarted:

# Verify ConfigMap has SMTP vars
kubectl get configmap keycloak-config -n identity -o yaml | grep SMTP

# Verify the secret was created by ExternalSecret
kubectl get secret keycloak-secrets -n identity \
  -o jsonpath='{.data.KC_SPI_EMAIL_SMTP_PASSWORD}' | base64 -d

# Restart to pick up new env vars
kubectl rollout restart deployment keycloak -n identity
kubectl rollout status deployment keycloak -n identity

ArgoCD Sync Timing

After pushing, prod synced almost immediately but dev did not. ArgoCD’s default polling interval means changes may not appear instantly. Forcing a hard refresh resolved it:

kubectl patch application keycloak -n argocd \
  --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

Problem 2: Fineract Production Missing from ArgoCD

The Fineract production application (fineract-production) was not registered in ArgoCD at all. The Application manifest existed in the repo but had never been applied to the cluster. Three sub-problems surfaced during the fix.

Sub-Problem A: Wrong ArgoCD Project

The application-production.yaml was configured to use the default ArgoCD project instead of prod:

# Before (incorrect)
spec:
  project: default

# After (correct)
spec:
  project: prod

This matters because the prod project has specific source repo restrictions (only <client>-Kitchen/* repos) and namespace allowlists. Using default bypasses these guardrails.

Sub-Problem B: Missing Namespace in ArgoCD Project

The prod ArgoCD project did not include fineract-production in its allowed destinations. Without this, the Application would be rejected:

# platform/argocd/projects/prod.yaml
spec:
  destinations:
    # ... existing namespaces ...
    - namespace: <namespace>      # REVIEW: redacted — confirm
      server: https://kubernetes.default.svc
    - namespace: fineract-production          # added
      server: https://kubernetes.default.svc

ArgoCD Projects act as a policy layer. An Application can only deploy to namespaces explicitly listed in its project’s destinations. This prevents a misconfigured Application from accidentally deploying resources into the wrong namespace.

Sub-Problem C: Missing Traefik Gateway TLS Listeners

After applying the ArgoCD Application, it synced but the HTTPRoutes failed:

message: "Parent traefik-gateway: "

The HTTPRoutes in fineract-production referenced Gateway listener names (websecure-fineract and websecure-mifos) that did not exist. The Traefik Gateway needed HTTPS listeners for the production domains:

# platform/cert-manager/traefik-gateway-tls.yaml
# Added to the Gateway spec.listeners array:

    # HTTPS listener for fineract.example.com (Fineract API Production)
    - name: websecure-fineract
      port: 8443
      protocol: HTTPS
      hostname: fineract.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: fineract-tls
            namespace: fineract-production
            kind: Secret
      allowedRoutes:
        namespaces:
          from: All

    # HTTPS listener for mifos.example.com (Fineract UI Production)
    - name: websecure-mifos
      port: 8443
      protocol: HTTPS
      hostname: mifos.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: fineract-tls
            namespace: fineract-production
            kind: Secret
      allowedRoutes:
        namespaces:
          from: All

The cert-manager Certificate in fineract-production already covers both domains with a single secret (fineract-tls):

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: fineract-cert
  namespace: fineract-production
spec:
  secretName: fineract-tls
  dnsNames:
  - fineract.example.com
  - mifos.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

A ReferenceGrant already existed in fineract-production allowing the Traefik Gateway to access the TLS secret, so no additional grants were needed.

Verification

After applying the Gateway update:

# HTTPRoutes now accepted
kubectl get httproute -n fineract-production \
  -o jsonpath='{range .items[*]}{.metadata.name}: accepted={.status.parents[0].conditions[?(@.type=="Accepted")].status}{"\n"}{end}'
fineract-backend-route: accepted=True
fineract-ui-route: accepted=True

The Full Checklist for Adding a Service to Production

Based on this work and previous experiences, here is the complete checklist when a service needs to go from sandbox to production:

  1. ArgoCD Project — Add the production namespace to platform/argocd/projects/prod.yaml destinations
  2. ArgoCD Application — Ensure spec.project is prod (not default), apply to the cluster
  3. Traefik Gateway Listeners — Add HTTPS listeners for each production hostname
  4. TLS Certificate — Ensure cert-manager Certificate exists in the production namespace with all required domain names
  5. ReferenceGrant — Ensure the production namespace grants the traefik namespace access to its TLS secrets
  6. ExternalSecrets — Verify AWS Secrets Manager has the production secrets and the ExternalSecret CRD is configured
  7. DNS — Ensure production domains point to the load balancer

Missing any one of these results in a partially working deployment with unhelpful error messages. The Gateway API "Parent traefik-gateway:" empty message is a particularly opaque indicator that the listener simply does not exist.

#keycloak#smtp#argocd#external-secrets#gateway-api#traefik#fineract#gitops

Share

Post on X LinkedIn

Discussion