Why This Stack Deserves Your Attention
Secrets management on Kubernetes is one of those problems that looks solved until you actually dig into the tradeoffs. HashiCorp Vault's BSL license change pushed a lot of teams to evaluate alternatives, and OpenBao — the Linux Foundation's open-source fork — has emerged as the most production-ready drop-in replacement. But OpenBao is only as reliable as its storage backend, and that's where most tutorials cut corners. (Read also: Reduce PDF File Size in Linux: Tools and Methods)
This guide wires OpenBao directly to a CloudNativePG (CNPG) PostgreSQL cluster running inside Kubernetes itself. No RDS. No Cloud SQL. No external database dependency that becomes a single point of failure or a vendor lock-in vector. Just two CNCF projects — Kubernetes and CloudNativePG — giving you a fully self-healing, synchronously replicated secrets backend with mTLS certificate authentication and zero passwords in the connection string. (Read also: VPS vs VDS vs Dedicated Servers: The Ultimate Comparison Guide)
This is the kind of setup I'd run in production at a company that takes both security posture and infrastructure independence seriously. Let's build it.
You can deploy this stack on a managed VPS at VPS Server in minutes.
Consider a professional security assessment from CyberXper to identify vulnerabilities in your infrastructure.
Architecture Overview: OpenBao + CloudNativePG on Kubernetes
Before touching a terminal, it's worth being precise about what we're deploying and why each decision was made.
Storage Backend: PostgreSQL via OpenBao's Native Driver
OpenBao ships a native postgresql storage backend. It creates an encrypted key-value table and, when ha_enabled = "true", a HA lock table for leader election. This means the Postgres cluster isn't just a config store — it's the source of truth for all your secrets, encrypted at rest by OpenBao's seal key.
Database Cluster: 3-Instance CNPG with Quorum Replication
We're running three PostgreSQL instances managed by the CloudNativePG operator with quorum-based synchronous replication (method: any, number: 1). This gives us RPO=0 — no committed write is ever lost — because at least one standby must confirm every transaction before it's acknowledged. If a standby goes down, writes pause rather than proceed unacknowledged. That's the right tradeoff for a secrets backend.
The dataDurability: required default enforces this. Don't change it.
Authentication: Passwordless mTLS via DatabaseRole CRDs
Here's where this setup gets interesting from a security standpoint. Both the schema-owning role and the application role OpenBao connects as use TLS client certificates issued by CNPG's DatabaseRole CRD — no passwords, no Kubernetes secrets containing plaintext credentials. The pg_hba.conf rules explicitly enforce cert authentication over SSL and reject any non-SSL connection attempt for these roles.
This is the principle of least privilege applied at the database layer: even if someone gets into the cluster, there are no passwords to extract.
Workload Isolation
PostgreSQL pods run on dedicated nodes with a node-role.kubernetes.io/postgres taint. Pod anti-affinity with topologyKey: topology.kubernetes.io/zone and podAntiAffinityType: required ensures all three instances land in separate failure domains. OpenBao itself schedules onto the remaining general-purpose nodes.
Setting Up a Local Test Environment
For local development, the cnpg-playground repository is the fastest path to a conformant Kubernetes cluster with the CloudNativePG operator pre-installed. It provisions a Kind cluster with six nodes: one control plane, one infrastructure node, one application node, and three nodes carrying the node-role.kubernetes.io/postgres taint that our CNPG cluster tolerations target. (Read also: How to Install AMD ROCm on Ubuntu 26.04 for AI & Deep Learning)
Prerequisites: Docker, Kind, Helm, and kubectl.
## Clone the playground repo
git clone https://github.com/cloudnative-pg/cnpg-playground.git
cd cnpg-playground
## Provision a single local cluster
./scripts/setup.sh openbao
## Deploy CloudNativePG operator, cert-manager, and Barman Cloud plugin only
## Skip the demo databases — we don't need them
REQUIREMENTS_ONLY=true ./demo/setup.sh
Passing a single label to setup.sh gives you one cluster and skips the two-region disaster recovery demo. The REQUIREMENTS_ONLY=true flag on the second script deploys the operator and a ClusterImageCatalog named postgresql-minimal-trixie — which our Cluster manifest references in Step 1.
Note: This setup is not Kind-specific. Any conformant Kubernetes cluster with dedicated nodes and sufficient worker capacity works. If you're running on a real VPS or cloud environment, check out vps-server.host for cloud VPS options that give you the raw capacity to run this stack properly.
Step 1: Deploy the CNPG Cluster, Roles, and Database
This is the foundation. We're deploying a Cluster, two DatabaseRole objects, and a Database object — all in the openbao namespace.
A few things worth calling out before you apply:
imageCatalogRefpoints to thepostgresql-minimal-trixiecatalog rather than pinning a tag. CNPG resolves it to the latest minimal PostgreSQL 18 image, so future patch updates require no manifest changes.pg_hbarules are explicit and critical. Without them, both roles fall through to the defaultscram-sha-256rule. Since neither role has apasswordSecret, every connection would fail silently. Thehostnossl rejectlines are belt-and-suspenders: they ensure no unencrypted connection attempt can even try.synchronousblock setsmethod: any, number: 1— either standby satisfies the durability requirement. CNPG doesn't pin a fixed synchronous standby, which means failover is clean.databaseRoleReclaimPolicy: retainon bothDatabaseRoleobjects means deleting the CRD won't drop the Postgres role. Safe default for a secrets backend.
## cnpg-stack.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: openbao-db
namespace: openbao
spec:
instances: 3
# Resolve latest minimal PG18 image from the catalog
# deployed by REQUIREMENTS_ONLY=true — no floating tags
imageCatalogRef:
apiGroup: postgresql.cnpg.io
kind: ClusterImageCatalog
name: postgresql-minimal-trixie
major: 18
# Pin Postgres pods to dedicated tainted nodes
# and spread them across failure domains
affinity:
nodeSelector:
node-role.kubernetes.io/postgres: ""
tolerations:
- key: node-role.kubernetes.io/postgres
operator: Exists
effect: NoSchedule
enablePodAntiAffinity: true
topologyKey: topology.kubernetes.io/zone
podAntiAffinityType: required
postgresql:
# Quorum sync replication: RPO=0, writes pause if no standby available
synchronous:
method: any
number: 1
# CRITICAL: without these rules, cert auth never fires.
# Both roles have no passwordSecret — they MUST use cert auth.
# hostnossl reject rules block any cleartext connection attempt.
pg_hba:
- hostssl openbao openbao all cert
- hostssl openbao openbao-rw all cert
- hostnossl openbao openbao all reject
- hostnossl openbao openbao-rw all reject
parameters:
max_connections: '100'
log_checkpoints: 'on'
log_lock_waits: 'on'
hot_standby_feedback: 'on'
shared_memory_type: 'sysv'
dynamic_shared_memory_type: 'sysv'
storage:
size: 10Gi
---
## Schema owner — runs DDL once, never connects at runtime
apiVersion: postgresql.cnpg.io/v1
kind: DatabaseRole
metadata:
name: role-openbao
namespace: openbao
spec:
cluster:
name: openbao-db
name: openbao
login: true
clientCertificate:
enabled: true
databaseRoleReclaimPolicy: retain
---
## Application role — OpenBao's runtime connection identity
apiVersion: postgresql.cnpg.io/v1
kind: DatabaseRole
metadata:
name: role-openbao-rw
namespace: openbao
spec:
cluster:
name: openbao-db
name: openbao-rw
login: true
clientCertificate:
enabled: true
databaseRoleReclaimPolicy: retain
---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: openbao-db
namespace: openbao
spec:
name: openbao
owner: openbao
cluster:
name: openbao-db
Apply everything:
kubectl create namespace openbao
kubectl apply -f cnpg-stack.yaml
## Watch pods come up — takes 2-3 minutes on a fresh cluster
kubectl get pods -w -n openbao
## Verify cluster health once all three are Running/Ready
kubectl cnpg -n openbao status openbao-db
A healthy cluster output looks like this:
Cluster Summary
Name: openbao/openbao-db
Status: Cluster in healthy state
Instances: 3
Ready instances: 3
Streaming Replication status
Name Sync State State
openbao-db-2 quorum streaming
openbao-db-3 quorum streaming
Both standbys show Sync State: quorum simultaneously — that's method: any working correctly. Neither standby is pinned as "the" synchronous replica; either one satisfies the durability requirement.
A Note on Secret Volume Permissions
Every pod that mounts a CNPG-issued client certificate secret needs defaultMode: 0640 on the volume. Kubernetes defaults to 0644, and libpq will refuse to use a private key file that's group- or world-readable. Since mounted files stay root-owned and only the group matches the pod's fsGroup, 0640 is the correct mode. This applies to both the schema-init Job in Step 2 and the OpenBao pods in Step 3 — don't skip it.
Once the cluster reconciles, the operator creates two client certificate secrets following the <databaserole-name>-client-cert naming convention:
role-openbao-client-certrole-openbao-rw-client-cert
The openbao role (database owner) already has CREATE on the public schema by default — PostgreSQL grants this to the database owner even after the v15 change that revoked it from PUBLIC. No extra schema grant is needed before the DDL step.
Step 2: Initialize the Schema
The openbao role (schema owner) needs to run OpenBao's DDL once to create the storage and HA lock tables. We do this with a Kubernetes Job that connects using the role-openbao-client-cert secret, runs the necessary SQL, and exits.
This is a clean separation of concerns: the schema owner touches the database exactly once, and the runtime application role (openbao-rw) never needs DDL privileges.
## schema-init-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: openbao-schema-init
namespace: openbao
spec:
template:
spec:
restartPolicy: OnFailure
securityContext:
# fsGroup must match the GID that libpq runs as
# so the 0640 cert files are group-readable
fsGroup: 999
containers:
- name: psql
# Pin the image version — never use :latest in production
image: ghcr.io/cloudnative-pg/postgresql:18.6-minimal
command:
- psql
- --host=openbao-db-rw.openbao.svc
- --port=5432
- --dbname=openbao
- --username=openbao
- --file=/sql/schema.sql
env:
- name: PGSSLMODE
value: verify-full
- name: PGSSLCERT
value: /certs/tls.crt
- name: PGSSLKEY
value: /certs/tls.key
- name: PGSSLROOTCERT
value: /certs/ca.crt
volumeMounts:
- name: client-cert
mountPath: /certs
readOnly: true
- name: schema-sql
mountPath: /sql
readOnly: true
volumes:
- name: client-cert
secret:
secretName: role-openbao-client-cert
# libpq rejects keys at 0644 — must be 0640 or stricter
defaultMode: 0640
- name: schema-sql
configMap:
name: openbao-schema-sql
The schema SQL itself creates the two tables OpenBao's PostgreSQL backend expects:
-- openbao-schema.sql
-- Run once by the schema owner (openbao role)
-- openbao-rw gets DML privileges only — no DDL
CREATE TABLE IF NOT EXISTS vault_kv_store (
parent_path TEXT COLLATE "C" NOT NULL,
path TEXT COLLATE "C",
key TEXT COLLATE "C",
value BYTEA,
CONSTRAINT pkey PRIMARY KEY (path, key)
);
CREATE INDEX IF NOT EXISTS parent_path_idx
ON vault_kv_store (parent_path);
CREATE TABLE IF NOT EXISTS vault_ha_locks (
ha_key TEXT COLLATE "C" NOT NULL,
ha_identity TEXT COLLATE "C" NOT NULL,
ha_value TEXT COLLATE "C",
valid_until TIMESTAMP WITH TIME ZONE NOT NULL,
CONSTRAINT ha_key PRIMARY KEY (ha_key)
);
-- Grant DML only to the runtime role — no DDL, no schema ownership
GRANT SELECT, INSERT, UPDATE, DELETE ON vault_kv_store TO "openbao-rw";
GRANT SELECT, INSERT, UPDATE, DELETE ON vault_ha_locks TO "openbao-rw";
Package the SQL into a ConfigMap and apply:
kubectl create configmap openbao-schema-sql \
--from-file=schema.sql=openbao-schema.sql \
-n openbao
kubectl apply -f schema-init-job.yaml
## Watch the job complete
kubectl logs -f job/openbao-schema-init -n openbao
Once the job completes successfully, the schema is in place and openbao-rw has exactly the DML privileges it needs — nothing more.
Step 3: Deploy OpenBao
With the database cluster healthy and the schema initialized, we're ready to deploy OpenBao itself. The configuration mounts the role-openbao-rw-client-cert secret for the runtime database connection and sets ha_enabled = "true" so OpenBao uses the vault_ha_locks table for leader election across its replicas.
For a production-grade deployment, use the official OpenBao Helm chart and override values to point at the CNPG cluster:
## openbao-values.yaml — Helm overrides for the OpenBao chart
server:
replicas: 3
# Resource limits are non-negotiable in production
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 512Mi
cpu: 500m
# Health checks — OpenBao exposes these natively
readinessProbe:
enabled: true
path: /v1/sys/health?standbyok=true
livenessProbe:
enabled: true
path: /v1/sys/health?standbyok=true
extraEnvironmentVars:
VAULT_LOG_LEVEL: info
VAULT_LOG_FORMAT: json
# Mount the runtime client cert secret
volumes:
- name: db-client-cert
secret:
secretName: role-openbao-rw-client-cert
defaultMode: 0640
volumeMounts:
- name: db-client-cert
mountPath: /vault/db-certs
readOnly: true
# Pod anti-affinity: spread OpenBao replicas across nodes
# They can't go to postgres-tainted nodes, so they compete
# for the two general-purpose nodes — require spread here
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: openbao
topologyKey: kubernetes.io/hostname
ha:
enabled: true
replicas: 3
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
# In production, configure TLS here
}
storage "postgresql" {
connection_url = "postgres://[email protected]:5432/openbao?sslmode=verify-full&sslcert=/vault/db-certs/tls.crt&sslkey=/vault/db-certs/tls.key&sslrootcert=/vault/db-certs/ca.crt"
ha_enabled = "true"
ha_table = "vault_ha_locks"
}
service_registration "kubernetes" {}
Deploy with Helm:
helm repo add openbao https://openbao.github.io/openbao
helm repo update
helm install openbao openbao/openbao \
--namespace openbao \
--values openbao-values.yaml \
--wait
## Verify pods are running
kubectl get pods -n openbao -l app.kubernetes.io/name=openbao
Initialize and Unseal OpenBao
Fresh OpenBao deployments need to be initialized and unsealed. In production, you'd use auto-unseal with a KMS key. For this setup:
## Initialize with 5 key shares, 3 required to unseal
kubectl exec -n openbao openbao-0 -- bao operator init \
-key-shares=5 \
-key-threshold=3
## Store the unseal keys and root token securely — you only see them once
## Then unseal each pod (repeat for openbao-1 and openbao-2)
kubectl exec -n openbao openbao-0 -- bao operator unseal <unseal-key-1>
kubectl exec -n openbao openbao-0 -- bao operator unseal <unseal-key-2>
kubectl exec -n openbao openbao-0 -- bao operator unseal <unseal-key-3>
## Verify status
kubectl exec -n openbao openbao-0 -- bao status
Production Hardening and Observability
A working deployment isn't a production-ready deployment. Here's what you need to add before this stack handles real secrets.
Backup Strategy for the PostgreSQL Backend
OpenBao's entire secrets store lives in that PostgreSQL cluster. You need continuous WAL archiving configured on the CNPG Cluster object. Add a backup stanza pointing at an S3-compatible store via the Barman Cloud plugin (which the playground already installed):
## Add to the Cluster spec
backup:
barmanObjectStore:
destinationPath: s3://your-bucket/openbao-db-backup
s3Credentials:
accessKeyId:
name: s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: s3-creds
key: SECRET_ACCESS_KEY
retentionPolicy: 30d
Without this, a cluster-wide failure means total secrets loss. Don't skip it.
Monitoring
Both CNPG and OpenBao expose Prometheus metrics natively. Add PodMonitor objects for both if you're running the Prometheus Operator:
## CNPG exposes metrics on port 9187 of each instance pod
## OpenBao exposes metrics at /v1/sys/metrics?format=prometheus on port 8200
## Quick health check
kubectl exec -n openbao openbao-0 -- \
bao read sys/health
Key alerting thresholds to configure:
- OpenBao sealed state (any replica)
- PostgreSQL primary failover events
- Replication lag > 5 seconds
- WAL archiving failures
- Disk usage > 80% on PostgreSQL PVCs
Network Policies
Lock down traffic with NetworkPolicy objects. Only OpenBao pods should reach the CNPG cluster on port 5432. Only your application pods should reach OpenBao on port 8200.
## Deny all ingress to the openbao namespace by default,
## then add explicit allow rules per workload
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: openbao
spec:
podSelector: {}
policyTypes:
- Ingress
For managed Kubernetes environments where you want someone else handling the operator-level complexity, NinjaIT's managed cloud services can take the operational burden off your team while you focus on the application layer.
Troubleshooting Common Issues
OpenBao pods crash with connection refused to PostgreSQL:
Verify the CNPG cluster is fully healthy first (kubectl cnpg status). Check that the role-openbao-rw-client-cert secret exists and has tls.crt, tls.key, and ca.crt keys. Confirm PGSSLMODE=verify-full is set and the CA cert matches the CNPG cluster's CA.
libpq rejects the private key file:
The cert secret volume is mounted with the wrong defaultMode. It must be 0640, not the Kubernetes default of 0644. Also verify fsGroup in the pod's securityContext matches the GID that the process runs as.
OpenBao connects but can't read/write:
The openbao-rw role is missing DML grants on one or both tables. Re-run the schema init job and check the output. Also verify the pg_hba rules are in the correct order — PostgreSQL uses the first matching rule.
Pods stuck in Pending on the playground cluster:
OpenBao pods can't schedule on postgres-tainted nodes, and with podAntiAffinityType: required, they need separate nodes. The playground has exactly two general-purpose nodes. Three OpenBao replicas won't fit — either relax anti-affinity to preferred for local testing or reduce replicas to 2.
Conclusion: A Fully Open-Source Secrets Backend Worth Running in Production
This OpenBao on Kubernetes deployment with a CloudNativePG PostgreSQL backend gives you something most secrets management tutorials don't: a genuinely production-ready stack with no proprietary dependencies, no cloud database lock-in, and no passwords in the connection chain. The combination of CNPG's self-healing replication, DatabaseRole-issued mTLS certificates, and OpenBao's native PostgreSQL storage backend is architecturally sound and operationally maintainable.
The key principles that make this work — 12-factor config via environment variables and mounted secrets, declarative infrastructure via CRDs, explicit authentication rules rather than implicit defaults — are the same ones that will keep this stack maintainable six months from now when you're debugging a 3am incident.
If you're moving this to a real cluster, start with a reliable VPS or cloud environment that gives you the node count and disk IOPS that a synchronously replicated PostgreSQL cluster actually needs. And check out the Data Mammoth blog for more deep dives on cloud-native data infrastructure.
For related reading, see our guides on Read more about this topic and Read more about this topic.