Operations

Monitor health, deploy safely, manage retention, and understand background work.

Use the health listener for probes and metrics. Keep operator, intake, and Relay listeners on separate network policies.

Health and metrics

• /healthz reports process liveness without checking dependencies.
• /readyz reports whether this instance can reach its required PostgreSQL placement.
• /metrics exposes Prometheus metrics without organization labels.

Alert on sustained readiness failures, failed or degraded integration verification, disconnected or outdated Relays, and failed investigations.

Background work

Work	Schedule or trigger
Audit retention	Hourly, using each organization’s retention setting
Kubernetes change-history retention	Hourly, using OC_CHANGE_LEDGER_RETENTION_DAYS
Relay inventory synchronization	OC_INVENTORY_INTERVAL
Investigation execution	After an investigation opens, with bounded concurrency

Deploy and stop

Migrations are forward-only and run under a database lock at startup. Multiple new instances can start without racing the schema. Review release notes and back up each placement before upgrading.

On SIGTERM, OpenCluster stops accepting new work and drains requests within OC_SHUTDOWN_TIMEOUT. Investigations still running when shutdown completes are marked failed rather than left running.

Integration health

Verification is a point-in-time check. Alertmanager verification reports the last accepted delivery, so use its timestamp rather than status alone. Slack and GitHub verification checks the vendor live. Kubernetes verification checks the current Relay session and advertised capabilities.

Retention

Audit entries use each organization’s configured retention. Kubernetes change history uses the deployment setting. Signals, incidents, investigations, and investigation provenance remain operational records and can prevent integration deletion.

See Troubleshooting for common failure states.