Prometheus Alertmanager

Create incidents from authenticated Alertmanager webhook deliveries.

Connect Alertmanager to create and update incidents from firing and resolved alerts. OpenCluster receives webhooks; it does not call Alertmanager.

What OpenCluster uses it for

OpenCluster records the alert status, times, labels, annotations, fingerprint, and generator URL. It uses Alertmanager’s grouping key to group delivered alerts into an incident. The triggering alert becomes the starting context for an investigation.

Prerequisites

• Prometheus Alertmanager 0.29.0 or later. Version 0.29.0 added the custom HTTP header configuration used for the webhook secret.
• Network access from Alertmanager to the OpenCluster intake endpoint over HTTPS.
• The Admin role in OpenCluster.

Access

This integration is inbound only. The generated secret authenticates webhook deliveries in the X-OpenCluster-Token header. OpenCluster stores only a digest of the secret and cannot show it again.

Connect

1. Create the integration

   In Integrations, choose Prometheus Alertmanager, enter a name, and create the integration. Copy the webhook URL and secret from the response.

2. Configure Alertmanager

   Add the receiver, then route alerts to it:

   receivers:
     - name: opencluster
       webhook_configs:
         - url: "https://oc.example.com/intake/v1/integrations/<integration-id>/signals"
           send_resolved: true
           http_config:
             http_headers:
               X-OpenCluster-Token:
                 secrets:
                   - "<webhook-secret>"
   
   route:
     routes:
       - receiver: opencluster
         continue: true
   

   Reload Alertmanager after validating the configuration.

Verify

Fire a test alert, open the integration, and select Verify. Verification reports the time of the last accepted delivery. If no delivery has arrived, check the receiver, route, URL, secret, and network path.

During investigations

The alert supplies the incident timing, labels, annotations, runbook and dashboard URLs when present, and the generator URL. Alertmanager is not a live investigation source; OpenCluster cannot query silences, inhibitions, or other active alerts.

Delivery behavior

Response	Meaning
202	The delivery was accepted.
200	The same body was already accepted; it was not applied twice.
400	The payload is invalid and should not be retried unchanged.
401	The secret is wrong, was rotated, or the integration is disabled.
413	The body is larger than 1 MiB.
429	The integration exceeded its delivery rate; retry after Retry-After.
503	OpenCluster could not record the delivery; retry.

Deliveries can contain at most 2,048 alerts. Only firing and resolved statuses are accepted. If Alertmanager marks a delivery as truncated, OpenCluster records that the incident context is incomplete.

Troubleshooting

• Verify shows no delivery: send a firing test alert and check Alertmanager logs.
• 401: replace the configured secret with the current value. Rotated secrets stop working immediately.
• 400: confirm middleware has not changed Alertmanager’s version 4 webhook body.
• 429: reduce delivery frequency or group alerts more aggressively.

Rotate or disconnect

Rotating creates a new secret and invalidates the old one immediately. Update Alertmanager in the same maintenance step. Disable the integration to reject new deliveries while retaining incident history. Deletion is refused when retained records depend on the integration.