Alerts & Alert Sources

This guide covers the alert record a responder works and the external connections that can create one. Both native telemetry alerts and externally delivered alerts use the same grouping, routing, paging, and incident paths.

On-call alert lifecycle

An on-call alert is a page routed to a responder through an escalation policy.

Alert states

• Triggered — an on-call alert was raised and notifications are being sent.
• Acknowledged — a responder has acknowledged it and is investigating.
• Resolved — the on-call alert has been resolved.

Alert actions

While an on-call alert is active, responders can:

• Acknowledge — stop escalation and indicate that you are working on it.
• Resolve — mark the on-call alert as resolved.
• Reassign — transfer the on-call alert to another team member.
• Escalate now — start the configured escalation path immediately.
• Add notes — document findings and actions taken.

Every on-call alert has an audit timeline showing when it was raised, who was notified, when it was acknowledged, notes added, and when it was resolved. Open Alerts to inspect the full record and its matched route, group, paging, and incident outcomes.

Alert sources

Moneat raises on-call alerts from its own telemetry, and it can also accept alerts from systems you already run. An alert source is a named ingress endpoint with its own token, configuration, and delivery history. Alerts arriving through a source join the same lifecycle, grouping, paging, and Alert Route evaluation as alerts Moneat raises itself.

Manage sources under On-Call → Sources. Creating, editing, rotating a token, and replaying an event require an organization owner or admin role, and the API enforces the same rule.

Creating a source

1. Choose an adapter. Datadog, Grafana, and Generic HTTP deliver over a webhook; PagerDuty also uses an API token so Moneat can read its service catalog.
2. Give the source a name. The name appears on alerts raised from it, so name it after the system that sends them.
3. Copy the ingress token shown on the last step. Moneat stores only a hash of it, so this is the one time it can be displayed. If you lose it, rotate the token rather than recreating the source.

Each source has a fixed ingress endpoint:

POST https://<your-moneat-host>/v1/alert-events/<source-id>
Authorization: Bearer <ingress-token>
Content-Type: application/json

Requests are rate limited and bodies are capped at 512 KB. Send an optional X-Alert-Event-ID header to make retries idempotent. Without one, Moneat derives an identity from the payload so an identical retry still folds into a single event.

Source health

The list shows what a source is actually doing, not just how it is configured:

• Receiving — events are arriving and being turned into alerts.
• Connecting — the connection exists but has not finished its first sync.
• Failing — recent events could not be processed. Open Events to see why.
• No events yet — nothing has been delivered to the endpoint yet.
• Disabled — the source is turned off and deliveries are rejected.

While a source has never received an event, its Setup tab watches for the first delivery and updates on its own when one lands.

Deliveries and replay

The Events tab records the most recent 100 deliveries with their state, timings, resulting alert count, request headers, and payload. Filter to Problems to see only deliveries that carried a parse warning or a processing error.

• A parse warning means the payload was accepted but did not match the expected shape, so the alert was raised with placeholder fields.
• A processing error means the delivery could not be turned into an alert at all.

Replay reruns a stored payload through the source's current configuration, which is how you verify a mapping change against a real request. A payload that has already been processed is recognized as a duplicate and is not sent again.

Configuration revisions and token rotation

Every save writes a new configuration revision, shown on the detail header and in Settings. New deliveries use the latest revision immediately; existing alerts are unchanged.

Rotating the ingress token invalidates the previous token the moment the new one is issued, so the sending system is rejected until it is updated. The new token, like the original, is shown once.

Adapter setup

Datadog

Moneat receives Datadog monitor transitions through a webhook.

1. In Datadog, open Integrations → Webhooks and add a webhook.

2. Set its Name to the handle Moneat shows on the Setup tab, and its URL to the ingress endpoint.

3. Add the bearer token under Custom Headers:

   JSON

   {"Authorization": "Bearer <your-ingress-token>"}

4. Set the Payload to the recommended template:

   JSON

   {
     "title": "$ALERT_TITLE",
     "transition": "$ALERT_TRANSITION",
     "deduplication_key": "$ALERT_CYCLE_KEY",
     "priority": "$ALERT_PRIORITY",
     "scope": "$ALERT_SCOPE",
     "alert_id": "$ALERT_ID",
     "message": "$TEXT_ONLY_MSG",
     "link": "$LINK"
   }

   ALERT_TRANSITION decides whether the alert fires or resolves, and ALERT_CYCLE_KEY folds a trigger and its recovery into one alert.

5. Add the webhook handle shown on the Setup tab to the message of every monitor that should page.

Grafana

Moneat receives Grafana Alerting notifications through a webhook contact point.

1. In Grafana, open Alerting → Contact points and add a contact point using the Webhook integration, with the ingress endpoint as its URL.

2. Under Optional Webhook settings, set Authorization Header Scheme to Bearer and Authorization Header Credentials to the ingress token. Leave the HTTP method at POST.

3. Point a notification policy at the contact point.

4. Choose how a Grafana notification becomes Moneat alerts:

   • One alert per notification — the whole group becomes one alert, deduplicated on the notification group key. A rule matching thirty series pages once.
   • One alert per series — each entry in the notification becomes its own alert, deduplicated on that series' fingerprint and resolving independently.

Generic HTTP

Any system that can post JSON can raise alerts. Describe where each alert field lives with a path, and Moneat reads the value at that path.

Paths are declarative data, not code. Only the payload root, named properties, and numeric array indexes are supported — for example $.title, $.alert.labels.service, or $.alerts[0].fingerprint. Filters, wildcards, recursive descent, and expressions are deliberately unsupported, so a mapping can never be the thing that executes.

Title and Deduplication key are required. A payload that does not yield both raises an unparsed alert with a warning rather than being dropped. Repeat deliveries carrying the same deduplication key fold into one alert, and a status of resolved, recovered, ok, closed, or normal resolves it. Anything else keeps it firing.

The editor resolves each path against a sample payload as you type and previews the exact alert the server would build, so you can verify a mapping before any traffic arrives.

PagerDuty

PagerDuty needs two connections. Setting up only one is the usual reason a source looks finished but pages no one.

1. API token — create a read-only key under Integrations → API Access Keys in PagerDuty and paste it into the Setup tab. It is encrypted at rest and used only to read the service catalog.
2. Catalog sync — pull the service list. Connection and sync state show the last successful sync time.
3. Service selection — choose which services page through Moneat. With nothing selected, incidents from every service are accepted.
4. Webhook subscription — add a generic webhook (v3) pointing at the ingress endpoint, with the bearer token as a custom header, subscribed to incident.triggered, incident.reopened, and incident.resolved. Other event types are accepted and ignored.

Alert groups

Alert Groups organize alert episodes; they are not native incidents. Open a group to review its state, paging result, grouping window, route behavior, and any incident relationship. An alert-only group can page and group episodes without creating an incident.

Suggested relationships remain reviewable: responders can mark an episode unrelated, remove it from the group, or attach it to an incident. Automatic grouping can attach matching episodes and create the configured incident or triage record without manual confirmation.

Alerts raised from a source are evaluated by Alert Routes like any other alert. Paging, grouping, and incident creation are configured there rather than per source.