Alerts & Alert Sources

This guide covers the alert record a responder works and the external connections that can create one. Both native telemetry alerts and externally delivered alerts use the same grouping, routing, paging, and incident paths.

On-call alert lifecycle

An on-call alert is a page routed to a responder through an escalation policy.

Alert states

• Triggered — an on-call alert was raised and notifications are being sent.
• Acknowledged — a responder has acknowledged it and is investigating.
• Resolved — the on-call alert has been resolved.

Alert actions

While an on-call alert is active, responders can:

• Acknowledge — stop escalation and indicate that you are working on it.
• Resolve — mark the on-call alert as resolved.
• Reassign — transfer the on-call alert to another team member.
• Escalate now — start the configured escalation path immediately.
• Add notes — document findings and actions taken.

Every on-call alert has an audit timeline showing when it was raised, who was notified, when it was acknowledged, notes added, and when it was resolved. Open Alerts to inspect the full record and its matched route, group, paging, and incident outcomes.

When a send fails

If a message, page, or incident fails to send, for example because Slack or an incident provider is unavailable, Moneat sends that part again as the same notification. It retries after a minute, then waits longer each time, up to half an hour, and stops after ten tries. A later notification for the alert takes its place, and a silence that has begun since holds the retry until the silence ends. The parts that went out the first time are not sent again.

A host-down alert that cannot be raised at all is raised again after a minute, then less often, up to every 15 minutes, until it is raised.

Error alerts notify, and page only when a route asks

An error alert is raised for an issue: New Issue for a new one, and Regressed when a resolved issue occurs again. An issue is work to look at, not an outage, so an error alert notifies your team by email, Slack and the other channels in your alert notifications, and it pages nobody by itself.

An error alert takes an alert route only when the route's matched condition group names its source, such as Source is Error alert. A route with no conditions, or one that matches on priority or project alone, does not take error alerts; route preview shows it as skipped with the reason Error alerts only take a route with a "Source is Error alert" condition. To page for errors in one project, add a route with Source is Error alert and a project condition.

Moneat raises a New Issue alert once the issue matters:

• Server projects, such as Node, Python, Java, Go or Ktor, raise it on the issue's first event. Your users never see a server error, so one is worth a look.
• App and browser projects raise it once 3 different users hit the issue in production within 24 hours, or 10 production events that carry no user id. Moneat checks every minute, so the alert goes out even when no further event arrives. One broken device or a flaky network starts a new issue on any app; a few users is the first sign that a release broke something. An issue that takes more than 7 days to get there raises no New Issue alert.
• Events from a non-production environment, such as staging, development, debug, test, uat or preprod, never raise an alert and don't reopen a resolved issue. A name counts as non-production when any of its parts does, so staging-eu and non-prod count too. An event with no environment counts as production.

A project set up without a platform, or with one Moneat doesn't know, is treated as an app when its SDK is a mobile, desktop or browser SDK, and as a server otherwise.

An app or browser issue that never reaches enough users, and a server issue that never notified, still shows up: the daily digest lists each day's new production issues, and release health reports list each release's.

Errors right after a deploy

A server issue first seen within 10 minutes of a deploy waits before it notifies. Restarts, cold caches and connections dropped while instances roll make errors that never come back, so the issue notifies only if it occurs again in production once those 10 minutes are over. Moneat takes the deploy time from production deploys you report, or else from when it first saw the newest release.

Health alerts page

Health alerts watch how your users are doing in aggregate, which is what an outage looks like. Moneat checks every 5 minutes and raises a Health monitor alert at priority P1 while a check fails:

• Crash rate: at least 2% of the production sessions active in the last 30 minutes crashed, across at least 200 sessions and 10 users, and the rate is at least 10% or at least twice the past week's. It needs session tracking turned on in your SDK.
• Critical operations: at least 10% of the attempts at an operation you marked critical failed in the last 15 minutes, across at least 20 attempts and 5 users. A failure with no user id counts as its own user. Mark operations critical, such as sign-in or checkout, under Critical operations in the service's settings; a service can have up to 20.

A health alert takes alert routes like any other, so a route with no conditions pages for it. It stays open while the check keeps failing and resolves on its own about 15 minutes after it passes again.

Spikes notify

A known issue whose production events in the last hour reach 5 times its usual hourly rate over the past week, with at least 20 events and 5 users, raises a Spiking error alert. Like the other error alerts it notifies your alert channels and pages only through a route that names its source. It is raised once a day per issue at most, and an issue you resolved or ignored never raises one.

Tune noisy alerts

On an alert's page, say whether it was Actionable, Expected, a Duplicate, or raised at the Wrong urgency. Each Monday at 9 AM, owners and admins get an email with the past week's noisiest alerts: the ones that fired three or more times, or that someone marked as noise. For each one it shows how often it fired, how often anyone acknowledged it, and how often it resolved without anyone acknowledging it. Tune, route, or turn off the ones that keep resolving on their own. Alerts on private incidents are left out. Turn the email off with Noisiest alerts in Settings › Notifications.

How alerts resolve

An alert resolves, and its pages close with it, when any of these happens:

• Its source reports that the problem cleared.
• A responder resolves its page. If the source is still failing, its next report raises a new alert that pages again under the usual rules.
• The incident it belongs to is resolved or closed.
• For an error alert, its issue is resolved, ignored or deleted, or a day passes with no new events. The issue itself stays unresolved until someone resolves it, and its later events raise no new alert. Moneat raises one again only if the issue is resolved and then regresses.
• For a host alert, the host is deleted. A host-down alert also resolves once the host reports again.
• For an alert from an alert source, the sender stops updating it for longer than the source's quiet period. See Resolving alerts a sender goes quiet on.

Alert sources

Moneat raises on-call alerts from its own telemetry, and it can also accept alerts from systems you already run. An alert source is a named ingress endpoint with its own token, configuration, and delivery history. Alerts arriving through a source join the same lifecycle, grouping, paging, and Alert Route evaluation as alerts Moneat raises itself.

Manage sources under Settings → Alerting → Sources. Creating, editing, rotating a token, and replaying an event require an organization owner or admin role, and the API enforces the same rule.

Creating a source

1. Choose an adapter. Datadog, Grafana, and Generic HTTP deliver over a webhook; PagerDuty also uses an API token so Moneat can read its service catalog.
2. Give the source a name. The name appears on alerts raised from it, so name it after the system that sends them.
3. Choose who gets paged. A paging choice adds an Alert Route for the source just ahead of the first route with no conditions, so a catch-all route cannot take its alerts first. Don’t page, just record adds a route for the source at the top of the order that pages nobody, so no other route pages for its alerts.
4. Copy the ingress token shown on the last step. Moneat stores only a hash of it, so this is the one time it can be displayed. If you lose it, rotate the token rather than recreating the source.

Each source has a fixed ingress endpoint:

POST https://<your-moneat-host>/v1/alert-events/<source-id>
Authorization: Bearer <ingress-token>
Content-Type: application/json

Requests are rate limited and bodies are capped at 512 KB. Send an optional X-Alert-Event-ID header to make retries idempotent. Without one, Moneat derives an identity from the payload so an identical retry still folds into a single event.

Source health

The list shows what a source is actually doing, not just how it is configured:

• Receiving — events are arriving and being turned into alerts.
• Connecting — the connection exists but has not finished its first sync.
• Failing — recent events could not be processed. Open Events to see why.
• No events yet — nothing has been delivered to the endpoint yet.
• Disabled — the source is turned off and deliveries are rejected.

While a source has never received an event, its Setup tab watches for the first delivery and updates on its own when one lands.

Deliveries and replay

The Events tab records the most recent 100 deliveries with their state, timings, resulting alert count, request headers, and payload. Filter to Problems to see only deliveries that carried a parse warning or a processing error.

• A parse warning means the payload was accepted but did not match the expected shape, so the alert was raised with placeholder fields.
• A processing error means the delivery could not be turned into an alert at all.

Replay reruns a stored payload through the source's current configuration, which is how you verify a mapping change against a real request. A payload that has already been processed is recognized as a duplicate and is not sent again.

Configuration revisions and token rotation

Every save writes a new configuration revision, shown on the detail header and in Settings. New deliveries use the latest revision immediately; existing alerts are unchanged.

Rotating the ingress token invalidates the previous token the moment the new one is issued, so the sending system is rejected until it is updated. The new token, like the original, is shown once.

Resolving alerts a sender goes quiet on

Some senders never send a resolve. In Settings, Resolve alerts after a quiet period takes a number of minutes: each delivery for an alert renews it, and an alert the sender stops updating resolves once that long passes without one. Leave it blank to keep alerts open until the sender resolves them. Alerts Moneat could not parse can never be resolved by their sender, so without this setting they resolve after a day without updates.

Adapter setup

Datadog

Moneat receives Datadog monitor transitions through a webhook.

1. In Datadog, open Integrations → Webhooks and add a webhook.

2. Set its Name to the handle Moneat shows on the Setup tab, and its URL to the ingress endpoint.

3. Add the bearer token under Custom Headers:

   JSON

   {"Authorization": "Bearer <your-ingress-token>"}

4. Set the Payload to the recommended template:

   JSON

   {
     "title": "$ALERT_TITLE",
     "transition": "$ALERT_TRANSITION",
     "deduplication_key": "$ALERT_CYCLE_KEY",
     "priority": "$ALERT_PRIORITY",
     "scope": "$ALERT_SCOPE",
     "alert_id": "$ALERT_ID",
     "message": "$TEXT_ONLY_MSG",
     "link": "$LINK"
   }

   ALERT_TRANSITION decides whether the alert fires or resolves, and ALERT_CYCLE_KEY folds a trigger and its recovery into one alert.

5. Add the webhook handle shown on the Setup tab to the message of every monitor that should page.

Grafana

Moneat receives Grafana Alerting notifications through a webhook contact point.

1. In Grafana, open Alerting → Contact points and add a contact point using the Webhook integration, with the ingress endpoint as its URL.

2. Under Optional Webhook settings, set Authorization Header Scheme to Bearer and Authorization Header Credentials to the ingress token. Leave the HTTP method at POST.

3. Point a notification policy at the contact point.

4. Choose how a Grafana notification becomes Moneat alerts:

   • One alert per notification — the whole group becomes one alert, deduplicated on the notification group key. If a Grafana rule has 30 firing alert instances, Moneat creates one grouped alert.
   • One alert per series — each entry in the notification becomes its own alert, deduplicated on that series' fingerprint and resolving independently. Choose One alert per series if you want those 30 instances to page independently. This creates separate Moneat alerts; the matching Alert Route must also page every episode and group each series separately.

Generic HTTP

Any system that can post JSON can raise alerts. Describe where each alert field lives with a path, and Moneat reads the value at that path.

Paths are declarative data, not code. Only the payload root, named properties, and numeric array indexes are supported — for example $.title, $.alert.labels.service, or $.alerts[0].fingerprint. Filters, wildcards, recursive descent, and expressions are deliberately unsupported, so a mapping can never be the thing that executes.

Title and Deduplication key are required. A payload that does not yield both raises an unparsed alert with a warning rather than being dropped. Repeat deliveries carrying the same deduplication key fold into one alert, and a status of resolved, resolve, recovered, ok, closed, cleared, or normal resolves it. Anything else keeps it firing.

The editor resolves each path against a sample payload as you type and previews the exact alert the server would build, so you can verify a mapping before any traffic arrives.

PagerDuty

PagerDuty needs two connections. Setting up only one is the usual reason a source looks finished but pages no one.

1. API token — create a read-only key under Integrations → API Access Keys in PagerDuty and paste it into the Setup tab. It is encrypted at rest and used only to read the service catalog.
2. Catalog sync — pull the service list. Connection and sync state show the last successful sync time.
3. Service selection — choose which services page through Moneat. With nothing selected, incidents from every service are accepted.
4. Webhook subscription — add a generic webhook (v3) pointing at the ingress endpoint, with the bearer token as a custom header, subscribed to incident.triggered, incident.reopened, and incident.resolved. Other event types are accepted and ignored.

Alert groups

Alert Groups organize alert episodes; they are not native incidents. Open a group to review its state, paging result, grouping window, route behavior, and any incident relationship. An alert-only group can page and group episodes without creating an incident.

Suggested relationships remain reviewable: responders can mark an episode unrelated, remove it from the group, or attach it to an incident. Automatic grouping can attach matching episodes and create the configured incident or triage record without manual confirmation.

Alerts raised from a source are evaluated by Alert Routes like any other alert. Paging, grouping, and incident creation are configured there rather than per source.