Skip to content
flowss
Start freeChecking your session
Sequence & interaction templates

Sev-1 paging escalation — Alertmanager to PagerDuty to on-call

Interaction diagram of a Sev-1 page from Prometheus through Alertmanager and PagerDuty to primary, secondary and manager escalation levels with ack timers, status-page update and resolution, for the engineer who owns the escalation policy.

Sequence & interactionOperations & OpsSequence & interaction syntax docs
Template previewSequence & interaction
Sev-1 paging escalation — checkout API23 messages · deepest 1 openaltacknowledgement · primary acksprimary times outboth time outPOST /alerts burn14.4x for 2m200 acceptedenqueuededup=chk-burn202 acceptedincident INC-4417 opened, Sev-1page level 1: push,SMS, voiceacknowledge at t+2minassigned, escalationhaltedpage level 2 at t+5 minacknowledge at t+7 minassigned, escalation haltedpage level 2 at t+5 minpage level 3 at t+15 minacknowledge at t+16 minassigned, escalation haltedPOST /incidents status=investigatingpublic status page updatedstatus page link postedburn below 1x for 30min200 acceptedresolvededup=chk-burnINC-4417 resolved,t+41 minpostmortem due within 5 working daysPrometheusAlertmanagerPagerDutyPrimary OCSecondaryEng manager#inc-checkoutStatuspageAll 17 checks ran over 8 lifelines and 23 messages and every one came back clean.INFO8 lifelines, 23 messages, 1 fragment. 8 synchronous calls, 8 answered; the deepest point of the interaction has 1 call open at once. Every message states whether it is a call, a reply or afire-and-forget, so the stack above was simulated rather than guessed at.OK17 of the 17 checks ran over every instance of their subject and found nothing: misdirected-reply, orphan-reply, conditional-reply, unreturned-call, deadlock, after-destroy, before-create,double-create, double-destroy, par-interference, guard-overlap, guard-incomplete, activation-claim, unused-participant, undeclared-lifeline, duplicate-id, fragment-unclosed.

Make it your own.

sequence "Sev-1 paging escalation — checkout API"

participant Prom  "Prometheus"
participant AM    "Alertmanager"
participant PD    "PagerDuty"
actor       P1    "Primary OC"
actor       P2    "Secondary"
actor       EM    "Eng manager"
participant Slack "#inc-checkout"
participant SP    "Statuspage"

Prom -> AM         : POST /alerts burn 14.4x for 2m
AM   reply Prom    : 200 accepted
AM   -> PD         : enqueue dedup=chk-burn
PD   reply AM      : 202 accepted
PD   async Slack   : incident INC-4417 opened, Sev-1
PD   async P1      : page level 1: push, SMS, voice

alt on acknowledgement : "primary acks","primary times out","both time out"
  when "primary acks"
    P1   -> PD     : acknowledge at t+2 min
    PD   reply P1  : assigned, escalation halted
  when "primary times out"
    PD   async P2  : page level 2 at t+5 min
    P2   -> PD     : acknowledge at t+7 min
    PD   reply P2  : assigned, escalation halted
  when "both time out"
    PD   async P2  : page level 2 at t+5 min
    PD   async EM  : page level 3 at t+15 min
    EM   -> PD     : acknowledge at t+16 min
    PD   reply EM  : assigned, escalation halted
end

PD   -> SP         : POST /incidents  status=investigating
SP   reply PD      : public status page updated
PD   async Slack   : status page link posted

Prom -> AM         : burn below 1x for 30 min
AM   reply Prom    : 200 accepted
AM   -> PD         : resolve dedup=chk-burn
PD   reply AM      : INC-4417 resolved, t+41 min
PD   async Slack   : postmortem due within 5 working days