A2P SMS Incident Runbook: Detect, Scope, and Manage Delivery Degradation

What constitutes A2P SMS delivery degradation

An A2P SMS incident does not have to be a complete outage. It may appear as an increase in pending messages, a decline in final delivery statuses, more pre-send errors, higher latency to final status, or different behavior within a specific combination of destination, sender, encoding, or route.

The purpose of a runbook is not to turn an initial signal into a premature conclusion. It is to create a repeatable process for determining which part of the service is affected, what evidence exists, and which actions can be taken without making the situation worse or breaching local requirements, anti-abuse controls, or consent rules.

The first operational discipline is not to use “delivered” as a synonym for “sent.” A message accepted by an upstream provider or carrier does not, on its own, confirm delivery to the end user.

• Connectivity degradation: the HTTP or SMPP interface does not respond reliably, returns errors, or experiences increased latency.
• Acceptance degradation: requests are not accepted upstream or fail before progressing.
• Final-status degradation: the proportion of pending, undelivered, unknown, or blocked messages rises according to the available taxonomy.
• Latency degradation: final statuses still arrive, but later for a specific cohort.
• Observability degradation: sending may continue, but callbacks or event ingestion are delayed, incomplete, or failing.

Define the service and its evidence boundaries

Before declaring an incident, define what each status represents in your integration. Many platforms distinguish between queued, sent, upstream carrier acceptance, reported delivery, and failure statuses. Names may vary, but the runbook must document their specific meaning for each provider or interface.

Upstream acceptance is useful evidence of progress, not absolute proof of receipt on the handset. Likewise, a status reported as delivered depends on information received through the messaging chain. Actual receipt on the device cannot always be independently proven from provider telemetry.

You must also separate message status from the health of the observability channel. If the callback endpoint fails, messages may still have status in the sending platform even though your system has not received the event.

• Immediate send result: HTTP response or SMPP PDU, assigned identifier, and error, if any.
• Asynchronous status: a later transition received through a callback, event stream, or message-resource query.
• Reported delivery: confirmation available through the carrier or relevant channel.
• Handset receipt: a level of certainty limited by the evidence the delivery ecosystem can report.
• Observability: availability, delay, and completeness of callbacks, queries, and internal ingestion processes.

Minimum detection signals

Detection should combine volume, status, timing, and connectivity signals. A single metric is often ambiguous: a decline in delivered messages may indicate a real issue, but it may also reflect a temporary increase in messages that are still pending or a failure in event reception.

Measure status rates across comparable cohorts and separate final from non-final statuses. An increase in pending messages requires observing their progression and age before classifying it as a failure. Establish time windows that align with your operational expectations and the possibility of late events.

For latency, record acceptance, transition, and event-arrival timestamps. Analyze percentiles in addition to averages: a stable average can hide a significant queue in part of the traffic.

• Accepted, failed, and pending volume per interval.
• Proportion of final statuses by cohort, separated from messages that have not yet reached a final status.
• Latency between acceptance and final status, measured by percentiles.
• HTTP errors, provider error codes, SMPP errors, and reconnection rates.
• Response time and success of SMPP link checks using enquire_link and enquire_link_resp.
• Callback endpoint availability, response codes, retries, and ingestion delay.
• Depth, age, and drain rate of internal queues.

Segment before diagnosing

Do not diagnose a mixed cohort as though it were a homogeneous route. Segment before assigning cause or impact. The same symptom may be limited to a country, mobile network, sender, traffic type, encoding, multipart messages, or a specific time window.

Retain the attributes available in your telemetry and use the same cohort definition throughout the incident. If segmentation changes between updates, record the change and avoid comparing percentages calculated from different populations.

Encoding and number of parts deserve particular attention. Data alphabets, including UCS2, and multipart messages may not be comparable with single-part messages using the default alphabet.

• Destination: country, mobile network where available, and internally defined prefix or grouping.
• Origin: sender, identifier type, and applicable configuration.
• Traffic: OTP, transactional, alerts, or other legitimate operational groups.
• Technical content: encoding, number of parts, and content template or version where recorded.
• Route or provider: only where that attribute is available and reliable in your own telemetry.
• Time: observed start, affected interval, and comparison with an equivalent reference window.

Initial incident classification

Initial classification should not assert a root cause. It should guide triage, mitigation, and escalation. Use categories that can be revised as late events or additional evidence arrive.

A successful SMPP enquire_link check confirms the basic health of the protocol link when the correlated response is received successfully. It does not confirm that messages are reaching handsets. Connectivity and delivery should therefore remain separate dimensions in the incident record.

• Interface connectivity: session errors, timeouts, authentication failures, reconnections, or lack of link response.
• Pre-send failure: rejected requests or messages that do not progress from the sending system.
• Acceptance without final status: accepted messages accumulating age without later confirmation.
• Final non-delivery: an increase in undelivered, expired, unreachable, or other reported final codes.
• Reported blocking or filtering: statuses or codes associated with spam, carrier blocking, or policy, where exposed by the channel.
• Callback or observability failure: status may exist at the source but does not reach your system, or arrives late.
• Measurement anomaly: instrumentation changes, duplicates, event loss, or incorrect segmentation.

Step-by-step triage in the first minutes

The first objective is to preserve the ability to investigate. Before changing routes, limits, or configurations, capture a snapshot of metrics and a representative sample of messages. Then confirm whether the issue is in sending, acceptance, reported delivery, or observability.

Work from general to specific. First check for a shared platform or connectivity issue; then narrow the issue by cohort. Avoid stating that a mobile network or provider is the cause without controlled comparison and correlatable data.

• 1. Open the incident record with detection time, detector, symptom, involved services, and scope still unknown.
• 2. Capture status rates, pending-age distribution, latency percentiles, interface errors, callback health, and queue depth.
• 3. Verify HTTP or SMPP connectivity. For SMPP, verify correct correlation between enquire_link and enquire_link_resp.
• 4. Compare the affected interval with an equivalent reference window and segment by destination, sender, traffic, encoding, parts, and available route.
• 5. Review a sample of message identifiers with their initial responses, original statuses, errors, and timestamps.
• 6. Query status from the source platform when callbacks are absent or suspected to be unreliable.
• 7. Classify provisionally, assign a technical owner, and decide whether limited mitigation is appropriate.
• 8. Record every change, when it was made, the cohort it applies to, and the observed result.

Test messages: useful, but not conclusive

Test messages are useful for checking specific hypotheses, not for proving the status of an entire route on their own. A single device may have particular conditions, such as coverage, temporary availability, configuration, storage, or device-specific behavior.

Use authorized destinations and properly normalized numbers, usually in accordance with the international E.164 numbering plan where applicable. Keep testing minimal, traceable, and equivalent to the cohort you want to observe: the same destination or destination group, sender, permitted traffic class, encoding, and message structure.

Do not use tests to bypass filters, anti-abuse controls, consent requirements, or local obligations. If a signal points to blocking or filtering, the appropriate response is to review compliance, configuration, and available evidence, not to change content in an attempt to circumvent controls.

• Define the hypothesis before sending the test.
• Use authorized test numbers and avoid including sensitive data.
• Record the identifier, origin, destination, time, encoding, number of parts, initial result, and later statuses.
• Compare several consistent samples when volume and operational controls permit.
• Contrast the result with aggregate telemetry; do not extrapolate from a single handset.
• If the issue appears limited to a few devices, rule out device-specific causes before attributing it to the route.

Controlled mitigation and protection of critical traffic

Mitigation should reduce risk while maintaining traceability. Do not apply global changes based on a signal limited to one cohort. Prioritize reversible, time-bounded, and documented measures.

Where congestion, increased queueing, or capacity errors occur, rate control, prioritization, and selective pauses can protect the most critical traffic. In OTP or transactional scenarios, separate queues by priority where your architecture allows and apply the limits defined by your integrations and agreements.

Alternative routes should only be used where they are authorized, configured, and appropriate for the destination and traffic class. An alternative must not be used to bypass sender, consent, content, or local regulatory controls.

• Prioritize OTP and critical transactional traffic over less urgent traffic, in line with internal rules.
• Apply rate limits or controlled pauses to the affected cohort if there are signs of saturation or sustained failures.
• Adjust queue management and monitor message age to prevent messages from losing operational value.
• Apply pre-authorized alternative routes only to permitted cohorts and record the change.
• Do not retry already accepted messages indiscriminately: it can increase duplicates, load, and reconciliation complexity.
• Do not treat reported blocking or filtering as a capacity issue that can be resolved by increasing volume.