Small Samples in A2P SMS Route Tests: How to Interpret Results Without Overreacting
A test with only a few messages can show what happened, but it may not support a firm conclusion. Learn to distinguish results, uncertainty, and signals that call for more evidence.
Why a low-volume test can give a misleading impression
A test involving only a few messages lets you observe what happened in one specific run. On its own, it is not enough to show that a route will behave the same way at other times, for other destinations, or under different conditions. A favorable result may depend on a limited combination of recipients, senders, content, and sending time; an unfavorable result may also reflect a one-off issue or incomplete measurement.
The first discipline is to separate the data from its interpretation. “Certain delivery statuses were observed in this test” is an observation. “The route is good” or “the route fails” are broader conclusions that require additional evidence and defined evaluation criteria.
- Record what was sent, when it was sent, and under what conditions.
- Do not turn a limited result into a guarantee of future performance.
- Treat an anomaly as a signal to investigate, not automatically as a verdict.
What a sample actually measures
Define the unit you are counting. It could be a message sent, a sending attempt, a technical response received, or an observed delivery status. These are not interchangeable measures. Also define the observation period: a report generated shortly after sending may not include the same statuses as one reviewed later.
Document the total sent and how many cases had an observable result within the agreed window. If the denominator differs between metrics, state it alongside each result. This helps prevent figures that answer different questions from being presented as comparable.
- Record the date, time, time zone, and observation period.
- Keep the total number of messages sent and the number with a recorded result.
- Describe what counts as a result and what is excluded from the analysis.
Segment around a specific question
Aggregated results can conceal differences between groups. When the test design and volume allow, review destination, operator, sender, content, and traffic type separately. The goal is not to split the data endlessly, but to check whether an anomaly is concentrated in a segment relevant to the decision.
Each breakdown reduces the number of observations available in each group. If a category contains very few cases, describe it as a limited observation and avoid comparing it as though it had the same evidential strength as a larger group. Define in advance which variable you are evaluating and keep the others as stable as possible.
- Group by destination and operator when those data are available and reliable.
- Record sender, content, and traffic type to understand the context of each test.
- Avoid attributing a difference to the route if other conditions also changed.
Report observed rates with their denominators
A percentage on its own does not show how much data supports it. Present the numerator and denominator alongside the observed rate—for example, how many messages had the defined result out of the number of messages included in that calculation. Do not use a percentage without explaining how unobserved or pending cases were classified.
With small samples, a handful of cases can change the percentage substantially. Avoid presenting small differences as conclusive evidence, and do not propose a universal sample-size or acceptance threshold. Sufficiency depends on the decision, the test design, and the quality of the available records.
- Show counts and percentages, not percentages alone.
- Explain exclusions, pending cases, and changes in the denominator.
- If you provide an uncertainty estimate, state the method and its assumptions; do not replace them with a figure that merely looks precise.
Missing, delayed, or inconsistent DLRs
A DLR is a status reported by the relevant system; it should not automatically be confused with independent verification that the message reached the recipient’s device. The absence of a DLR does not, by itself, prove that the message was not delivered. Keep a delayed or inconsistent status classified as such until you understand what it represents in the context of the test.
Define how you will handle statuses before comparing routes. Separate received, pending, missing, and unreconciled statuses. Record the time the message was sent and the time the status was received, when available, and keep the original data so you can review the classification.
- Do not automatically classify a missing DLR as a success or failure.
- Distinguish a reported status from verified receipt on the device.
- Document the waiting window and how you handle delayed or duplicate statuses.
Do not combine availability, latency, and delivery
Availability, latency, and delivery statuses answer different questions. An available connection does not show that every message was delivered; a fast response is not the same as confirmed delivery. Combining these dimensions into a single assessment can make it difficult to identify what was measured and what needs attention.
Define each metric separately, including its start and end points and the events it includes. When comparing periods or routes, keep the same definitions and observation windows; if that is not possible, note the difference and limit the conclusion.
- Report availability, latency, and delivery statuses in separate sections.
- Specify which timestamps are used in any latency measurement.
- Do not treat one technical metric as proof of a different dimension.
When to repeat or expand the test
Repeat or expand the test when the result is sensitive to a few cases, many statuses remain unresolved, or an important condition has changed. Before doing so, define the question you want to answer and what evidence would change the decision. Increasing volume without a clear question can add data without making the comparison easier to interpret.
When comparing routes, fix the relevant conditions and change one variable at a time if the goal is to isolate its effect. Record unavoidable differences, such as variations in destination, operator, sender, content, or time period. Do not extrapolate beyond the groups and conditions actually observed.
- Repeat the test if you suspect a one-off issue or incomplete measurement.
- Expand it if a few additional cases could change how the result is interpreted.
- Keep relevant conditions constant or document any changes.
Prudent decision rules
Classify the evidence in operational rather than absolute terms. “Insufficient” means the data do not answer the question asked. “Signal to investigate” describes an observation that merits review but does not, by itself, prove a cause. “Basis for broader evaluation” means the result justifies continuing with a better-documented design; it does not mean the route has been approved for every use.
Before approving, rejecting, or changing a route, weigh the evidence against the test objective, the conditions observed, the quality of the statuses, and the consequences of the decision. If the evidence is not enough, the most rigorous choice may be to leave the decision open and gather more information.
- Insufficient: too few cases are observable, data are pending, or the comparison is not controlled.
- Investigate: a difference or anomaly appears and should be checked in its segment and context.
- Evaluate further: the result is clear enough to justify an expanded test, without promising future outcomes.
Frequently asked questions
Is there a universal number of messages needed for a reliable test?
There is no evidence-based universal threshold for every route and decision. Sufficiency depends on the question, how the test was designed, the observed variability, and how many results can be interpreted.
Does a positive DLR prove that the recipient received the SMS on their phone?
Not necessarily. A DLR is a reported status and is not automatically equivalent to independent verification that the message reached the device.
Should I treat a missing DLR as a failure?
Not automatically. Record it as missing or unresolved according to the test definition, apply an explicit observation window, and avoid classifying it without additional evidence.
Can I compare two routes using only their delivery percentages?
Relying on percentages alone is not prudent. Also compare numerators, denominators, pending or unobserved statuses, time period, segments, and test conditions.
Sources consulted
- 3GPP specifications3GPP
- ITU-T E.164International Telecommunication Union
- GSMA resourcesGSMA
- Sinopsis de pruebas estadísticas no paramétricasUniversidad de Granada