Two conversations happen in most engineering orgs, and they never happen in the same room.
In one, the on-call engineer explains that they get paged eleven times a night and nine of them are nothing.
In the other, finance asks why the observability vendor invoice grew faster than the infrastructure it observes.
These are the same problem. The cost of noise is paid twice — once in ingestion and retention, once in the incident your team was trained not to notice.
The first bill: data nobody reads
Observability pricing is usually a function of volume: custom metrics, log ingestion, indexed spans, host count. All of these grow by default, because the default behaviour of every instrumentation library is to emit generously and let you filter later.
Later never arrives.
Three specific patterns account for most of it:
Unbounded metric cardinality. A metric tagged with something high-cardinality — a user ID, a request ID, a full URL path — does not create one time series. It creates one per distinct value. A single tag on a hot metric can multiply your custom metric count by four orders of magnitude, and most vendors bill per time series.
The tell is a custom-metrics bill that grew sharply with no corresponding traffic growth. Find it by sorting metrics by series count, not by name.
Debug logging that shipped to production. A log line added during an investigation, useful for a week, never removed. It costs per gigabyte ingested, per gigabyte stored, and per query that has to scan past it.
Retention set once, at the maximum. Someone chose the longest retention available during onboarding, because it was the safe answer and there was no data to argue otherwise. Nobody has revisited it since.
Ask the question directly: when was this data last queried? Most platforms can tell you which log indexes and which metrics are actually read. The answer is routinely uncomfortable — a large fraction of what you ingest has never been queried by a human even once.
That is not observability. That is storage with a subscription.
The second bill: the incident you miss
The expensive part is not the invoice.
An alert that fires and requires no action teaches a specific lesson: alerts from this system can be ignored. That lesson is learned quickly, applied broadly, and does not distinguish between the noisy alert and the one that matters.
You can measure this. For a month, tag every page with whether it led to an action — a fix, a rollback, an escalation, anything other than acknowledge-and-return-to-sleep.
That ratio is your signal-to-noise. When fewer than one page in four leads to action, your on-call rotation has — correctly and rationally — stopped reading pages carefully. They are pattern-matching on the alert name and going back to sleep.
At that point your alerting system has a coverage number that looks excellent and a real-world effectiveness close to zero. Worse, the number looks fine on a dashboard, so nobody knows.
Alert coverage measures what you monitor. It says nothing about whether anyone is still listening.
The question that cuts volume
For every alert that currently pages a human, ask:
What would the responder do, at 3 a.m., in the first five minutes?
If there is a specific answer — roll back the deploy, fail over the database, scale the pool — it is a page. Keep it, and put that answer in the alert body as a runbook link.
If the answer is look at it and probably nothing, it is not a page. It is a dashboard, or a ticket, or a weekly digest.
If the answer is it depends, the alert is too broad. Split it into the case that has an action and the case that does not.
Applied honestly, this typically removes more than half of a mature alerting setup on the first pass. Not because the underlying conditions stopped mattering — because most of them never warranted waking a person.
Alert on symptoms, not causes
The most reliable structural fix is to page on what your users experience, not on what your infrastructure is doing.
High CPU on a node is a cause. It might mean an incident, or it might mean a batch job is doing exactly what it was built to do. Elevated error rate on a customer-facing endpoint is a symptom, and it is unambiguously worth waking someone for.
Cause-based alerts are attractive because they are easy to write and they feel proactive. They are also the primary source of pages that resolve themselves before the responder finishes logging in.
Keep the cause metrics. Graph them. Use them during diagnosis. Just do not page on them.
Give every alert an owner and an expiry
An alert without a named owner is an alert that will never be deleted, because deleting someone else's alert feels like removing a safety net.
Two rules fix this:
- Every alert has an owning team recorded in its definition.
- Every alert is reviewed on a schedule — quarterly is enough — against a simple record: how many times did it fire, and how many times did that lead to action?
An alert that fired forty times and produced zero actions is not a safety net. It is the thing preventing the safety net from working.
Where the two bills meet
Cutting noise is the same work as cutting cost, done from the other end.
Drop the high-cardinality tag and your custom metric count falls, and the dashboard built on it becomes legible. Remove the debug log line and ingestion falls, and searches over that index get faster. Delete the alert that never led to action and your on-call starts reading alerts again.
None of these are trade-offs between spend and coverage. They are the same fix, and the reason they do not happen is not disagreement — it is that no single person owns both numbers. The observability bill sits with platform or finance. Alert quality sits with whoever is on call this week. Nobody is looking at the ratio that connects them.
Start there. Put the ingestion bill and the action-rate ratio on the same page, in front of the same people, once a quarter. Most of the decisions make themselves after that.
DevLift's observability agent, Iris, manages Datadog, CloudWatch and Grafana from one interface — surfacing unqueried indexes, runaway metric cardinality, and alerts that have never led to an action, alongside what each is costing. Book a walkthrough.