← Back to Prometheus Cardinality Cost

Why Prometheus metrics bills explode, and which labels to drop

Hosted metrics plans bill by active series: every distinct combination of a metric name and its label values. The metric count barely matters; the label values do. A team adds path to a request counter in a hurry, and a quiet 40-series metric becomes 40,000 on the next deploy.

Series multiply, they do not add

A counter with a method label (4 values), a status label (6 values) and a path label (5,000 values) can reach 4 × 6 × 5,000 = 120,000 series per target. Then multiply by the number of pods exposing it. The size of one scrape is the cheap thing to measure, and it is usually enough to find the culprit.

The label names that usually cause it

Compute the saving instead of guessing it

Remove one label from every series of a metric and count the distinct series that remain. The difference is exactly what you save, and it needs no model: the analyzer does this for every label of every metric in the scrape and lists the biggest wins first.

curl -s localhost:9100/metrics | node cli.mjs analyze - --targets 40

Then keep it from coming back

Put a series budget in CI against a scrape from a staging build. The CLI exits 1 when series × targets exceeds the number you set, so a new high-cardinality label fails the pull request instead of the invoice.

node cli.mjs analyze staging-metrics.txt --targets 40 --max-series 50000

What this cannot see

One scrape shows one moment. Labels that churn over days, targets that scale, recording rules, and your vendor's own counting rules (some bill per tag combination or multiply histogram aggregations) are outside it. Dropping a label can also break a dashboard or alert that groups by it. Use the figures to find where to look, and check your plan's rules for the final number.

Common questions

How do I find which metric has the most series?

Paste one target's /metrics output into the analyzer: it lists active series per metric, largest first, and which label drives each. On the Prometheus server itself, a query that counts series by metric name gives the same view across all targets.

Is dropping a label always safe?

No. A dashboard, alert or recording rule may group by that label. The analyzer tells you the series saved, not what depends on the label, so search your rules and dashboards before relabelling.

Why do histograms cost so much?

A classic histogram emits one series per bucket plus a sum and a count for every label combination, so 12 buckets over 100 label combinations is 1,400 series from one metric.