LLM spend can increase by 50x in minutes — a prompt injection, a runaway retry loop, or a misconfigured context window. Here is how to detect it before the invoice arrives.
Cloud infrastructure costs move on daily or hourly cycles and rarely spike by more than 2-3x without a deployment event. LLM inference costs can spike by two orders of magnitude in minutes. A retry loop that sends a 32k-token context on every attempt, a prompt injection that triggers long completions, or a batch job that accidentally runs in production — all of these can generate thousands of dollars of spend before anyone notices.
The detection problem is harder than it looks. LLM spend has natural volatility: business hours vs. off-hours, weekday vs. weekend, feature launches, and seasonal patterns all create legitimate baseline shifts. A naive threshold alert (spend > $X/hour) pages too often on legitimate spikes and misses slow-burn anomalies that stay under the threshold but run for days.
Statistical approach: rolling z-score with minimum floor
The approach that balances sensitivity and noise: compare current window spend to a rolling historical baseline using a z-score, but gate on a minimum spend floor to suppress noise at low-volume windows. The z-score measures how many standard deviations the current period is from the recent mean. The floor prevents a $0.02 → $0.08 swing from generating an alert when absolute spend is trivial.
Implementation: compute mean and standard deviation of daily spend over the trailing 14 days for each scope (project + environment + model is the most granular; project + environment is often sufficient). Calculate z = (current - mean) / stddev. Alert if z > threshold AND current spend > floor. Typical starting values: z-threshold of 2.5 (catches ~1% of normal days), minimum spend floor of $5/day. Tune down the floor for high-value production workloads; tune up the z-threshold for volatile experimental environments.
Comparison of alert strategies: static threshold — low setup cost, high false positive rate, misses slow-burn anomalies. Percentage change from prior period — catches sudden spikes but not gradual drift, misses weekly periodicity. Rolling z-score with floor (recommended) — catches both sudden spikes and gradual drift, minimum configuration, tunable per scope. ML-based seasonal decomposition — highest accuracy, significant setup and maintenance cost, justified at $50k+/month spend.
Anomaly detection strategy comparison
| Approach | Strength | Limitation | Recommended use |
|---|---|---|---|
| Static threshold | Fast to launch | High false positives | Early-stage visibility |
| Period-over-period change | Catches sharp jumps | Misses slow drift | Known traffic cycles |
| Rolling z-score + floor | Balanced signal/noise | Needs baseline data | Default production control |
| Seasonal ML decomposition | Highest precision | Heavy setup/maintenance | Large mature programs |
Scoping alerts correctly
Org-level alerts are trailing indicators. By the time total org spend is anomalous, the root cause has been running for hours. The right scoping hierarchy: alert at the project + environment level first, then aggregate to org level for executive reporting. A spike in 'customer-support / prod' is actionable — an engineer can route, throttle, or roll back. A spike in 'org total' is a report.
Model-specific alerts catch a distinct class of problems: model routing drift. If your application routes requests dynamically (by quality score, latency, or cost), a bug in routing logic can silently send all traffic to your most expensive model. An alert on spend for Claude Opus 4 specifically, scoped to an environment that should be using a cheaper model, catches routing failures that aggregate spend alerts miss entirely.
Feature-level anomaly detection is the highest-signal tier for teams with feature attribution in place. A feature that normally costs $50/day spiking to $800/day is almost certainly a code path issue — an infinite loop, a missing early-exit condition, or a context window that unexpectedly grew. Feature-level scope eliminates the need to diagnose which part of the system is at fault after the alert fires.
Response runbooks
Detection without response is just noise. Every anomaly alert should link to a runbook with three pre-decided actions: how to throttle or disable the affected feature without a deployment (feature flags are the right mechanism here), who to page, and what business impact looks like at $X/hour run rate. Teams that define runbooks before an incident respond in minutes; teams that figure it out during the incident respond in hours.
For ingestion pipelines, add a circuit breaker at the SDK or middleware layer: if usage events from a feature exceed N events in M seconds, pause ingestion and surface a warning. This does not stop LLM calls — it stops cost accumulation from being invisible. The application still works; the cost anomaly becomes visible before it grows.
Dry-run mode during alert rule tuning is essential. Before enabling a new alert rule in production, evaluate it against the last 30 days of historical spend to see how many times it would have fired and on which days. Most first-pass thresholds fire too often or not at all. One tuning iteration against historical data is worth several weeks of production noise.
Notification and escalation design
Alerts should go to the team that owns the code, not to a central FinOps inbox. A Slack channel per project or per environment means the alert lands with the person who can act on it. FinOps and finance get a daily or weekly rollup, not real-time pings — real-time paging on cost anomalies for non-engineering teams creates panic without actionability.
Deduplicate notifications per rule per time window. If an anomaly persists for six hours, send one alert at detection and one escalation at the 2-hour mark if not resolved. Sending an alert every 15 minutes for a persistent anomaly trains teams to ignore the channel.
CostLynx evaluates alert rules on a rolling basis with configurable z-thresholds, minimum spend floors, and notification deduplication per rule and time window. Rules scope to org, project, or environment and notify via Slack webhook. The dry-run evaluation endpoint lets you test rule sensitivity against real historical data before enabling live notifications.