Prometheus Metrics for AI Safety

Prometheus metrics give you quantitative insight into your AI safety system's behavior. Proper instrumentation answers critical questions: Is the system keeping up with traffic? Are we blocking more requests than expected? Is latency within budget?

Essential Metrics

Counter: authensor_evaluations_total with labels for decision (allow, deny, escalate) and policy version. This is your primary throughput metric. The deny rate (denials divided by total) is the single most important safety metric to track.

Histogram: authensor_evaluation_duration_seconds measures policy evaluation latency. Use buckets at 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, and 250ms. Alert when p99 exceeds your latency budget.

Counter: authensor_aegis_detections_total with labels for detection type (regex, classifier, embedding) and category (injection, harmful content, PII). Tracks what your content scanner is catching and how.

Counter: authensor_sentinel_anomalies_total with labels for anomaly type and severity. Tracks behavioral anomaly detections.

Gauge: authensor_active_policies shows the number of currently active policies. Unexpected drops indicate configuration issues.

Histogram: authensor_receipt_write_duration_seconds measures audit trail write latency. Spikes indicate database pressure.

Recording Rules

Create recording rules for frequently queried aggregations:

The 5-minute deny rate per policy version helps identify problematic policy updates. The evaluation throughput per agent group helps with capacity planning. The content scanning hit rate per category helps tune scanner sensitivity.

Alert Rules

High deny rate: Alert when the deny rate exceeds the 7-day rolling average by more than two standard deviations. This catches both attacks and policy misconfigurations.

Latency degradation: Alert when p99 evaluation latency exceeds 50 milliseconds for 5 consecutive minutes.

Error rate: Alert when the evaluation error rate exceeds 0.1%. Errors in safety checks require immediate investigation.

Missing metrics: Alert when no evaluations are recorded for 3 minutes. This indicates either a complete outage or a bypass.

Capacity Planning

Use predict_linear() on evaluation throughput to forecast when you need to scale. Track the ratio of evaluation time to total request time to understand whether safety checks are becoming a larger portion of your latency budget.

Export these metrics to your existing Prometheus stack. Authensor's control plane exposes them on a configurable /metrics endpoint in the standard Prometheus exposition format.