Memory Leak Prevention in Safety Monitors

Safety monitors run continuously, processing a stream of agent actions and maintaining statistical models of behavior. Unlike request-response services that start fresh with each request, monitors accumulate state over hours, days, and weeks. This makes them vulnerable to memory leaks that only manifest after extended operation.

Common Leak Patterns

Unbounded event buffers. Monitors that buffer events for batch processing can accumulate unbounded buffers if the processing rate falls behind the ingestion rate. A slow database write or a spike in agent activity can cause the buffer to grow until memory is exhausted.

Fix: Set a maximum buffer size. When the buffer is full, either drop the oldest events (with a counter for dropped events) or apply backpressure to the event source.

Retained references in sliding windows. Statistical monitors like EWMA and CUSUM maintain sliding windows of recent observations. If old observations are not properly removed from the window, memory grows linearly with time.

Fix: Use fixed-size circular buffers for sliding windows. When a new observation arrives and the buffer is full, the oldest observation is overwritten.

Listener accumulation. If the monitor registers event listeners or callbacks without deregistering them, each new subscription adds a reference that prevents garbage collection.

Fix: Use weak references where possible. Implement explicit cleanup in lifecycle hooks. Track the number of active listeners as a metric.

String interning. Monitors that track tool names, agent identifiers, and action types may intern these strings for efficient comparison. If the set of unique values is unbounded (for example, if tool names include dynamic components), the intern table grows indefinitely.

Fix: Use an LRU cache instead of a plain map for interned values. Limit the cache size to a reasonable maximum.

Detection

Monitor the process's resident set size (RSS) over time. A stable process has a flat memory profile after initial warmup. A leaking process shows a steady upward trend.

# Check memory usage
ps -o rss,vsz,pid -p $(pgrep -f sentinel)

Authensor's Sentinel engine is designed with zero runtime dependencies and fixed-size internal data structures to minimize leak risk. If you extend Sentinel with custom monitors, apply these same principles: bound all data structures, use circular buffers for time series, and clean up references explicitly.

Set up alerts on memory consumption thresholds. A monitor that crashes from an out-of-memory error leaves your agents unmonitored.