All articles

SRE

Database Observability: 5 Critical Golden Signals for SREs

Nov 18, 2026 observability · prometheus · sre · metrics · monitoring

Most DevOps teams monitor database instances through simplistic host-level metrics: overall CPU utilization, free memory, and remaining disk space. While necessary, these infrastructure metrics are trailing indicators. A database can operate at a completely calm 35% CPU while simultaneously locking hundreds of transactions and causing application-wide request failure.

True database site reliability engineering requires tracking internal database runtime state through specialized telemetry signals exported to systems like Prometheus, Grafana, and Percona Monitoring and Management (PMM).

1. Active Executing Threads vs. Connected Clients

In MySQL and MariaDB, Threads_connected simply indicates idle TCP sockets. The mission-critical signal is Threads_running, which reveals how many queries are currently competing for CPU execution cycles. If Threads_running exceeds twice the physical core count, query execution queues are forming. Alerting on Threads_running sustained above this ceiling gives your SRE team a 5-to-10 minute head start before upstream API timeouts begin firing.

2. Row Lock Wait Accumulation and Deadlock Rate

Tracking the cumulative Innodb_row_lock_time and row lock wait count highlights transaction contention that never shows up in slow query logs. A fast 2-millisecond UPDATE statement can hold exclusive row locks for 3 seconds if wrapped inside an uncommitted distributed transaction. Monitoring lock wait accumulation reveals concurrency bottlenecks long before explicit deadlock exceptions emerge in application error trackers.

3. Buffer Pool Churn and Page Eviction Velocity

A sudden decline in buffer pool hit ratio—or a spike in pages read from physical disk per second—is an unmistakable signature of unindexed analytical queries scanning tables into memory. In PostgreSQL, tracking the ratio of shared_blks_hit against shared_blks_read demonstrates whether your working set fits in RAM. Alerting on rapid page churn prevents unbudgeted IOPS spikes and SSD thermal throttling.

Tags

observability prometheus sre metrics monitoring

Work with Burj

Your database deserves engineering, not guesswork.

Tell us what is slow — CPU, queries, migrations, or cost — and we will scope the smallest safe fix.

Plan an engagement