Real-Time Metrics Monitoring

Real-time monitoring of ClickHouse metrics, events, and asynchronous metrics.

---

Diagnostics

Run all queries from the file checks.sql and analyze the results.

---

Ad-Hoc Query Guidelines

Key Tables

• system.metrics - Current gauge values
• system.events - Cumulative counters since restart
• system.asynchronous_metrics - System-level metrics
• system.metric_log - Historical metrics
• system.asynchronous_metric_log - Historical async metrics

Useful Patterns

-- Find metrics by pattern
select * from system.metrics where metric like '%pattern%'
select * from system.asynchronous_metrics where metric like '%pattern%'
select * from system.events where event like '%pattern%'

---

Cross-Module Triggers

Finding	Load Module	Reason
High memory metrics	altinity-expert-clickhouse-memory	Memory analysis
High replica delay	altinity-expert-clickhouse-replication	Replication issues
High parts count	altinity-expert-clickhouse-merges	Merge backlog
High load average	altinity-expert-clickhouse-reporting	Query analysis
High connections	altinity-expert-clickhouse-reporting	Connection analysis

---

Monitoring Recommendations

Key Metrics to Alert On

Metric	Warning	Critical
ReadonlyReplica	-	> 0
Query	> 75% max	> 90% max
MemoryResident	> 80% RAM	> 90% RAM
MaxPartCountForPartition	> parts_to_delay	> parts_to_throw
ReplicasMaxAbsoluteDelay	> 5 min	> 1 hour
LoadAverage1	> CPU count	> 2x CPU count

Prometheus/Grafana Export

ClickHouse exposes metrics at :9363/metrics in Prometheus format when enabled.