Cephalocon 2024

Prometheus is the metrics ecosystem of choice for modern computing, with exporters for Ceph, RAID HBAs, Redfish, time synchronization, and the panoply provided by node_exporter. Exporters are scraped multiple times per minute for effective queries, each ingesting as many as several housand metrics per system. Data may be kept locally or in external solutions including Ceph RGW. Retention of a year or more is valuable for trending and comparisons. A moderate-size deployment can easily fill tens or hundreds of terabytes. As retention and cardinality grow, so does processing. Prometheus will GC and flush its WAL every two hours, which can manifest visible yet spurious artifacts in visualization tools like Grafana and false alarms from alertmanager rules. Rotational media just don't cut it. While HDDs of capacities as large as 30TB are available, rotational + seek latencies, SATA stenosis, interminable resilvering, and SMR severely limit their viability. SSDs are increasingly viable as HDD replacements. We can improve cost and density by tailoring to the workload: intermittent sequential writes, and frequent random reads. This is a classic workload for modern QLC SSDs.