Cephalocon 2024

Your goal when benchmarking should be to ensure that the results are both continuously repeatable and fairly comparable to previously attempts. This is all too easy to get wrong. Benchmarking of any kind often presents tricky business, but storage has always presented particularly difficult challenges as even the simple Hard Drive has interesting performance characteristics that vary greatly depending on the workload or even chance. You might hope that was solved by SSDs, and that is true to an extent for real workloads, but they tend to give even more misleading results during synthetic benchmarks. I'll work through many different causes of inconsistent results in benchmarking both individual components and over-all performance of a Ceph cluster, with specific examples and graphs of real attempts. Items covered include - Working set size - Bi-modal SSD performance due to flash block management - Thin provisioning - Bandwidth limitations of SSDs, Backplanes, PCIe buses, CPUs, Memory and Networks - Filesystems - Caches of all kinds - Inconsistencies from benchmarking freshly deployed Ceph clusters - Benchmarking tools (Don't use anything other than fio, aws-cli is slow) - And more