Cephalocon 2024

CERN IT operates a large-scale storage and computing infrastructure at the service of scientific research and its user community: Ceph provides block, object, and file storage at a scale of 100 PBs, while OpenStack provisions bare-metal nodes, VMs, and virtual networking managing more than 450k CPUs. With the advent of a new computing center a few kilometers away from the main campus, compute and storage resources have been re-imagined to extend the capabilities offered by the infrastructure, while putting upfront clear design choices to favor availability and ease of operations. . In this presentation we report on how the new computing center was designed to host cohesively compute and storage resources, how integration with the existing computing center was achieved, and which new capabilities have been unlocked thanks to the newly-built DC. For Ceph in particular, we share insights on achieving data locality with compute resources, deploying a multi-site object storage service, and running a CephFS service that spans across both data centers.

Discover the current status and our future plans for the Ceph NVMe-oF gateway. This session will cover the latest developments, highlighting new features focusing on security and performance enhancements. Join us to see what’s next for Ceph NVMe-oF!

Maintaining performance and reliability in large Ceph clusters, especially with Rados Gateway (RadosGW), is challenging. Traditional observability approaches often generate excessive data without providing actionable insights. This speech introduces an advanced observability architecture that combines basic monitoring with on-demand detailed and event-triggered monitoring, ensuring continuous visibility and dynamic responsiveness. We'll explore: - The four-layer architecture, detailing producers, the NATS messaging backbone, and consumers. - Monitoring Techniques, including basic, on-demand, and event-triggered monitoring. - A Case Study, sharing proof of concept insights and lessons learned. - Future Directions, discussing potential advancements. This presentation is designed for cloud infrastructure engineers, SREs, and DevOps professionals looking to implement a scalable observability framework that improves system health and performance.

Data powers Bloomberg’s financial products. Ceph clusters are the backbone of Bloomberg’s internal S3 cloud storage systems, which host this data and serve billions of requests a day.During the intensive usage of the Ceph RGW object storage with multi-site settings, we encountered different types of data inconsistencies, such as bucket-index and RADOS object inconsistency, unfinished transactions, and multi-site replication inconsistency, etc. These inconsistencies may potentially be caused by software bugs, race conditions, system timeout, and other reasons. Since we cannot guarantee the system is always bug-free and operating smoothly, it’s crucial that we can identify the inconsistency – should it happen – and fix or report it. While there are existing tools and code in place to help address some of these issues, their usage has limitations. As such, we are proposing a scalable and extensible bucket scrubbing approach to systematically check and identify and fix any inconsistency in the RGW object storage system at the bucket-level, if possible. This talk will discuss the design of this bucket scrubbing system and a prototype of it that we are implementing at Bloomberg.

This talk explores why Ceph is the best software-defined storage solution available, highlighting its evolution since 2004 and its leadership over today's alternatives. Ceph uses innovative technologies to stay relevant. It offers block, object, and file storage through a unified system, reducing complexity and management overhead. CRUSH and Placement Groups provide scalability and resilience, allowing Ceph clusters to span hardware generations without disruptive migrations. BlueStore enhances performance with flexible replication and erasure coding. Stretch clusters and mirroring enable robust disaster recovery. Scale-out metadata in RGW and CephFS support performance for AI workloads. Ceph remains trendy with top integrations for on-premises cloud platforms, thanks to its pluggable architecture and community contributions. It's free, open-source, vendor-free, and easily installable with orchestration tools. With competition closing in, Ceph's community must innovate to stay ahead. We'll offer insights from today's toughest storage requirements, suggesting technical evolutions for the OSD, RGW, and MDS to keep Ceph at the forefront of software-defined storage solutions.

Drawing from the insights gained from attending Ceph Days 2022, Customers previous talks, and my personal experience with ServiceMesh deployments, I have observed that many organizations need help implementing a cluster-wide rate limiting and metrics visibility across their RGW and Buckets. We discover a range of exciting use cases as we explore Envoy's capabilities in rate limiting, filtering, and collecting metrics on S3 activities executed against RGW. These include header-based filtering for multi-region deployments, additional OAuth token enforcement, and the ability to monitor user-to-bucket metrics without post-processing command outputs at ease. To illustrate these capabilities, I will lead a hands-on lab demonstration, showcasing Envoy's frontend role for RGW on various use cases. Rate limiting S3 requests per client Rate limiting S3 requests per region Rate limiting S3 requests per user/address/bucket (more granular level of control) Utilize Prometheus for metrics collection and monitoring Examine the use of geo-regional traffic flow scenarios in RGW Examine the use of traffic stream replication for disaster recovery scenarios in RGW

Join us for an interactive Birds of a Feather session where Ceph users come together with developers to share their experiences. This is your chance to provide direct feedback on what’s working well and what’s not in your Ceph deployments. Whether you’ve had successes or faced challenges, your insights are crucial. Together, we’ll discuss concrete technical ideas for improvements and collaborate on shaping the future of Ceph. Bring your experiences, pain points, and suggestions to help make Ceph even better!

A Ceph cluster stretched across 3 zones faces a potential scenario where data loss can occur due to unforeseeable circumstances. An example of such a scenario is when we have 6 replicas spread across 3 datacenters with a min_size of 3 and the setup is intended to prevent I/O from happening when there is only 1 datacenter available, however, there is an edge case where a placement group (PG) becomes available due to a lack of safeguarding during the process of temporary PG mappings in order ensure data availability. This scenario poses a risk when the sole surviving data center accepts writes, and then suddenly the 2 unavailable data centers come back up. At the same time, the surviving data center suddenly goes down, which means we would have a data loss situation. To prevent such a scenario from happening, we created a solution that utilizes an existing feature in stretch mode that would restrict how we choose the OSDs that would go into the acting set of a PG. This talk will take a deep dive into how this feature is implemented in the latest Ceph upstream as well as other features that improve the user experience with stretch cluster in the latest Ceph upstream release.