Cephalocon 2024

CERN IT operates a large-scale storage and computing infrastructure at the service of scientific research and its user community: Ceph provides block, object, and file storage at a scale of 100 PBs, while OpenStack provisions bare-metal nodes, VMs, and virtual networking managing more than 450k CPUs. With the advent of a new computing center a few kilometers away from the main campus, compute and storage resources have been re-imagined to extend the capabilities offered by the infrastructure, while putting upfront clear design choices to favor availability and ease of operations. . In this presentation we report on how the new computing center was designed to host cohesively compute and storage resources, how integration with the existing computing center was achieved, and which new capabilities have been unlocked thanks to the newly-built DC. For Ceph in particular, we share insights on achieving data locality with compute resources, deploying a multi-site object storage service, and running a CephFS service that spans across both data centers.

Discover the current status and our future plans for the Ceph NVMe-oF gateway. This session will cover the latest developments, highlighting new features focusing on security and performance enhancements. Join us to see what’s next for Ceph NVMe-oF!

Maintaining performance and reliability in large Ceph clusters, especially with Rados Gateway (RadosGW), is challenging. Traditional observability approaches often generate excessive data without providing actionable insights. This speech introduces an advanced observability architecture that combines basic monitoring with on-demand detailed and event-triggered monitoring, ensuring continuous visibility and dynamic responsiveness. We'll explore: - The four-layer architecture, detailing producers, the NATS messaging backbone, and consumers. - Monitoring Techniques, including basic, on-demand, and event-triggered monitoring. - A Case Study, sharing proof of concept insights and lessons learned. - Future Directions, discussing potential advancements. This presentation is designed for cloud infrastructure engineers, SREs, and DevOps professionals looking to implement a scalable observability framework that improves system health and performance.

Data powers Bloomberg’s financial products. Ceph clusters are the backbone of Bloomberg’s internal S3 cloud storage systems, which host this data and serve billions of requests a day.During the intensive usage of the Ceph RGW object storage with multi-site settings, we encountered different types of data inconsistencies, such as bucket-index and RADOS object inconsistency, unfinished transactions, and multi-site replication inconsistency, etc. These inconsistencies may potentially be caused by software bugs, race conditions, system timeout, and other reasons. Since we cannot guarantee the system is always bug-free and operating smoothly, it’s crucial that we can identify the inconsistency – should it happen – and fix or report it. While there are existing tools and code in place to help address some of these issues, their usage has limitations. As such, we are proposing a scalable and extensible bucket scrubbing approach to systematically check and identify and fix any inconsistency in the RGW object storage system at the bucket-level, if possible. This talk will discuss the design of this bucket scrubbing system and a prototype of it that we are implementing at Bloomberg.

This talk explores why Ceph is the best software-defined storage solution available, highlighting its evolution since 2004 and its leadership over today's alternatives. Ceph uses innovative technologies to stay relevant. It offers block, object, and file storage through a unified system, reducing complexity and management overhead. CRUSH and Placement Groups provide scalability and resilience, allowing Ceph clusters to span hardware generations without disruptive migrations. BlueStore enhances performance with flexible replication and erasure coding. Stretch clusters and mirroring enable robust disaster recovery. Scale-out metadata in RGW and CephFS support performance for AI workloads. Ceph remains trendy with top integrations for on-premises cloud platforms, thanks to its pluggable architecture and community contributions. It's free, open-source, vendor-free, and easily installable with orchestration tools. With competition closing in, Ceph's community must innovate to stay ahead. We'll offer insights from today's toughest storage requirements, suggesting technical evolutions for the OSD, RGW, and MDS to keep Ceph at the forefront of software-defined storage solutions.

Drawing from the insights gained from attending Ceph Days 2022, Customers previous talks, and my personal experience with ServiceMesh deployments, I have observed that many organizations need help implementing a cluster-wide rate limiting and metrics visibility across their RGW and Buckets. We discover a range of exciting use cases as we explore Envoy's capabilities in rate limiting, filtering, and collecting metrics on S3 activities executed against RGW. These include header-based filtering for multi-region deployments, additional OAuth token enforcement, and the ability to monitor user-to-bucket metrics without post-processing command outputs at ease. To illustrate these capabilities, I will lead a hands-on lab demonstration, showcasing Envoy's frontend role for RGW on various use cases. Rate limiting S3 requests per client Rate limiting S3 requests per region Rate limiting S3 requests per user/address/bucket (more granular level of control) Utilize Prometheus for metrics collection and monitoring Examine the use of geo-regional traffic flow scenarios in RGW Examine the use of traffic stream replication for disaster recovery scenarios in RGW

Join us for an interactive Birds of a Feather session where Ceph users come together with developers to share their experiences. This is your chance to provide direct feedback on what’s working well and what’s not in your Ceph deployments. Whether you’ve had successes or faced challenges, your insights are crucial. Together, we’ll discuss concrete technical ideas for improvements and collaborate on shaping the future of Ceph. Bring your experiences, pain points, and suggestions to help make Ceph even better!

A Ceph cluster stretched across 3 zones faces a potential scenario where data loss can occur due to unforeseeable circumstances. An example of such a scenario is when we have 6 replicas spread across 3 datacenters with a min_size of 3 and the setup is intended to prevent I/O from happening when there is only 1 datacenter available, however, there is an edge case where a placement group (PG) becomes available due to a lack of safeguarding during the process of temporary PG mappings in order ensure data availability. This scenario poses a risk when the sole surviving data center accepts writes, and then suddenly the 2 unavailable data centers come back up. At the same time, the surviving data center suddenly goes down, which means we would have a data loss situation. To prevent such a scenario from happening, we created a solution that utilizes an existing feature in stretch mode that would restrict how we choose the OSDs that would go into the acting set of a PG. This talk will take a deep dive into how this feature is implemented in the latest Ceph upstream as well as other features that improve the user experience with stretch cluster in the latest Ceph upstream release.

Whether you are a Ceph user or a developer, you have probably wondered at some point: How many Ceph clusters are out there? What Ceph versions are they running? What does their storage capacity distribution look like? Answers to these questions and more are available thanks to Ceph’s telemetry module. In this session, we will deep dive into this module and explore the value it brings to users and developers alike.

This talk will cover our experiences managing and scaling a Ceph cluster (pacific) from 4.5PB to 9PB) (raw) while in production, primarliy using CephFS and RGW. We will go through the cluster design and hardware, operational challenges, bugs hit, decisions made and how we resolved them.

This talk focusses on the current (and near future) state of the three pieces that make up a Ceph File System - Ceph Metadata Sever (MDS), Clients and a set of Ceph Manager Plugins. Much advancements have been made to the Ceph File System recently, opening up gateways for wider adoption. Some features are already available in recent releases and some are under development. We detail these enhancements by breaking up nicely into each of the three pieces. Ceph File System specific manager plugins have come a long way to now becoming the de-facto for subvolume/crash-consistent snapshot management and mirroring. We discuss about those. And finally, we peek into what is upcoming in CephFS for Tentacle ("T") release. Existing and new CephFS users would find it helpful to assess and plan ahead for its adoption.

The ARM ecosystem is becoming increasingly robust, and ensuring the stability and high performance of Ceph on the ARM platform is crucial. Due to some reasons, the ARM CI has been almost abandoned in recent years. Over the past year, Rongqi's team has been actively fixing environment and software issues to maintain Ceph's operation on ARM. The team also continuously monitors Ceph's performance on ARM. In this Lightning Talk, he will showcase the achievements on the ARM CI up to now. Additionally, in the performance section, he will discuss how Ceph's integration with UADK (User Space Accelerator Development Kit, a general-purpose user space accelerator framework that uses shared virtual address (SVA) technology to provide a unified programming interface for hardware acceleration computing cryptography and compression algorithms) enables compression under low CPU load. The T version will include this by default.

The Ceph Ambassador program recognizes individuals in the community who are participating and helping with community activities and engaging with other community members. They are an amazing group of Ceph experts and community leaders in various regions across the world who are available to support the Ceph community within their region. This talk will cover an Overview of the Ceph Ambassador program and its goals. I’ll be sharing about the various events and initiatives involving the ambassadors to inspire and drive future activities, community growth, engagement, and development collecting feedback from the community. It has been a nice year in the Ceph community with Ceph Days being successfully organized across the world. The Ceph Ambassadors look forward to inspiring the community and contributing towards the Ceph project's success.

We have been working to improve erasure coding performance, such that it performs almost as well as replica 3 configurations for all configurations, including block and file. Our aim here is to expose the wider Ceph community to the changes we're making to erasure coding and pave the way for wider adoption as we increase visibility of performance improvements over time. In this talk we will share the latest news on erasure coding improvements, highlight both merged and in progress PRs that are of interest, and include performance benchmarks on the latest experimental code.

This Birds of a Feather session aims to explore best practices in the backup of CephFS deployments, encourage technical exchange and highlight where the system can evolve to improve support for this crucial use-case. We encourage participants to share their experience in this area. Discussion points - Applications: i) Which backup applications have been tried? ii) Tricks to avoid iterating on every inode, e.g., recursive mtime. - Performance and configuration: Handling/controlling impact of backups on the live system. - Snapshots: i) Impact of snapshots on the live system; ii) Snapdiff efficiency; iii) Ability to export snapshots as blobs to arbitraty backends, e.g., tape; iv) Experiences with snapshot mirroring. - Integrations: OpenStack Manila.