Cephalocon 2024: Full Schedule

4 - 5 December 2024 | Geneva, Switzerland
View More Details & Registration

The Sched app allows you to build your schedule but is separate from your event registration. You must be registered for Cephalocon 2024 to participate in the sessions. If you have not registered but would like to join us, please go to the event registration page to purchase a registration.

This schedule is automatically displayed in Central European Time. To see the schedule in your preferred timezone, select from the drop-down menu located at the bottom of the menu to the right.

13:30 CET

Bringing a Ceph Based Enterprise Email System Into the Field - Danny Al-Gaaf, Deutsche Telekom AG

Wednesday December 4, 2024 13:30 - 14:05 CET

Globe 1

Deutsche Telekom operates a growing email system with several million accounts and billions of emails stored on traditional NFS. Six years ago we introduced librmb (librados mailbox) to the community, a universal open source library to store emails in a Ceph cluster. Librmb uses RADOS to store email directly in Ceph to achieve maximum performance through parallel access from many email gateways simultaneously, for millions of active customers. Email systems are much too complex to be simulated in a way which would allow to verify if the switch to librmb will work for a large number of users. Therefore a field test with active customers was necessary to provide an educated guess on the behavior of the final setup. This presentation will cover the results from artificial and real field tests with more than 1 million accounts/users. The results include the experience and learnings of migrating from the existing email system into Ceph, an extended time of running the system and from migrating the accounts out of the test system. We will provide an insight into our learnings, found issues, potential solutions and an outlook into our next steps towards a Ceph based email system.

Speakers

Danny Al-Gaaf

Senior Cloud Technologist, Deutsche Telekom AG

Danny Al-Gaaf is a Senior Cloud Technologist working for Deutsche Telekom. As a Ceph upstream developer he is a driver for using Ceph at Deutsche Telekom. For the last 15 years his professional focus has been on Linux and open source. He works actively in several upstream communities... Read More →

Wednesday December 4, 2024 13:30 - 14:05 CET
Globe 1

Session Presentation

Audience Level Intermediate

13:30 CET

Crimson Project Update - Matan Breizman & Aishwarya Mathuria, IBM

Wednesday December 4, 2024 13:30 - 14:05 CET

Auditorium B

The Crimson project is an effort to build a replacement ceph-osd well suited to the new reality of low latency, high throughput, persistent memory and NVMe technologies. Built on the Seastar C++ framework, crimson-osd aims to be able to fully exploit modern devices by minimizing latency, CPU overhead, and cross-core communication. This talk will discuss the current state of Crimson going into the Tentacle release.

Speakers

Aishwarya Mathuria

Senior Software Engineer, IBM

Matan Breizman

Crimson's Tech Lead, IBM

Matan is Crimson's tech lead, which is the next generation Ceph OSD. Part of the Core Ceph development group since 2021.

Wednesday December 4, 2024 13:30 - 14:05 CET
Auditorium B

Session Presentation

Audience Level Intermediate

14:15 CET

Follow the Advice, but Not Too Closely - Building the Cluster YOU Need - Gregory Orange, Pawsey Supercomputing Centre

Wednesday December 4, 2024 14:15 - 14:25 CET

Globe 1

Pawsey receives infrequent blocks of government funding to provide supercomputing, cloud and data services typically at no cost to scientific researchers in Australia. This has a big impact on both the way requirements are determined and on the design and procurement model. Our new 20PB cluster pushes the bounds of what is recommended, for performance and for node density, but we are confident it will serve researchers' needs over the lifespan of the equipment and beyond. This talk will look at the details of this design, and how we determined them, tested them, and increased our confidence of success. Many of these factors are useful in deploying any new cluster, so you will take away some tips on how to get started, or use the experience you already have to build the next one.

Speakers

Gregory Orange

Gregory Orange, Pawsey Supercomputing Centre

Greg deploys and runs Ceph and OpenStack for scientists in Australia, and is involved in the floor tiles, the RGW nodes, the teams who look after it all, and nearly everything in between. Automation and efficiency are of great interest to him, and so is the well-being of all the people... Read More →

Wednesday December 4, 2024 14:15 - 14:25 CET
Globe 1

Lightning Talk

Audience Level Intermediate

14:15 CET

Conditional End2end Tracing - Yuval Lifshitz, IBM & Deepika Upadhyay, Clyso GmBH

Wednesday December 4, 2024 14:15 - 14:50 CET

Auditorium B

End to end tracing can help debug latency issues between the RGW and the OSD, giving a complete picture of request flow, but tracing itself has a performance impact on the system. When your production system runs into issues, the last thing you want is to put it under more pressure! In this session we will demonstrate how to use Lua scripting on the RGW to turn opentelemetry based tracing only for some of the incoming requests. Allowing us to focus on the problem without slowing down the entire system.

Speakers

Yuval Lifshitz

Senior Technical Staff Member, IBM

Yuval Lifshitz works as a Senior Technical Staff Member at IBM. His current focus is enriching the Ceph ecosystem by adding connectivity between the Rados Object Gateway and external message brokers (Kafka, Knative, RabbitMQ, etc.). He also added Lua scripting into the Rados Object... Read More →

Deepika Upadhyay

Ceph Engineer, Clyso GmBH

Deepika is currently working as a Ceph Engineer at Clyso GmBH and is a contributor to Ceph and Rook project, she has worked as an Outreachy intern for Ceph with focus on adding tracing to Ceph OSD. She has also worked as a RADOS and RBD(Block based storage) teams and now working with... Read More →

Wednesday December 4, 2024 14:15 - 14:50 CET
Auditorium B

Session Presentation

Audience Level Intermediate

14:15 CET

The Art of Teuthology - Patrick Donnelly, IBM, Inc.

Wednesday December 4, 2024 14:15 - 14:50 CET

Auditorium C

The Ceph project has used the Teuthology testing framework for much of its history. The custom framework is used to schedule batch jobs that perform e2e testing of Ceph. This is orchestrated using a suite of YAML fragments to alternate test modes, configurations, workloads, and other parameters. Teuthology assembles these fragments into a static matrix with potentially dozens of dimensions ultimately producing a combinatoric explosion of jobs which are evaluated, in practice, as smaller subsets for scheduling. We will explore an alternative directed graph model for constructing jobs from a suite of YAML fragments using path walks. Code adapted to this model has been constructed to produce subsets in linear time and provide Lua scriptable control of YAML fragment generation. The latter new feature empowers us to test Ceph with more rigor and completeness. For example, upgrade suites can be constructed using all possible versions of Ceph that are valid upgrade paths to a target release. We will explore this and other enhancements in depth. The audience can expect to leave with a firm and visual understanding of how QA is performed on Ceph and a vision for the future testing.

Speakers

Patrick Donnelly

Software Architect, IBM, Inc.

Patrick Donnelly is a Software Architect at IBM, Inc. working as part of the global development team on the open source Ceph distributed storage system. Patrick has principally worked on the Ceph file system (CephFS) since 2016. He has been working on Open Source projects for the... Read More →

Wednesday December 4, 2024 14:15 - 14:50 CET
Auditorium C

Session Presentation

Audience Level Intermediate

14:27 CET

Erasure Coding: 5 Ways to Split a Squid - Jamie Pryde, IBM

Wednesday December 4, 2024 14:27 - 14:37 CET

Globe 1

Ceph supports a wide range of Erasure Coding plugins and techniques. This talk will give a brief overview of the different erasure coding plugins supported by Ceph along with some interesting detail about how the various techniques for each plugin works. After giving an overview of the different plugins and techniques, we will discuss our motivations for changing the default plugin from Jerasure to ISA-L, including highlighting some of the more recent optimisations supported by ISA-L. We will also show some preliminary performance results supporting this decision.

Speakers

Jamie Pryde

Storage Software Engineer, IBM

I have been working Ceph performance and erasure coding since the beginning of 2024, after almost 10 years of developing encryption and security solutions for the IBM FlashSystems line.

Wednesday December 4, 2024 14:27 - 14:37 CET
Globe 1

Lightning Talk

Audience Level Intermediate

15:00 CET

Exploring RocksDB in RGW: How We Manage Tombstones - Sungjoon Koh & Ilsoo Byun, LINE Plus Corporation

Wednesday December 4, 2024 15:00 - 15:35 CET

Auditorium C

LINE, a global mobile messenger, has adopted Ceph as its main object storage. It is used to store different kinds of data, such as log files and application data. Thanks to its scalability, billions of objects are stored in our clusters. However, over time, object deletions lead to the accumulation of tombstones in RocksDB, resulting in delays during iteration. Slow iteration not only impacts LIST operation but also stalls subsequent requests. To address this issue, we first collected RocksDB metric called "skip count", which indicates the total number of tombstones detected during iterations. We then deployed a new job which compacts OSDs with high skip counts to prevent stalls. Additionally, we analyzed the pattern of tombstones and found out that a few prefixes account for over 80% tombstones, throughout the entire OSD. Based on this observation, we propose range-based compaction. In this presentation, we will first explain the basics of RocksDB and its role in Ceph Object Storage. Then, we will share our experience how we handled the RocksDB issue. Lastly, we will discuss our proposal for range-based compaction, which could further optimize overall system performance.

Speakers

Ilsoo Byun

Senior Manager, LINE Plus Corporation

Storage engineer at LINE

Sungjoon Koh

Cloud Storage Engineer, LINE Plus Corporation

Sungjoon Koh is a cloud storage engineer at LINE Plus Corporation, focusing on object storage and NVMe-oF-based block storage services. His current interests include enhancing Ceph's compatibility with the S3 standard and developing object migration features. Before joining LINE Plus... Read More →

Wednesday December 4, 2024 15:00 - 15:35 CET
Auditorium C

Session Presentation

Audience Level Intermediate

15:55 CET

Revisiting Ceph's Performance After 4 Years - Wido den Hollander, Your.Online

Wednesday December 4, 2024 15:55 - 16:30 CET

Auditorium C

As new generations of hardware become available and Ceph is improved, how does it's performance change? If we look back 4 years, how did Ceph's performance improve (or not)?

Speakers

Wido den Hollander

CTO, Your.Online

Wido has been a part of the Ceph community for over 10 years. Long time user, developer and advocate of the future of storage. He has worked as Ceph consultant and trainer and is now CTO of Your.Online, a European-based hosting group with companies throughout Europe and a large Ceph... Read More →

Wednesday December 4, 2024 15:55 - 16:30 CET
Auditorium C

Session Presentation

Audience Level Intermediate

17:25 CET

Improving Ceph Economics with QAT Hardware Offload - Philip Williams, Canonical

Wednesday December 4, 2024 17:25 - 18:00 CET

Auditorium B

Ceph, the words most popular open source software defined storage system, has offered storage efficiency features such as block device compression, object compression and server-side object encryption for a number of releases. However, enabling these features has always come as a trade-off between the additional performance required (in terms of cores/GHz) vs the raw storage cost, ultimately driving users away from these features. In this talk we will walk through several different scenarios where Intel's QAT offload is used to enable these features without significant overhead to primary processing, and still yields greater performance without causing increased cost per GB.

Speakers

Philip Williams

Product Manager, Canonical

Philip is a Product Manager at Canonical responsible for Ceph and other storage solutions. He has over 18 years experience in the storage industry, having previously been responsible for storage infrastructure and products at a number of leading technology companies.

Wednesday December 4, 2024 17:25 - 18:00 CET
Auditorium B

Session Presentation

Audience Level Intermediate

10:50 CET

Migrating AI/ML Data Pipelines for Real Time Streaming Applications from HDFS to Ceph - Vahideh Alinouri, Etraveligroup

Thursday December 5, 2024 10:50 - 11:25 CET

Auditorium B

Etraveligroup’s AI/ML team has migrated data pipelines from HDFS to Ceph to ensure efficient management and performance. To achieve this, they deployed Ceph RGW with a replicated data pool on NVMe disks to meet the technical demands and automation needs. The request flow involves migrating real time streaming applications and training datasets for machine learning that assist with real time sales analytics and automated pricing. The crucial migration steps include transferring production pipelines from HDFS to in-house object storage. Precise tuning is necessary to manage c-states, power saving, and IOMMU on physical servers and preventing write locking during the dynamic resharding process. Key factors such as failure domain, co-locating RGWs with OSDs or MONs, beast thread pool size, memory profiling, mclock profile, and the number of RGW services are essential for achieving the target metrics and ensuring a successful migration.

Speakers

Vahideh Alinouri

Cloud engineer, Etraveli Group AB

Vahideh Alinouri is a well-rounded Cloud Engineer with a strong background in cloud computing and academic experience. At EtraveliGroup, a leading company for tech solutions and fulfillment capabilities for online sales of flight, she designed and supported their private cloud infrastructure... Read More →

Thursday December 5, 2024 10:50 - 11:25 CET
Auditorium B

Session Presentation

Audience Level Intermediate

10:50 CET

SWITCH: Operations, Data Management and Automation - Theofilos Mouratidis, SWITCH

Thursday December 5, 2024 10:50 - 11:25 CET

Auditorium C

SWITCH is the national research and education network (NREN) of Switzerland, a non-profit organisation that provides services to the universities and schools of the country. In the storage circle of the cloud team at SWITCH, we maintain and procure Ceph clusters mainly for S3. We have 3 iterations that differ in terms of automation and features namely OSv1/2/3. We currently develop the latest iteration using Ansible in a GitOps way, where the code is the source of truth and changes to the code automatically deploy configuration changes to various clusters. In this session, we will talk about the OSv3 Ansible collection and configuration management repos, where from defining an inventory that looks like the `ceph orch host ls` output and a short yaml file, we can immediately bootstrap clusters that connect together and provide multisite S3, without any manual steps. Now that we deploy our new clusters using the new technologies, we are in the migration phase where we try to maintain the old dying clusters (OSv1/2) and slowly migrate S3 data to the new ones with minimal or no user intervention.

Speakers

Theofilos Mouratidis

Cloud Engineer, SWITCH

My name is Theofilos Mouratidis and I am from Greece. I am currently a cloud engineer at SWITCH. I have a strong theoretical background and research interest in distributed systems. In the past I have worked for CERN and Proton in similar positions. I enjoy the sunny weather and go... Read More →

Thursday December 5, 2024 10:50 - 11:25 CET
Auditorium C

Session Presentation

Audience Level Intermediate

11:35 CET

D4N in RGW - Pritha Srivastava, IBM

Thursday December 5, 2024 11:35 - 12:10 CET

Auditorium B

D4N is an intelligent caching framework within Ceph RGW. This project is a collaboration between the MOC team at Boston University and Ceph RGW team. This talk outlines the brief history of the D4N Research Project and its use cases. Then it will discuss the current ongoing effort to upstream D4N into Ceph which will include its architecture based on the Zipper API in RGW and its use-cases for s3, analytics workload acceleration and use cases in the AI/ML domain.

Speakers

Pritha Srivastava

Software Architect, IBM

I am a Software Developer with IBM and have been working on Ceph RGW for the last 8 and a half years. Prior to that I worked with HP for StoreVirtual. I have completed my MS from University of Colorado at Boulder.

Thursday December 5, 2024 11:35 - 12:10 CET
Auditorium B

Session Presentation

Audience Level Intermediate

13:40 CET

Archive Zone: Lessons Learned - Ismael Puerto Freire & Xabier Guitián Domínguez, INDITEX

Thursday December 5, 2024 13:40 - 14:15 CET

Auditorium C

In this session, we will delve into the history and evolution of our Ceph clusters dedicated to the archive zone in production. We'll cover the entire journey, from the initial hardware selection to the deployment, and share the critical lessons we've learned along the way. Key topics include: Hardware Selection: How we chose the right hardware for our archive zone, including considerations and trade-offs. Common Mistakes: The pitfalls and mistakes we encountered during the deployment process, and how we overcame them. Best Practices: Steps and strategies to ensure a successful deployment, focusing on reliability, scalability, and performance. Optimization Tips: Techniques to optimize your Ceph cluster for archival purposes, ensuring efficient storage and retrieval of data. By the end of this talk, you will have a comprehensive understanding of the challenges and solutions involved in deploying a Ceph archive zone, enabling you to avoid common pitfalls and achieve a successful implementation in your environment.

Speakers

Xabier Guitián Domínguez

Technical Lead of Infrastructure, INDITEX

I am the Technical Lead of Infrastructure at Inditex, overseeing the operation and continuous evolution of the company's services. My role focuses on ensuring reliability, scalability, and innovation in infrastructure solutions to support Inditex's global operations

Ismael Puerto Freire

Solution Architect, INDITEX

I am a Solution Architect at Inditex, responsible for operating and evolving services based on Ceph and Kubernetes. I have been working with Ceph for six years, handling all types of storage: Object, Block, and FileSystem. My top priorities are maintaining resilience, performance... Read More →

Thursday December 5, 2024 13:40 - 14:15 CET
Auditorium C

Session Presentation

Audience Level Intermediate

13:40 CET

Understanding Ceph One Performance Counter at a Time - Marcel Lauhoff, Clyso GmbH

Thursday December 5, 2024 13:40 - 14:15 CET

Auditorium B

Let’s demystify Ceph performance counters by following the journey of data through the system, from the initial client I/O request to the moment it reaches the disk. We'll introduce performance counters, explaining what they are, how they work, and how to access them effectively. Using a simple workload, we'll analyze key Ceph performance counters to illustrate how data flows through various subsystems and what these metrics reveal about system performance. By understanding these metrics, attendees will gain insights into diagnosing issues and optimizing their Ceph deployments.

Speakers

Marcel Lauhoff

Staff Software Engineer, Clyso GmbH

Marcel Lauhoff is a seasoned software engineer with nearly a decade of experience in software-defined storage. His Ceph journey began with his Master's thesis, which focused on adapting Ceph for use in archives. He previously worked on the S3GW project, a standalone cloud S3 service... Read More →

Thursday December 5, 2024 13:40 - 14:15 CET
Auditorium B

Session Presentation

Audience Level Intermediate

14:25 CET

Advancing BlueStore with Real-World Insights - Adam Kupczyk, IBM

Thursday December 5, 2024 14:25 - 15:00 CET

Auditorium C

In past years we have invested significant effort to improve BlueStore's IO latency and throughput. Testing, including aging, have always done using artificial workloads. Obviously we optimized for those scenarios. Now we want to open new chapter in BlueStore maturity. Whenever possible we will use real-life workloads, provided by Ceph users. We will test new components and new proposed settings against those workloads. Testing aging will be augmented by shortcuts that will get the aging process complete faster. The ultimate goal is to preserve high performance new deployments enjoy as long as possible. We want to share this plan with community, get developers involved and convince users to share their workloads.

Speakers

Adam Kupczyk

Engineer, IBM

Mathematician by education. Engineer and programmer by job. Tester by necessity. Graduated Adam Mickiewicz University, Poznan. 25 years in software development.

Thursday December 5, 2024 14:25 - 15:00 CET
Auditorium C

Session Presentation

Audience Level Intermediate

14:25 CET

Ceph Notification at Scale: Notification V2, an Enterprise-Level Feature - Krunal Chheda, Bloomberg

Thursday December 5, 2024 14:25 - 15:00 CET

Auditorium B

Data powers Bloomberg’s financial products. Ceph clusters are the backbone of Bloomberg’s internal S3 cloud storage systems, which host this data and serve billions of requests a day. Given the importance of these cloud storage systems to the company’s broader infrastructure, it is crucial that any S3 feature we enable must be an enterprise-level feature that scales, provides telemetry, is secure, and has fault-tolerant capabilities. Working together with the community, Bloomberg has enhanced the existing notification feature built into Ceph enabling it to be used by Bloomberg and other enterprise users of Ceph. This talk will discuss all the work that went into building “notification_v2,” the new enterprise-class S3 notifications platform and all of its new features.

Speakers

Krunal Chheda

Senior Software Engineer, Bloomberg

Krunal Chheda is a Senior Software Engineer at Bloomberg and is part of the Storage Engineering team in the company's Technology Infrastructure department. He has worked in the industry for more than 12 years and holds a Masters degree in Software Engineering from BITS Pilani India... Read More →

Thursday December 5, 2024 14:25 - 15:00 CET
Auditorium B

Session Presentation

Audience Level Intermediate

15:10 CET

Benchmarking: Repeatable & Comparable - Trent Lloyd, Canonical (Ubuntu)

Thursday December 5, 2024 15:10 - 15:45 CET

Auditorium C

Your goal when benchmarking should be to ensure that the results are both continuously repeatable and fairly comparable to previously attempts. This is all too easy to get wrong. Benchmarking of any kind often presents tricky business, but storage has always presented particularly difficult challenges as even the simple Hard Drive has interesting performance characteristics that vary greatly depending on the workload or even chance. You might hope that was solved by SSDs, and that is true to an extent for real workloads, but they tend to give even more misleading results during synthetic benchmarks. I'll work through many different causes of inconsistent results in benchmarking both individual components and over-all performance of a Ceph cluster, with specific examples and graphs of real attempts. Items covered include - Working set size - Bi-modal SSD performance due to flash block management - Thin provisioning - Bandwidth limitations of SSDs, Backplanes, PCIe buses, CPUs, Memory and Networks - Filesystems - Caches of all kinds - Inconsistencies from benchmarking freshly deployed Ceph clusters - Benchmarking tools (Don't use anything other than fio, aws-cli is slow) - And more

Speakers

Trent Lloyd

Mr, Canonical (Ubuntu)

Trent Lloyd is a long time passionate speaker and member of the Linux & Open Source community, having first presented at the age of 15 to an audience at linux.conf.au 2003. He has spent the last 9 years in the Ubuntu Support Team at Canonical as a Sustaining Software Engineer specialising... Read More →

Thursday December 5, 2024 15:10 - 15:45 CET
Auditorium C

Session Presentation

Audience Level Intermediate

15:10 CET

From ISCSI to NVMe-of: A Comparative Look on Storage Protocol Advantages and Challenges - Andy Muthmann, croit GmbH

Thursday December 5, 2024 15:10 - 15:45 CET

Auditorium B

We’d like to share our experience with NVMe-oF: * What deployment options are available for NVMe-oF, how they work and what advantages/disadvantages they have * Performance of NVMe-oF compared to iSCSI and raw kernel rbd * Issues with iSCSI and how NVMe-oF fits here * NVMe-of Multipathing

Speakers

Andy Muthmann

CEO, croit GmbH

Thursday December 5, 2024 15:10 - 15:45 CET
Auditorium B

Session Presentation

Audience Level Intermediate

16:05 CET

Ceph Made Easy: One Dashboard for Multiple Ceph Clusters - Nizamudeen A, IBM India Private Ltd

Thursday December 5, 2024 16:05 - 16:40 CET

Auditorium C

The presentation is about a solution that we have created in the Ceph Dashboard for managing and monitoring multiple ceph clusters from a single cluster that we call a hub cluster. This approach simplifies the complexities of managing multiple clusters by providing a more streamlined and efficient user experience. I will describe the architecture of our implementation and how this would help admins to manage many clusters, ensuring optimal performance, reliability and ease of use. I will also demo various features which can leverage the multi-cluster setup like setting up replication between multiple clusters. Also with multiple clusters connected on a single cluster, this will also provide an overview dashboard where important information on the other clusters can be monitored including the real-time alerts that goes on in the other clusters. I’ll also share how we are planning to improve the feature and our testing strategies around it.

Speakers

Nizamudeen A

Software Engineer, IBM India Private Ltd

Software Engineer and component lead of Ceph Dashboard. I started 5 years ago as an intern at Red Hat contributing to Rook Operator. Eventually moved into Ceph Dashboard and started looking into the usability improvements and implementing workflows in the UI. Later picked up the lead... Read More →

Thursday December 5, 2024 16:05 - 16:40 CET
Auditorium C

Session Presentation

Audience Level Intermediate

16:05 CET

Get That Cluster Back Online, but Hurry Slowly - Gregory Orange, Pawsey Supercomputing Centre

Thursday December 5, 2024 16:05 - 16:40 CET

Globe 1

Pawsey offers supercomputing, cloud and data services to Australian researchers, and our Petabyte-scale Ceph RGW clusters have become the central storage solution no matter which combination of services is required. Ceph is both complicated and resilient, meaning there are many things which can go wrong. However, with a suitable design and in the right hands the data is protected and the service can be brought back online. I would like to tell you about the months-long situation leading to a significant outage of our largest cluster, problems we encountered, how we discovered and fixed them, who helped along the way, and the lessons we have recorded for the future. These lessons relate to chasing masked errors, technology choices, team resources and community engagement.

Speakers

Gregory Orange

Gregory Orange, Pawsey Supercomputing Centre

Thursday December 5, 2024 16:05 - 16:40 CET
Globe 1

Session Presentation

Audience Level Intermediate

16:50 CET

Cost-Effective, Dense, and Performant Prometheus Storage via QLC - Anthony D'Atri, Dreamsnake Productions

Thursday December 5, 2024 16:50 - 17:25 CET

Auditorium C

Prometheus is the metrics ecosystem of choice for modern computing, with exporters for Ceph, RAID HBAs, Redfish, time synchronization, and the panoply provided by node_exporter. Exporters are scraped multiple times per minute for effective queries, each ingesting as many as several housand metrics per system. Data may be kept locally or in external solutions including Ceph RGW. Retention of a year or more is valuable for trending and comparisons. A moderate-size deployment can easily fill tens or hundreds of terabytes. As retention and cardinality grow, so does processing. Prometheus will GC and flush its WAL every two hours, which can manifest visible yet spurious artifacts in visualization tools like Grafana and false alarms from alertmanager rules. Rotational media just don't cut it. While HDDs of capacities as large as 30TB are available, rotational + seek latencies, SATA stenosis, interminable resilvering, and SMR severely limit their viability. SSDs are increasingly viable as HDD replacements. We can improve cost and density by tailoring to the workload: intermittent sequential writes, and frequent random reads. This is a classic workload for modern QLC SSDs.

Speakers

Anthony D'Atri

Principled Engineer, Dreamsnake Productions

Anthony has run Ceph at scale for over eleven years and one can say that it is literally a part of him - ask him in person and he'll show you why. He is also an observability advocate and contributes daily to Ceph documentation and may be open to new opportunities.

Cost effective, Dense, and Performant Prometheus Storage via QLC Cephalocon 2024 6 pdf

Thursday December 5, 2024 16:50 - 17:25 CET
Auditorium C

Session Presentation

Audience Level Intermediate
Session Slides Attached Yes

17:02 CET

Characterizing Crc Operation on Ceph - Rixin Luo, Huawei Technologies Co., Ltd

Thursday December 5, 2024 17:02 - 17:12 CET

Globe 1

A cyclic redundancy check (CRC) is an error-detecting code widely used in digital networks and storage devices to identify accidental changes in digital data. Ceph employs CRC checksums to ensure data correctness during network transmission and to maintain data integrity in Bluestore. For RBD 4K random write and random read workloads, we observed that the CRC32C operation size primarily ranges from 4 to 292 bytes. Specifically, for 4K random writes, 80% of the operations are smaller than 292 bytes, while for 4K random reads, 83% are smaller than 254 bytes. The most common sizes are 28 bytes and 41 bytes, accounting for nearly 40%; these correspond to 'preamble_block_t' and 'ceph_msg_header2', respectively. Given the characteristics of the CRC32C operation, we optimized it for smaller sizes by using inline assembly code to unfold the CRC32C function 'ceph_crc32c_aarch64'. Additionally, we optimized the encoding procedure to minimize the occurrence of small-sized data segments. We tested these optimizations on the RBD 4K random write and random read workloads, resulting in a 2% improvement in throughput.

Speakers

Rixin Luo

Senior software engineer, Huawei Technologies Co., Ltd

I am a senior software engineer from Huawei. I have worked for years on enabling and optimizing Ceph performance on aarch64.

Thursday December 5, 2024 17:02 - 17:12 CET
Globe 1

Lightning Talk

Audience Level Intermediate

17:15 CET

Async I/O with Libcephfs - Dhairya Parmar, IBM

Thursday December 5, 2024 17:15 - 17:25 CET

Globe 1

File system gateways like NFS-Ganesha are capable of immense throughput but can be hindered by thread-centric synchronicity. The default I/O interfaces built in libcephfs rely on threading this way which makes it difficult to achieve high performance throughput. Enter async I/O in libcephfs – a transformative approach that changes how file system gateways interact with CephFS. By leveraging async I/O, protocols can maximize resource utilization, enhance system responsiveness especially to low latency operations. In this talk, we will explore how NFS-Ganesha led to the development of async I/O in CephFS, detailing its journey from inception through various phases of improvement and stabilization. We will discuss the current state of async I/O in CephFS, including how NFS-Ganesha makes the most out of it, how it boosts performance with CephFS, and future applications of the API on other gateways. Think I/O in CephFS? Think Async I/O!

Speakers

Dhairya Parmar

Software Engineer, IBM

Dhairya is an open-source enthusiast currently part of the CephFS team. He primarily works on the CephFS client, MDS (Metadata Server), and the manager NFS module. Before joining Ceph, he interned at Red Hat, where he worked on converting legacy sudo-SSSD bash scripts into efficient... Read More →

Thursday December 5, 2024 17:15 - 17:25 CET
Globe 1

Lightning Talk

Audience Level Intermediate

17:35 CET

Deploy Ceph in Kubernetes: Rook Deep Dive - Travis Nielsen & Subham Rai, IBM

Thursday December 5, 2024 17:35 - 18:10 CET

Globe 1

Do you run Kubernetes? Rook is the answer to natively integrate Ceph with Kubernetes, with K8s workloads running in production for over six years. Rook configures Ceph to provide stable block (RWO), shared file system (RWX), and object storage (S3). We will give an overview of Rook as well as deep dive into the core configuration features including encryption, recent feature additions, and future plans for the project. Rook was accepted as a graduated project by the Cloud Native Computing Foundation in October 2020.

Speakers

Travis Nielsen

Rook Lead Maintainer, IBM

Travis Nielsen is a Senior Technical Staff Member for IBM where he is a maintainer on Rook and member of the ODF and Ceph engineering team. Prior to IBM and Red Hat, Travis worked in storage at Quantum and Symform, a P2P storage startup, and was an engineering lead for the Windows... Read More →

Subham Rai

Software Engineer, IBM

I'm Subham Rai from India. I have more than 3 years of experience in software or more specifically in the storage industry working mainly in rook-ceph. I hold a B.tech degree and I'm also RHCSA certified. I have spoken at FOSDEM 2022, Cephalocon 2023(Amsterdam). I'm in the top 4... Read More →

Thursday December 5, 2024 17:35 - 18:10 CET
Globe 1

Session Presentation

Audience Level Intermediate