Cephalocon 2024

Etraveligroup’s AI/ML team has migrated data pipelines from HDFS to Ceph to ensure efficient management and performance. To achieve this, they deployed Ceph RGW with a replicated data pool on NVMe disks to meet the technical demands and automation needs. The request flow involves migrating real time streaming applications and training datasets for machine learning that assist with real time sales analytics and automated pricing. The crucial migration steps include transferring production pipelines from HDFS to in-house object storage. Precise tuning is necessary to manage c-states, power saving, and IOMMU on physical servers and preventing write locking during the dynamic resharding process. Key factors such as failure domain, co-locating RGWs with OSDs or MONs, beast thread pool size, memory profiling, mclock profile, and the number of RGW services are essential for achieving the target metrics and ensuring a successful migration.