https://static.xtremeownage.com/blog/2023/proxmox-building-a-ceph-cluster/
Having around 10 total enterprise NVMes, and 10G networking, I am pretty happy with the results.
It runs all of my VMs, kubernetes, etc, and doesn't bottleneck.
This is a great article but it definitely shows that you shouldn’t expect much
He’s not even reaching the IOPS of a single drive in his testing :(
I might have to find something else lol
I did put the disclaimer front and center! Ceph really needs a ton of hardware before it starts even comparing to normal storage solutions.
But, the damn reliability is outstanding.
WIth any kind of advanced software you will never be getting drive speed out of a soulution.
Raw nvmes can hit 800k iops. Add XFS and you may be able to get that.
with MDADM you get like 200-300k
ZFS shrink it to 20-50k
Anything network be glad if you get 5-20k
The more software is involved the worse performance gets especially for IO. Sequentials often scale better or even linearly, but IO is a PITA
Hmm, I guess the most IOPs and latency cut will come from a storage protocol use. I mean, with 10GbE and iSCSI or NFS, you might not feel the benefits of NVMe. Especially in terms of latency. And as far as i know, there is no NVMe-oF support yet.