Deciding on a Distributed Filesystem for HPC
by CPT-GrayWolf from LinuxQuestions.org on (#501YT)
I'm currently working on an HPC cluster for arbitrary reasons, but I'm having trouble deciding on what distributed filesystem to use.
There are so many options it's a bit overwhelming (ceph, glusterfs, lustre, beegfs, etc.), and it's hard to find conclusive comparisons between them, especially for HPC environments as most seem to focus on HA.
In my setup, I plan to use the filesystem in parallel with each node acting as both a host (contributing to the storage pool) and client (connecting to the pool). I need whatever I choose to run on Ubuntu's current LTS release, and preferably to support client side caching (RDMA support is also a welcome inclusion).
Thank you in advance for your input.
There are so many options it's a bit overwhelming (ceph, glusterfs, lustre, beegfs, etc.), and it's hard to find conclusive comparisons between them, especially for HPC environments as most seem to focus on HA.
In my setup, I plan to use the filesystem in parallel with each node acting as both a host (contributing to the storage pool) and client (connecting to the pool). I need whatever I choose to run on Ubuntu's current LTS release, and preferably to support client side caching (RDMA support is also a welcome inclusion).
Thank you in advance for your input.