Designing Convergent HPC and Big Data Software Stacks: An Overview of the HiBD Project

DK Panda from Ohio State University gave this talk at the 2019 Stanford HPC Conference. "This talk will provide an overview of challenges in designing convergent HPC and BigData software stacks on modern HPC clusters. An overview of RDMA-based designs for Hadoop (HDFS, MapReduce, RPC and HBase), Spark, Memcached, Swift, and Kafka using native RDMA support for InfiniBand and RoCE will be presented. Enhanced designs for these components to exploit HPC scheduler (SLURM), parallel file systems (Lustre) and NVM-based in-memory technology will also be presented. Benefits of these designs on various cluster configurations using the publicly available RDMA-enabled packages from the OSU HiBD project will be shown."

The post Designing Convergent HPC and Big Data Software Stacks: An Overview of the HiBD Project appeared first on insideHPC.