Interactive Supercomputing with Jupyter and DASK

Anderson Banihirwe from NCAR gave this talk at SciPy 2019. "This talk demonstrates how to use Dask and Jupyter on large high-performance computing (HPC) systems to scale and accelerate large interactive data analysis tasks -- effectively turning HPC systems into interactive big-data platforms. We will introduce dask-jobqueue which allows users to seamlessly deploy and scale dask on HPC clusters that use a variety of job queuing systems such as PBS, Slurm, SGE, and LSF. We will also introduce dask-mpi, a Python package that makes deploying Dask easy from within a distributed MPI environment."

The post Interactive Supercomputing with Jupyter and DASK appeared first on insideHPC.