Article 79TWR Python parallel debugging comes to HPC thanks to TotalView upgrade

Python parallel debugging comes to HPC thanks to TotalView upgrade

by
Joab Jackson
from www.theregister.com - Articles on (#79TWR)
Story Image

The OG debugger for supercomputer iron has buddied up with the data-science darling now muscling in on HPC: Python.

At the SC26 supercomputing show this week (booth #2829), Perforce Software is demonstrating how its TotalView parallel debugging tool now understands Python, easing the path for using the language on large, distributed apps, particularly those in the high-performance computing (HPC) space.

In the HPC community, "Python has grown in popularity," said Bill Burns, Perforce vice president of software engineering, in an interview with The Register.

TotalView is a debugger specializing in picking apart parallel applications for developers.

Built in the late 1980s for military applications, TotalView today largely serves developers in high-performance computing (HPC), where large scientific applications are spread across multiple servers, running in parallel on thousands or even tens of thousands of processes. (Perforce acquired TotalView in 2019 in its purchase of the DevOps-focused Rogue Wave Software.)

To date, TotalView has supported C/C++ and Fortran. With the latest quarterly release, the tool now supports Python as well.

Debugging a thousand cores

C/C++ has long had a decent dedicated debugger in GDB, but figuring out what went wrong across multiple threads is a special kind of torture. Each thread needs to be instrumented, and all the output needs to be aggregated and filtered to just highlight the interesting stuff. Regular debuggers get overwhelmed with all this info.

Even if you have 10,000 things going on, if you have one crash in there, TotalView can pinpoint that," Burns said. It can look at the state of the process, the threads within that process or the GPU that process is using.

In the HPC space, computing work has traditionally been written in C/C++. And at least some HPC iron still runs Fortran.

Thanks to it being easy to learn and use, Python has gained a foothold in this community as well. The mpi4py library bridges a Python program to the Message Passing Interface (MPI), allowing different parts of an application running on different hardware to communicate.

Python became more amenable to parallel programming at the thread level with version 3.14, released a year ago, which included the ability to use concurrent interpreters, as well as the long-awaited free threading support, thanks to the disabling of the global interpreter lock (GIL).

If you have a CPU with 64 cores, you can run 64 independent threads all working in unison on that CPU. Without GIL, those threads will run fully asynchronously," Burns said.

Multinode scientific simulations, distributed AI training, and large applications that would benefit from shared memory have all become fair game for Python.

Of course, with 500 or 1,000 interpreters all running in parallel, debugging will also get more difficult.

Now you're going to end up with lock and thread contention and race conditions and all the things that C++ and Fortran developers have dealt with for years," Burns said.

A view into parallel Python

Under the hood, TotalView uses Microsoft's debugpy engine (via the Debug Adapter Protocol) to power its own hierarchical tree server. TotalView can aggregate hundreds of stack traces into a single view, highlighting only those outside of expected bounds.

To kick off a debugging session, a developer can launch a single-process or parallel program under TotalView's control. They can then set breakpoints to pause execution and inspect variables and data structures, as well as evaluate Python expressions.

The package doesn't offer full cross-language debugging, which would be truly helpful for Python apps that call C/C++ libraries and modules. For now, developers have to choose between debugging the C/C++ side (while viewing the Python stack) or debugging Python on its own - though Burns said Perforce plans to merge both modes into a single workflow in an upcoming release.

And though TotalView supports mpi4py and Python's own multiprocessing module out of the box, future releases will support other Python concurrency frameworks, such as Dask and PyTorch's torchrun.

TotalView is priced on a token-based model, where each token buys the capability to debug a process. (R)

External Content
Source RSS or Atom Feed
Feed Location http://www.theregister.co.uk/headlines.atom
Feed Title www.theregister.com - Articles
Feed Link https://www.theregister.com/
Reply 0 comments