Re-Architecting Spark For Performance Understandability

"This talk will describe Monotasks, a new architecture for the core of Spark that makes performance easier to reason about. In Spark today, pervasive parallelism and pipelining make it difficult to answer even simple performance questions like "what is the bottleneck for this workload?" As a result, it's difficult for developers to know what to optimize, and it's even more difficult for users to understand what hardware to use and what configuration parameters to set to get the best performance."

The post Re-Architecting Spark For Performance Understandability appeared first on insideHPC.