Parallel computing in Python: Current state and recent advances

Pierre Glaser

Distributed Systems Multi-Processing Multi-Threading Performance Scientific Libraries (Numpy/Pandas/SciKit/...)

See in schedule

Parallel computing in Python: Current state and recent advances
---------------------------------------------------------------

Modern hardware is multi-core. It is crucial for Python to provide
high-performance parallelism. This talk will expose to both data-scientists and
library developers the current state of affairs and the recent advances for
parallel computing with Python. The goal is to help practitioners and
developers to make better decisions on this matter.

I will first cover how Python can interface with parallelism, from leveraging
external parallelism of C-extensions –especially the BLAS family– to Python's
multiprocessing and multithreading API. I will touch upon use cases, e.g single
vs multi machine, as well as and pros and cons of the various solutions for
each use case. Most of these considerations will be backed by benchmarks from
the scikit-learn machine
learning library.

From these low-level interfaces emerged higher-level parallel processing
libraries, such as concurrent.futures, joblib and loky (used by dask and
scikit-learn) These libraries make it easy for Python programmers to use safe
and reliable parallelism in their code. They can even work in more exotic
situations, such as interactive sessions, in which Python’s native
multiprocessing support tends to fail. I will describe their purpose as well as
the canonical use-cases they address.

The last part of this talk will focus on the most recent advances in the Python
standard library, addressing one of the principal performance bottlenecks of
multi-core/multi-machine processing, which is data communication. We will
present a new API for shared-memory management between different Python
processes, and performance improvements for the serialization of large Python
objects ( PEP 574, pickle extensions). These performance improvements will be
leveraged by distributed data science frameworks such as dask, ray and pyspark.

Type: Talk (30 mins); Python level: Intermediate; Domain level: Beginner