ASYNC / Concurreny Case Study Multi-Processing Multi-Threading Performance
See in schedule Download SlidesYou've been given a task. You need to download some files from a server to your local machine. The files are fairly small, and you can list and access these files from the remote server through a REST API. You'd like to download them as fast as possible. The catch? There's a billion of them. Yes, one billion files.
How would would you do this? Would you do this synchronously in a single for loop? Would you use a producer/consumer queue with threads? Multiprocessing? Asyncio?
In this talk, we'll examine 3 different mechanisms for concurrently downloading files: multithreading, multiprocessing, and asyncio.
For each of these mechanisms we'll look at design best practices, how to handle debugging and error handling, and of course the overall performance. By examining three different approaches using the same data set, we gain a better understanding of the tradeoffs of each approach so we can pick the right library for the job.
Type: Talk (30 mins); Python level: Advanced; Domain level: Intermediate
James Saryerwinnie is a Software Development Engineer at Amazon Web Services where he works on the AWS CLI and boto3, the AWS SDK for Python. He's also the creator of AWS Chalice, a serverless microframework for python.
