The dos and don'ts of task queues

Lessons learned about task queues and the final solution of ours in Celery

Petr Stehlík

ASYNC / Concurreny Architecture Best Practice Case Study failures/mistakes

At Kiwi.com we heavily rely on task queues and asynchronous execution of code to process large amounts of requests coming to our back-ends. With the separation of our codebase to microservices, we can quickly try new tools and different approaches to process these large volumes of requests. The microservice we’ll be talking about is making unreliable slow 3rd party services reliable and asynchronous with a bit of business logic sprinkled on top of it. We’ll tell a failure story of ours but resulting in a valuable lesson.

Most of our services use Celery and it’s the go-to tool for new services as well but we wanted to be different with this new microservice. RQ is the next best choice for task queues and it is presented as simpler and more straightforward than Celery. That can definitely be true but after 3 weeks of research, development and struggling we found out the unpleasant truth about being simple and making the right choices. We won’t talk about comparing the frameworks but rather about the approach on how to experiment with new things in your environment. After that, we’ll present our current setup which can take upon any number of tasks*. How we orchestrate the app and continuously integrate and deploy and what fun things await ahead of us in the development.

*Conditions may apply.

Type: Talk (30 mins); Python level: Intermediate; Domain level: Beginner

Petr Stehlík

Kiwi.com

Python engineer at Kiwi.com focusing on finance apps. Based in Brno, Czech Republic.

I am a developer who likes to break stuff and then put it back together which then works even better. I am always keen to share my knowledge and learn from others.