Evolving a helper script into a 180,000-lines-of-Python-code project

Why best practices are called best practices

Mikhail Bushkov, Max Vogler

Databases Deployment/Continuous Integration and Delivery Development Open-Source Security

See in schedule

GRR Rapid Response (https://github.com/google/grr) is an incident response framework focused on remote live forensics.

It consists of a Python client (agent) that is installed on target systems, and Python server infrastructure that can manage and talk to clients. The goal of GRR is to support forensics and investigations in a fast, scalable manner to allow analysts to quickly triage attacks and perform analysis remotely.

GRR was started at Google in 2009 as a simple Python helper script used by Incident Response engineers. Eventually a little Python script got a little server component, was adapted to run on multiple systems (Mac, Linux, Windows), then a little UI was added and a few nice features were introduced (large-scale hunts, collection of predefined artifacts, memory analysis). A helper script has eventually evolved into a sophisticated framework with 180,000 lines of Python code.

In the presentation we’ll talk about the process of evolving a small prototype-like Python project into a production-ready system, using GRR as an example. The topics that we’ll cover are:
* Taking shortcuts - both in terms of design and implementation. Reasons for taking them and their eventual costs.
* Relying on Python’s power features (i.e. meta-classes, mixins)? Long-term consequences on maintainability and readability.
* Organising the project into separate PyPI packages - benefits of doing that.
* Continuous integration, testing and automated builds for various platforms - implementation costs and maintainability effects.

Type: Talk (45 mins); Python level: Intermediate; Domain level: Beginner