Designing Data Systems: Complexity & Modular Design

Going from notebooks to creating machine learning systems that work in the real-world means shifting context from writing simple scripts, notebooks and visualizing data in the lab environment. Now it's the time to think about building a system and good systems have key characteristics namely resilience, performance and reliability. Often, as we design systems the frictional drag of complexity starts to become apparent due to incremental decisions taken during the developmental process. These incremental decisions are a result of several constraints such as deadlines, budget constraints, and technical skills of the development team. Complexity in the context of a system is anything that makes it hard to understand the system and makes it difficult to modify the system at a later stage.