Budapest Data 2015

Agile software development values responding to change over following a plan. Responding to changes means allowing data scientists to experiment with data and allowing developers to easily modify data processing and even make mistakes without taking huge risks. Well designed data pipelines gives organizations flexible data analysis. In this session we'll show how to design architectures that make it easy and safe to extend and modify data analysis software.

We will look at how to design an agile data processing architecture using Apache Hadoop, Apache Kafka and stream processing frameworks. The architectures we’ll discuss make it easy to add new data sources, experiment with new analysis algorithms and correct data processing errors. All this makes the data pipeline both flexible and safe.