In today’s software development landscape, data plays a powerful role, which has led to the rise of the data driven development paradigm. The purpose of this repository is to consolidate and provide all the resources, guides and documentation necessary to foster data driven development.
Good data driven development should not only collect lots of data and make them available, but we also need to aggregate the information in meaningful ways. The data-driven approach, particularly with well-defined benchmarks and metrics provides greater visibility into in-progress work being done by teams and also helps in important decision making processes. This allows each contributor to see the impact of their individual effort on overall project success (or failure) and also enables teams to effectively measure the level of success.
Similar to test driven development, data driven development means that you must be able to answer questions quick, i.e. combining different data sources, going back in time, changing the granularity of data, being able to change the visualizations such as prototypying in Jupyter notebooks vs creating dashboards vs generating PDF reports.
The repository is organized into the following directories:
-
data-presentation- This folder comprises of documentation/guides and best practices on how data can be represented in different ways by using various tools such as creating dashboards in Grafana or Superset, developing data visualizations using Jupyter notebooks etc. -
data-sources- This folder comprises of documentation and guides for various data sources used in the team such as Prometheus, GitHub etc, what type of data is collected from these sources and in what context these data sources can be used. -
notebooks- This folder comprises of Jupyter notebooks providing examples on how to interact with the different data sources and perform analysis on top of the data. -
examples- This folder comprises of monitoring guides for different examples of applications and services.