/data-intensive-app

Repository to hold some studies on data intensive apps.

data-intensive-app

Repository to hold a study with implementation related to data intensive apps.

Overview

The data-intensive application I am refering to are those applications where there are near or more than 1 TB of data stored, and it is necessary to perform fast queries to reply to the user, less than 500ms.

This kind of challenge is growing with some specific types of business working online, such as social media, online banking, etc. These type of business have to usually store a lots os user information, either for fraud analisys, data analysis, or to allow the user to navigate through anytime on the history, such as old social media posts, all banking statements, previous purchages online on the online store.

The implementation and this repository

The idea of this repository is to gather and apply some best practices related to data-intensive applications, addressing some common problems when dealing with data-intensive apps such as caching, pagination, indexes, concurrency, etc.

Out of scope

The data-intensive app use case that will be developed on this repository will only consider some software layers to be able to contain more reproducable scenarios. So layers such as infrastructure, cloud, specific cloud provider technology will not be considered.

Also some implementation not related with the intensive-data goal will not be considered, such as authentication & authorization, payment methods, UX, well designed front end.

Application Use Cases

News. The use case will be based on News inserted on the data layer. List of use cases below. use case daigram

Architecture

  • Domain Driven Design
  • Clean Architecture

Stack

Backend

  • App
    • NET 6
    • C# 10
    • Dapper
    • gRPC & REST
  • Storage
    • redis
    • mySQL - TBD
    • MongoDB - TBD
  • Performan Tests
    • Locust

References

Tools