Backend Engineering Case Study

Introduction

This project is a simple backend application that is created for the Dream Case Study.

Redis Methodology

With redis methodology, microservice is capable of holding high throughput and decreased latency on related features

Realtime Group Pool Mechanism

With in the live tournament, we hold hot group details (empty slots, group id, group name, etc.) in redis. When a user wants to join a tournament, we can easily find the available group with the user's country. With this way, we can decrease the latency of the system.

Live Ranking Mechanism.

With the live ranking mechanism, we can easily find the user's ranking in the leaderboard with respect to the user's country and group.

Fast Updated Level Details with Redis.

With the updated level details, application updates the user's level details in the redis. With this way, we can easily re rank the leaderboard of countries and related group with respect to the user's current score.

Note that

All tournament related data is stored in the MySQL database. Redis is used for the live data that. After / with the tournament, application updates the MySQL database with the final results. After tournament, application clears the redis data for the next tournament.

How to run the application

To run the application, you need to have docker and docker-compose installed on your machine. You can Install them via the following links:

After installing docker and docker-compose, you can run the following command to start the application

cd deployment && docker-compose up -d

Diagrams

Project Organization Diagram

graph LR
    Project[Dream Case Study]
    PerformanceTesting[Application Performance Testing]
    A[App]
    B[api]
    C[common]
    D[configuration]
    E[enumaration]
    F[exceptions]
    G[tournament]
    H[user]
    I[resources]
    Project --> A
    Project --> PerformanceTesting
    A --> B
    A --> C
    A --> D
    A --> E
    A --> F
    A --> G
    A --> H
    A --> I
    B --> BA[StatusController.java]
    B --> BB[dto]
    BB --> BBA[request]
    BB --> BBB[response]
    C --> CA[utils]
    C --> CB[validation]
    G --> GA[controller]
    G --> GB[entity]
    G --> GC[repository]
    G --> GD[service]
    H --> HA[controller]
    H --> HB[entity]
    H --> HC[repository]
    H --> HD[service]
    I --> IA[application-test.yml]
    I --> IB[application.yml]
    I --> IC[db]

Docker Compose Diagram

C4Deployment
    title System Context diagram for Docker Compose
    Enterprise_Boundary(b0, "local-network") {
        Deployment_Node(redis, "Redis") {
            Container(redis, "Redis", "In Memory Database", "Provides in-memory database for tournament and user progress data")
        }

        Deployment_Node(mysqldb, "MySQL") {
            Container(mysqldb, "MySQL", "Relational Database", "Provides relational database for user and tournament data")
        }

        Deployment_Node(phpmyadmin, "PhpMyAdmin") {
            Container(phpmyadmin, "PhpMyAdmin", "UI for Database", "Provides web interface for managing MySQL database")
        }

        Deployment_Node(redisinsight, "RedisInsight") {
            Container(redisinsight, "RedisInsight", "UI for Redis", "Provides web interface for managing and monitoring Redis")
        }

        Deployment_Node(app, "Spring Boot") {
            Container(app, "Spring Boot", "Java Application", "Provides backend services for the application")
        }

        Rel(app, mysqldb, "Connects to")
        Rel(phpmyadmin, mysqldb, "Manages")
        Rel(app, redis, "Connects to")
        Rel(redisinsight, redis, "Manages and monitors")
    }

Sequence Diagram for Enter Tournament

sequenceDiagram
    participant U as User
    participant A as Application
    participant UserProgressService as UserProgressService
    participant TournamentService as TournamentService
    participant LeaderboardService as LeaderboardService
    participant RewardService as RewardService
    participant GroupPoolService as GroupPoolService
    participant GroupService as GroupService
    participant ParticipantService as ParticipantService
    participant M as MySQL
    participant R as Redis
    U ->> A: Enter Tournament
    A ->> UserProgressService: Get User Progress
    UserProgressService ->> M Query: Get User Progress
    M ->> UserProgressService: User Progress
    UserProgressService ->> A: User Progress
    A ->> UserProgressService: Check Minimum Requirements
    UserProgressService ->> A: Minimum Requirements
    A ->> GroupService: Enter Tournament
    GroupService ->> RewardService: Check unclaimed rewards
    RewardService ->> M Query: Get unclaimed rewards
    M ->> RewardService: Unclaimed rewards
    RewardService ->> GroupService: Unclaimed rewards
    GroupService ->> LeaderboardService: Get joined group [already]
    LeaderboardService ->> R Query: joined group id
    R ->> LeaderboardService: joined group id
    LeaderboardService ->> GroupPoolService: Get available groups
    GroupPoolService ->> R Query: Get available groups
    R ->> GroupPoolService: Available groups
    GroupPoolService ->> GroupService: Available groups
    GroupService ->> R Transaction: Create new group
    GroupService ->> M Transaction: Create new group
    M ->> GroupService: New group
    ParticipantService ->> M Transaction: Add participant to group
    M ->> ParticipantService: Participant added
    ParticipantService ->> GroupService: Participant added to group
    GroupService ->> A: group created
    A ->> LeaderboardService: Create / Update leaderboard
    LeaderboardService ->> R Transaction: Create / Update leaderboard
    R ->> LeaderboardService: Leaderboard Updated / Created
    LeaderboardService ->> A: Leaderboard Updated / Created
    A ->> UserProgressService: Withdraw required points
    UserProgressService ->> M Transaction: Withdraw required points
    M ->> UserProgressService: Points Withdrawn
    UserProgressService ->> A: Points Withdrawn
    A ->> U: Success

Postman Collection

You can find the postman collection here.

Performance Testing

On each commit, the application is tested with the performance tests. You can find the performance test workflow here. After each performance test workflow run, it generates simplified overviews of the performance tests as tables:

GetCountryLeaderboard

transaction	sampleCount	errorCount	errorPct	meanResTime	medianResTime	minResTime	maxResTime	pct1ResTime	pct2ResTime	pct3ResTime	throughput	receivedKBytesPerSec	sentKBytesPerSec
Total	400	0	0.0	18.81750000000002	13.0	1.0	100.0	35.900000000000034	88.5499999999999	96.0	766.2835249042146	0.0	0.0
Get Country Leaderboard	400	0	0.0	18.81750000000002	13.0	1.0	100.0	35.900000000000034	88.5499999999999	96.0	766.2835249042146	0.0	0.0

UpdateLevelUserWhenEnterTournament

transaction	sampleCount	errorCount	errorPct	meanResTime	medianResTime	minResTime	maxResTime	pct1ResTime	pct2ResTime	pct3ResTime	throughput	receivedKBytesPerSec	sentKBytesPerSec
Total	400	0	0.0	58.90999999999996	39.0	9.0	272.0	118.30000000000024	199.95	241.96000000000004	284.29282160625445	0.0	0.0
Update level of a user when user enters a tournament	400	0	0.0	58.90999999999996	39.0	9.0	272.0	118.30000000000024	199.95	241.96000000000004	284.29282160625445	0.0	0.0

EnterTournament

transaction	sampleCount	errorCount	errorPct	meanResTime	medianResTime	minResTime	maxResTime	pct1ResTime	pct2ResTime	pct3ResTime	throughput	receivedKBytesPerSec	sentKBytesPerSec
Enter a tournament	400	0	0.0	67.06250000000013	60.5	15.0	230.0	91.0	168.8999999999993	212.97000000000003	261.95153896529143	0.0	0.0
Total	400	0	0.0	67.06250000000013	60.5	15.0	230.0	91.0	168.8999999999993	212.97000000000003	261.95153896529143	0.0	0.0

GetGroupLeaderboard

transaction	sampleCount	errorCount	errorPct	meanResTime	medianResTime	minResTime	maxResTime	pct1ResTime	pct2ResTime	pct3ResTime	throughput	receivedKBytesPerSec	sentKBytesPerSec
Total	400	0	0.0	17.9325	14.0	1.0	82.0	34.0	74.29999999999984	78.0	806.4516129032257	0.0	0.0
Get Group Leaderboard	400	0	0.0	17.9325	14.0	1.0	82.0	34.0	74.29999999999984	78.0	806.4516129032257	0.0	0.0

UpdateLevelUser

transaction	sampleCount	errorCount	errorPct	meanResTime	medianResTime	minResTime	maxResTime	pct1ResTime	pct2ResTime	pct3ResTime	throughput	receivedKBytesPerSec	sentKBytesPerSec
Update level of a user	400	0	0.0	53.4875	42.0	6.0	235.0	82.80000000000007	223.3999999999985	233.0	298.28486204325134	0.0	0.0
Total	400	0	0.0	53.4875	42.0	6.0	235.0	82.80000000000007	223.3999999999985	233.0	298.28486204325134	0.0	0.0

CreateNewUser

transaction	sampleCount	errorCount	errorPct	meanResTime	medianResTime	minResTime	maxResTime	pct1ResTime	pct2ResTime	pct3ResTime	throughput	receivedKBytesPerSec	sentKBytesPerSec
Total	400	0	0.0	96.76749999999996	69.5	17.0	520.0	137.90000000000003	485.5999999999974	507.99	180.83182640144665	0.0	0.0
Create a new user	400	0	0.0	96.76749999999996	69.5	17.0	520.0	137.90000000000003	485.5999999999974	507.99	180.83182640144665	0.0	0.0

Also it generates a detailed report for the performance test. You can find the detailed report from the artifacts section of the workflow.

Detailed Report

configuration.json

In this file, I aimed to configure some basic jmeter parameters easily. Also, you can add a new test by adding a new json object to the configuration.json file. You can find the related file here

[
  {
    "testName": "CreateNewUser",
    "testDescription": "Create a new user",
    "className": "com.dreamgames.backendengineeringcasestudy.PerformanceTest",
    "methodName": "createUserTest",
    "threadCount": 20,
    "loopCount": 20,
    "duration": 20
  },
  {
    "testName": "UpdateLevelUser",
    "testDescription": "Update level of a user",
    "className": "com.dreamgames.backendengineeringcasestudy.PerformanceTest",
    "methodName": "updateLevelTest",
    "threadCount": 20,
    "loopCount": 20,
    "duration": 20
  }
]

How to run the performance tests Locally

In case you want to run the performance tests locally, you can use the following command:

  cd deployment/performance-test
  docker compose up -d --build --force-recreate

It will create a simple non-gui docker environment for the performance tests. You can find the related docker-compose file here. After the environment is up It will check the application with trying rest api calls over /status endpoint. You can find the related script here.

If you have already up and running docker environment. And, dont want to wait docker build stages. Still, you can use jmeter_run.py script to run the performance tests. With parameterized methodology:

If the running environment is local:

defines user variables for the jmeter test
builds the project with maven
moves required jar files and configures with configuration.json file
runs the jmeter tests with the given parameters
creates a detailed report

  cd scripts
  python3 jmeter_run.py

If the running environment is github actions:

fetches user variables
configures the jmeter tests with the given parameters
runs the jmeter tests with the given parameters
creates a detailed report and saves it as proper artifact

TODO

Notes

I did not focus on the security of the application. Because it was not mentioned in the case study. However, I also asked for the security requirements of the application via email. Yet, I did not get any response. So, I did not focus on the security of the application.

What I can add for the security of the application:

Add SSO Service to the docker environment
Enable Spring Security for the application
Add Security Layers with Spring 3.0
Implement an JWT Decoder & User Details Service for the application
Cors Configuration for the application
Add CSRF Protection for the application
Add JWT Auth Converter for the application

abdullahbodur/dream-case-study