Rule-based Marketing Platform to Manage Call Detail Record (CDR)

A toy project of markting platform to manage call by using Spark Streaming as stream processor, Django as back-end, SB Admin 2 as front-end template and Redis as Message Queue.

This is final project for ELEN E6889 Large-Scale Stream Processing @ Columbia.

Introduction

Our project implement a rule-based marketing platform to manage Call Detail Record(CDR). We develop a Data Generator to simulate real-time CDR data. Streaming is achieved by message queue based on Redis and the queue manager, which is used to simulate subscriber mode. Five templates are developed based on Spark streaming to analyzed the streaming data. We choose Django framework to integrate the front-end and back-end and interact with the front-end UI. The front-end UI provides visualization of our results and the control of the back-end. Optimization and streaming algorithms are implemented to improve the performance of our system. We discuss how the size of data stream would affect the CPU utilization.

Structure

System Architecture

See more about our system from report

Code Structure

.
├── doc
├── pic
├── cdr_controller # -> Django back-end core
│   ├── data_generator.py # -> workload generator
│   ├── filters # -> different templates to process streams
│   │   ├── template_01.py
│   │   ├── ...
│   │   └── template_05.py
│   ├── get_result.py # -> get real-time result
│   ├── queue_manage.py # -> Django back-end core
│   ├── settings.py # -> Django setting
│   ├── urls.py # -> Django urls
│   ├── views.py # -> Django views: render HTML and data
│   └── wsgi.py # -> Django file
├── res # -> tmp result used in front-end
├── static # -> js packages used in front-end
├── templates # -> HTML files shown in front-end
├── db.sqlite3
├── manage.py # -> Django deploy file
├── LICENSE
├── CONTRIBUTING.md
└── README.md

Requirements

Python3: python3 environment and pip
- pip3 install Django: install Django framework
- pip3 install uuid: install uuid to generate uuid
- pip3 install redis: install redis package in Python3
- pip3 install phonenumbers: install phone number package in Python3
- pip3 install numpy pandas: install numpy and pandas in Python3
- pip3 install pyspark: install Spark package in Python3
Redis: install Redis. For more details, please see Get Started with Redis
```
 sudo apt update
 sudo apt install redis-server
```
Spark: install Spark, if don't want to built-in Spark environment provide by pyspark
- For more details, please see Spark Documentation

Deploy

run Django server on localhost:8000

# at root directory
python3 manage.py runserver localhost:8000

Usage

Homepage
- First visit localhost:8000/, which is a welcome page.
- press the start button to start workload generator in back-end
Dashboard
- After pressing the start button, web will redirect to localhost:8000/index.
- The dashboard page shows extracted features of stream in real-time.
- You can access other componment by the sidebar.
Workload Generator
- Workload generator will be shown at localhost:8000/workload_generator
- You can change the controlling parameters by filling the form.
- After pressing Update, back-end may need several minutes to update workload generator configuration.
Templates:
- Each templates details can be also found at localhost:8000/filters/template<id>
Policy & Plan page
- This page is used to decide different policy and send promotion to corresponding users.
- A table showing those users respesents the backend applys certain rules or send promotions to those users.
- The form in the left represent some rules that can be applied.

Details

See more details about how we design this project, how we implement each components, what algorithms we applied to optimize stream processing from report

Documentation

Proposal: proposal pdf
Report: report pdf
Presentation: presetation slides

Contribution

Workload Generator: Wenjie Chen, Yanchen Liu
Queue Manager: Yanchen Liu, Chong Hu
Templates: Wenjie Chen, Chong Hu
Django back-end construction: Chong Hu, Yanchen Liu
Front-end: Chong Hu
Documentation: Wenjie Chen, Chong Hu, Jiajing Su, Yanchen Liu

Acknowledgements

First, we would like to thank Professor Deepak S. Turaga. Without you, we can never finish our final project from scratch. Thank you for the instructions and help of a whole semester!

Second, we would like to thank all the authors and the contributors of our reference material. They have done a great work before us.

License

The code is distributed under a MIT license. See LICENSE for information.

Contact

Chong Hu ch3467: email
Wenjie Chen wc2685: email
Yanchen Liu yl4189: email
Jiajing Sun js5504:email

JackSnowWolf/Rule-based-Marketing-Platform-to-Manage-Call-Detail-Record