asifrahaman13/online-learning

The porject aims to create a automated system to train models based on data it received through the apis. The trained models then can be used to make the predictions. Used kafka for the data streaming process.

About the application.

The application can consume data using kafka and create models out of it.

API is used to make predictions.

Run the kafka server

Make sure you run your kafka application, zookeeper etc

Open kafka directory and run the following:

bin/zookeeper-server-start.sh config/zookeeper.properties

Open another terminal and inside the kafka directory

bin/kafka-server-start.sh config/server.properties

Next create topics and other configurations

bin/kafka-topics.sh --create --topic web-logs --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Clone the repository

git clone https://github.com/asifrahaman13/online-learning.git 

Create virtual environment.

virtualenv .venv

Next activate the virtual environment.

source .venv/bin/activate

Install the dependencies.

pip instal -r requirements.txt

Run the script.

uvicorn main:app --reload

chmod +x start.sh
bash start.sh

Now you need to start the server which will consume data from kafka.

uvicorn src.kafka_main:app --reload

Run the script to start the kafka server. That will consume all the data.

python3 kafka_sdk.py

In another terminal run the following command:

uvicorn src.main:app --reload --port=5000

Enter the sample data to train

http://127.0.0.1:8000/items/

Body:

{
  "Demand": 15520,
  "Cost": 38,
  "Price": 298.92049116490034,
  "Recession": true,
  "Economy": "Weak",
  "Competition": "High",
  "Market_Size": 1498
}

Predict the output

URL:

http://127.0.0.1:5000/predict/

{
  "Demand": [10, 150, 200],
  "Cost": [50, 75, 100],
  "Recession": ["low", "medium", "high"],
  "Economy": ["stable", "unstable", "recession"],
  "Competition": ["low", "medium", "high"],
  "Market_Size": [500, 1000, 1500]
}