/pyspark-realtime-streaming-sentiment-analysis

⏱ Real-Time Sentiment Analysis using PySpark and simulation of Twitter/X API using FastAPI

Primary LanguagePython

PYSPARK REALTIME STREAMING SENTIMENT ANALYSIS

⏱ Real-Time Sentiment Analysis using PySpark and FastAPI


PROTOTYPE VIDEO

video.mov



HOW TO EXECUTE

Terminal

git clone https://github.com/raghavtwenty/pyspark-realtime-streaming-sentiment-analysis.git


cd code/


pip install -r requirements.txt

Run Fast API Server

uvicorn _1_fastapi_server:app --reload

Run PySpark

python _3_sentiment_analysis_pyspark.py

DOMAIN

Big Data Analytics


OBJECTIVE

Perform sentiment analysis for the realtime streaming data


INTRODUCTION

In the era of big data, the ability to process and analyze real-time data streams is crucial for gaining actionable insights. This project aims to demonstrate a real-time streaming sentiment analysis application using PySpark and FastAPI. The sentiment analysis model processes incoming data in real time, determining the sentiment polarity (positive, negative, or neutral) of each data point. This prototype is particularly beneficial for applications such as social media monitoring, customer feedback analysis, and other domains where timely sentiment information is critical.

I haven't used Twitter/X API, Instead simulated the same using FastAPI. Free of cost.


FEATURES

  • Highly Scalable
  • Real-time
  • Parallel Processing


TECHNOLOGIES USED

  • Spark
  • Fast API


END USERS

  1. Students
  2. Data Analyst
  3. Data Scientists


OUTPUTS

  • Fast API Server

    1

  • PySpark

    2

  • PySpark

    3


END OF README