jainamshah102/Big-Data-Analytics-Using-Spark

Big data analytics project using Apache Spark.

Big-Data-Analytics-Using-Spark

Project By

Jainam Shah	Peeya Thacker
1911120	1911127
B4	B4

College

K.J Somaiya College Of Engineering, Vidhyavihar

Faculty

Prof. Pradnya Bhangale	Prof. Vaibhav Vasani

What is spark?

• Spark is a unified analytics engine for large-scale data processing.
• It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis.
• It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

How to install Apache Spark on Windows 10?

• Prerequisites

  • A system running Windows 10
  • A user account with administrator privileges (required to install software, modify file permissions, and modify system PATH)
  • Command Prompt or Powershell
  • A tool to extract .tar files, such as 7-Zip

Install Apache Spark on Windows

• Installing Apache Spark on Windows 10 may seem complicated to novice users, but this simple tutorial will have you up and running.
• If you already have Java 8 and Python 3 installed, you can skip the first two steps.

Step 1: Install Java 8

• Open Browser and navigate to https://www.java.com/download/ie_manual.jsp.
• Download the latest stable version.
• Add it to the environment variable.

Step 2: Install Python

• Open Browser and navigate to https://www.python.org/downloads/.
• Download the latest stable version.
• Add it to the environment variable.

Step 3: Download Apache Spark

• Open a browser and navigate to https://spark.apache.org/downloads.html.
• Under the Download Apache Spark heading, there are two drop-down menus. Use the current non-preview version.

Running the project

• Make sure Java, Python and Spark are installed correctly by running the following command.

  • python --version
  • java --version
  • pyspark --version

• Open the terminal/command prompt.

• Clone the repository.
  

  • https://github.com/jainam2385/Big-Data-Analytics-Using-Spark

• Next open jupyter notebook and run each cell.

Included components:

• Apache Spark: An open-source, fast and general-purpose cluster computing system.
  

• Jupyter Notebooks: An open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text.

  
  
  

Featured technologies:

• Data Science: Systems and scientific methods to analyze structured and unstructured data in order to extract knowledge and insights.
  

• Python: Python is a programming language that lets you work more quickly and integrate your systems more effectively.

Tools and Technologies used