meowpunch/ApacheSparkWithScala

Apache Spark With Scala - hands on with big data

Apache Spark with Scala

• hands on exercises of an udemy course
  • spark core, Resilient Distributed Dataset
  • spark SQL, Datasets and Dataframes
    • item based collaborative filtering, recommendation system
  • spark ML, machine learning
  • spark streaming, DStreams
  • Structured Streaming

Data

• social network

  • fakefriends.csv: id, name, age, # of friends

• e-commerce

  • customer-orders.csv: customerId, productId, priceId

• superhero

  • Marvel-names.txt: heroId, heroName
  • Marvel-graph.txt: heroId, connectionId...

Exercise

Predict Real Estate Values with Decision Tree Model

predict real estate values per unit area base on several features

• features(X): HouseAge, DistanceToMRT, NumberConvenienceStores, Latitude, Longitude
• label(y): PriceOfUnitArea

Realtime processing with Twitter Streaming data

Twitter API

CHANGELOG, 15 Nov 2021 : Today, we are announcing that Twitter API v2 is now the primary version of the Twitter API. We have launched enough endpoints and functionality into Twitter API v2 to satisfy the needs of 90% of all existing Apps built on the Twitter API.

• sign up Twitter Developer site and get Keys
  • consumerKey: API Key
  • consumerSecret: API Secret

consumerKey 12345
consumerSecret 12345
accessToken 12345
accessTokenSecret 12345 

Reference

• Decision Tree with Spark