kaopanboonyuen/2110531_DataScience_2022s1

Data Science Tools Course at Dept. of Computer Engineering, Chula 2022

🐳 2110531 Data Science and Data Engineering Tools @Chula 2022

Welcome to the Data Science and Data Engineering Tools Course (2110531) at Chulalongkorn University! This repository contains the code, exercises, and resources to guide you through the fascinating world of data science, deep learning, and more. Each week, you'll dive into new concepts, reinforced with hands-on labs, to help you build a strong foundation in data science.

📚 Weekly Labs and Exercises

Week 1: Intro to Numpy, Pandas

1. Numpy: Introduction to numerical computing with Numpy.

2. Pandas: Basic data manipulation with Pandas.

3. Pandas with Youtube stat data: Analyzing YouTube video statistics with Pandas.

4. (Advanced) Pandas with Youtube stat data: Advanced analysis of YouTube video statistics.

5. Assignment (Pandas with Youtube stat data): Hands-on assignment analyzing YouTube video statistics.

Week 2: Data Preparation

1. EDA: Exploratory Data Analysis on loan data.

2. Impute Missing Value: Handling missing data in loan datasets.

3. Split Train/Test: Splitting datasets into training and testing sets.

4. Outliers with Log: Identifying and handling outliers using logarithmic transformation.

5. Outliers with Log (Titanic DataSet): Advanced outlier analysis using the Titanic dataset.

6. Assignment: Titanic dataset analysis assignment.

Week3-4: Traditional ML

1. Decision Trees:

2. Linear Regression:

3. Logistic Regression:

4. Neural Network:

5. K Nearest Neighbors:

6. SVM:

7. Save and Load Model:

8. K-Means:

9. Market-Basket Analysis:

Assignment for Week3 (Safe to eat or deadly poison?):

Week5-6: Intro to Deep Learning

1. Image classification (basic): flower classification

2. Image classification (advanced): flower classification

3. Semantic Segmentation (UNET): The Oxford-IIIT pet dataset

4. LSTM: Stock price prediction

5. SARIMAX: PM2.5 forecasting

Assignment (Fashion MNIST):

Week8: Data Storage with Redis

Redis Example using local data

Assignment (connect to redis server)

Week9: Data Storage with Redis

1. Basic Webpage Scarping

2. Wikipeia Data Extraction

3. Settrade Rest API

4. Twitter Data Extraction

5. Selenium

Assignment (Counting วันพระ)

Week10: Data Ingestion with Kafka

1. Several simple examples including both produxer and consumer in simple folder

2. Complex example in complex folder

3. AVRO Producer and Consumer

4. Group example in group folder

Assignment (Transaction Verifier)

Note: Do not forget to upload the following schema files to your Colab

• transaction.avsc
• submit.avsc
• result.avsc

Week11: Big Data Processing with Spark

1. Basic Spark

Note: Do not forget to upload the following data file to your Colab

• (star-wars.txt)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/blob/main/code/week11_spark/star-wars.txt]

2. Spark SQL

Note: Do not forget to upload the following data file to your Colab

• (bank-additional-full.csv)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/blob/main/code/week11_spark/bank-additional-full.csv]

3. Spark ML

Note: Do not forget to upload the following data file to your Colab

• (bank-additional-full.csv)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/blob/main/code/week11_spark/bank-additional-full.csv]

Assignment (Analyze IMDB)

Note: Do not forget to upload the following data file to your Colab

• (netflix-rotten-tomatoes-metacritic-imdb.csv)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/blob/main/code/week11_spark/assignment/netflix-rotten-tomatoes-metacritic-imdb.csv]

Week12: Ops Stars

1. Several airflow examples in (airflow folder)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/tree/main/code/week12_orchestration/airflow]

2. Several fastapi examples in (fastapi folder)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/tree/main/code/week12_orchestration/fastapi]

🛠 Environment Setup

The code in this repository is designed to run in Google Colab or a local Python environment. To get started locally, ensure you have Python 3.8+ installed and use the following steps to set up your environment:

git clone https://github.com/kaopanboonyuen/2110446_DataScience_2021s2.git
cd 2110446_DataScience_2021s2
pip install -r requirements.txt

📚 References

1. https://www.kaggle.com/code
2. https://www.tensorflow.org/tutorials
3. https://github.com/topics/machine-learning
4. https://archive.ics.uci.edu/ml/datasets.php
5. https://colab.research.google.com/notebooks/

🎓 License

This project is licensed under the MIT License. See the LICENSE file for more information.

🛡️ Disclaimer

This repository is for educational purposes only. All code and resources are provided as-is, without any guarantees or warranties.

For any questions or feedback, please contact me at Kao Panboonyuen.