This repository shows, how to identify and remove the outliers using Pyspark
The dataset is collected from https://archive.ics.uci.edu/ml/datasets/wholesale+customers
- FRESH: annual spending (m.u.) on fresh products (Continuous);
- MILK: annual spending (m.u.) on milk products (Continuous);
- GROCERY: annual spending (m.u.)on grocery products (Continuous);
- FROZEN: annual spending (m.u.)on frozen products (Continuous)
- DETERGENTS_PAPER: annual spending (m.u.) on detergents and paper products (Continuous)
- DELICATESSEN: annual spending (m.u.)on and delicatessen products (Continuous);
- CHANNEL: customers’ Channel - Horeca (Hotel/Restaurant/Café) or Retail channel (Nominal)
- REGION: customers’ Region – Lisnon, Oporto or Other (Nominal)
- Lisbon 77
- Oporto 47
- Other Region 316
- Horeca 298
- Retail 142
- The initial dataset contains 440 records with 8 features
- There are total 6 numerical features & 2 categorical features
- The original spark dataframe has the datatypes of strings
- The numerical features were converted into IntegerType using cast function
- Created a customized function to identify outliers in each record
- Applyng the above customized function, enables us to identify total outliers in each record, based on each feature
- Filtering the dataset based on the total outliers which are <=1, to eliminate the records with more than 2 outliers
- The new dataframe, contains 399 records after removing the outliers against 440 records in the inital data frame
- Comparing the outliers from the original dataset to the new dataset after outlier removal using a box plot
- There are still some outliers available in the dataset., even after removing majority of the outliers.
- This shows that the data is skewed, however, the majority of the outliers are removed