DataAnalyser .py [v1.0]

Creator: Stephen Alger
Machine Learening Introduction - Modelling Data, Deriving Modal Values, Measures of Central Tendency, Variation & Standard Deviation.
Last Update: 29 October 2019

Program Descriptor:

Utilising Pandas and Jupyter to create Dataframes to read in 1994 US Census data to compile an Analytical Base Table (ABT) which displays a Categorical and Continuous Data Quality Report. Dealing with Data Quality Issues (DQIs) such as Cardinality Issues, Outliers and Missing Values.

How to Run DataAnalyser Script:

--Run This Script as Follows
$ cd /Github/ML/ML_DataQuality
$ chmod 755 DataAnalyser.py     #-- if needed 
$ python3 DataAnalyser.py

Data Set In Use: 1994 US Census Data - see files below:

https://archive.ics.uci.edu/ml/datasets/Adult http://cseweb.ucsd.edu/classes/sp15/cse190-c/reports/sp15/048.pdf

Design Notes:

Read in US Census Data from .csv into panda Dataframe
Get the Column Headings from .txt file
Separate Continuous and Categorical Columns
Process Both types of Data in their respective ways: *Continuous -[1st Quartile ,Mean ,Median ,3rd Quartile ,Max ,Standard Deviation] *Categorical -[Modal Value, Mode FreQ, Mode %, 2nd Modal Value, 2nd Mode FreQ, 2nd Mode %]
Build respective output dataframes with input data columns as features
Output both Categorical and Continuous Data to separate CSV files in Analytical Base Table Format

Compatibility:

Tested on Python 3.7.2 - Mac OS Catalina (Not Compatible with Python 2.7)
Also Tested in Jupyter Notebook

Sample Output Graphs

Output Files:

ContinuousDataQualityReportABT.csv
CategoricalDataQualityReportABT.csv

Known Issues/ Points of Improvement (Non-Exhaustive) :

Yet to Be Added...

Stephen2697/MachineLearning