This project is part of the Udacity Azure ML Nanodegree. In this project, we build and optimize an Azure ML pipeline using the Python SDK and a provided Scikit-learn model. This model is then compared to an Azure AutoML run. In this project pipeline was build using HyperDrive and AutoML and performance of both the approach was compared.
Data Set Information:
The data is related with direct marketing campaigns of a Portuguese banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be ('yes') or not ('no') subscribed.
The classification goal is to predict if the client will subscribe (yes/no) a term deposit (variable y).
Input variables:
- age (numeric)
- job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')
- marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)
- education (categorical: 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')
- default: has credit in default? (categorical: 'no','yes','unknown')
- housing: has housing loan? (categorical: 'no','yes','unknown')
- loan: has personal loan? (categorical: 'no','yes','unknown')
- contact: contact communication type (categorical: 'cellular','telephone')
- month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')
- day_of_week: last contact day of the week (categorical: 'mon','tue','wed','thu','fri')
- duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.
- campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)
- pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)
- previous: number of contacts performed before this campaign and for this client (numeric)
- poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')
- emp.var.rate: employment variation rate - quarterly indicator (numeric)
- cons.price.idx: consumer price index - monthly indicator (numeric)
- cons.conf.idx: consumer confidence index - monthly indicator (numeric)
- euribor3m: euribor 3 month rate - daily indicator (numeric)
- nr.employed: number of employees - quarterly indicator (numeric)
- y - has the client subscribed a term deposit? (binary: 'yes','no')
The best performing model
There were two approach taken to solve the problem by creating pipelines:
- Using a LogisticRegression model with scikit-learn, and tuning the model's hyperparameters using Azure's HyperDrive.
- Using Azure AutoML for training several kinds of models such as LightGBM, XGBoost, Logistic Regression, VotingEnsemble, among others algorithms and find the best one based on accuracy.
The best performing model was VotingEnsemble with accuracy of 0.9172 which was build by second approach i.e. Azure's AutoML.
Pipeline architecture, including data, hyperparameter tuning, and classification algorithm.
- Steps involved in the entry script(train.py):
- Importing data using URL using TabularDatasetFactory. Link for dataset
- Cleaning the data - dropping columns, removing rows with missing entries, one hot encoding the categorical data, feature engineering etc.
- Splitting the data into train and test sets.
- Training the logistic regression model using arguments from the HyperDrive runs.
- Calculating the accuracy score.
- Creating best run model output file.
- Steps involved in the project notebook (udacity-project.ipynb):
- Assigning a compute cluster to be used as the target.
- Specifying the parameter sampler (RandomSampling).
- Specifying an Early Stopping Policy (BanditPolicy).
- Creating a SKLearn Estimator.
- Creating a HyperDriveConfig using the estimator, hyperparameter sampler, and policy.
- Getting the best run model and its metrics.
- Saving the model.
Benefits of parameter sampler
- Parameter sampler is used to define the different hyperparameters as well as different range of values of that hyperparameters that one wants to tune the model traing process.
- In this project, Discrete search sampler was used for both hyperparemeters (C and max_iter) along with Random sampling method.
- In random sampling, hyperparameter values are randomly selected from the defined search space.
- In case of hyperparameters, C represents the inverse regularization parameter and max_iter represents the maximum number of iterations.
Benefits of Early Stopping Policy
- Early Stopping policy is used to prevent model training process from running for a long time and consuming up resources.
- Early stopping policy used in this project is BanditPolicy.
- The benefit of using this early stopping policy is that it terminates training process as soon as it finds that the primary metric (accuray in this project) is not within the pre-defined threshold.
- Azure AutoML is a no code environment, capable of training many different models in a short period of time like RandomForests, BoostedTrees, XGBoost, LightGBM, SGDClassifier, VotingEnsemble, etc.
- Following config parameters are set for AutoML:
- experiment_timeout_minutes=30 : By default set in simulator for reducing excessive use of computational resources.
- task='classification' : As the problem was classification based to check the client subscribed a term deposit or not.
- compute_target : The compute target with specific compute i.e. "Standard_D2_V2" and max_nodes.
- training_data : The data on which the algorithm will be trained.
- label_column_name : The name of the column that used for prediction (target column).
- n_cross_validations=3 : It is how many cross validations to perform when user validation data is not specified.
- primary_metric = 'accuracy' : The primary metric for optimizing the model selection.
- enable_early_stopping = True : Parameter to enable early termination policy.
The best performing model was VotingEnsemble with accuracy of 0.9172
- Accuracy acheived using PythonSDK HyperDrive is 0.9096611026808296 whereas incase of Azure AutoML is 0.9173.
- There is much difference in accuracy of AutoML and HyperDrive but the advantage of AutoML is one doesn't require to code much and using little efforts one can train various models and also get its performance metrics and visualizations.
- And as compared to HyperDrive, AutoML architecture is quite superior, which enables to training 'n' number of models quitely and efficiently.
- Handling implance data.
- Bayesian Parameter Sampler might improve model accuracy.
- Implementing neural network or trees to implement model using HyperDrive.
- Following is proof for custer clean up
