Regression Model Validation - Lab

Introduction

In this lab, you'll be able to validate your Ames Housing data model using a train-test split.

Objectives

You will be able to:

Perform a train-test split
Prepare training and testing data for modeling
Compare training and testing errors to determine if model is over or underfitting

Let's Use Our Ames Housing Data Again!

We included the code to load the data below.

# Run this cell without changes
import pandas as pd
import numpy as np
ames = pd.read_csv('ames.csv', index_col=0)
ames

Perform a Train-Test Split

Use train_test_split (documentation here) with the default split size. At the end you should have X_train, X_test, y_train, and y_test variables, where y represents SalePrice and X represents all other columns.

# Your code here: split the data into training and test sets

Prepare Both Sets for Modeling

This code is completed for you and should work as long as the correct variables were created.

# Run this cell without changes
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder

continuous = ['LotArea', '1stFlrSF', 'GrLivArea']
categoricals = ['BldgType', 'KitchenQual', 'Street']

# Instantiate transformers
log_transformer = FunctionTransformer(np.log, validate=True)
ohe = OneHotEncoder(drop='first', sparse=False)

# Fit transformers
log_transformer.fit(X_train[continuous])
ohe.fit(X_train[categoricals])

# Transform training data
X_train = pd.concat([
    pd.DataFrame(log_transformer.transform(X_train[continuous]), index=X_train.index),
    pd.DataFrame(ohe.transform(X_train[categoricals]), index=X_train.index)
], axis=1)

# Transform test data
X_test = pd.concat([
    pd.DataFrame(log_transformer.transform(X_test[continuous]), index=X_test.index),
    pd.DataFrame(ohe.transform(X_test[categoricals]), index=X_test.index)
], axis=1)

Fit a Linear Regression on the Training Data

# Your code here: import the linear regression model class, initialize a model

# Your code here: fit the model to train data

Evaluate and Validate Model

Generate Predictions on Training and Test Sets

# Your code here: generate predictions for both sets

Calculate the Mean Squared Error (MSE)

You can use mean_squared_error from scikit-learn (documentation here).

# Your code here: calculate training and test MSE

If your test error is substantially worse than the train error, this is a sign that the model doesn't generalize well to future cases.

One simple way to demonstrate overfitting and underfitting is to alter the size of our train-test split. By default, scikit-learn allocates 25% of the data to the test set and 75% to the training set. Fitting a model on only 10% of the data is apt to lead to underfitting, while training a model on 99% of the data is apt to lead to overfitting.

Level Up: Evaluate the Effect of Train-Test Split Size

Iterate over a range of train-test split sizes from .5 to .9. For each of these, generate a new train/test split sample. Preprocess both sets of data. Fit a model to the training sample and calculate both the training error and the test error (MSE) for each of these splits. Plot these two curves (train error vs. training size and test error vs. training size) on a graph.

# Your code here

Extension

Repeat the previous example, but for each train-test split size, generate 10 iterations of models/errors and save the average train/test error. This will help account for any particularly good/bad models that might have resulted from poor/good splits in the data.

# Your code here

What's happening here? Evaluate your result!

Summary

Congratulations! You now practiced your knowledge of MSE and used your train-test split skills to validate your model.

ovilar/dsc-regression-model-validation-lab