In regression, you are predicting continous values so it makes sense to discuss error as a distance of how far off our estimates were. When classifying a binary variable, however, a model is either correct or incorrect. As a result, we tend to quantify this in terms of how many false positives versus false negatives we come across. In particular, we examine a few different specific measurements when evaluating the performance of a classification algorithm. In this lab, you'll review precision, recall, accuracy, and F1 score in order to evaluate our logistic regression models.
In this lab you will:
- Implement evaluation metrics from scratch using Python
Let's take a moment and review some classification evaluation metrics:
At times, it may be best to tune a classification algorithm to optimize against precision or recall rather than overall accuracy. For example, imagine the scenario of predicting whether or not a patient is at risk for cancer and should be brought in for additional testing. In cases such as this, we often may want to cast a slightly wider net, and it is preferable to optimize for recall, the number of cancer positive cases, than it is to optimize precision, the percentage of our predicted cancer-risk patients who are indeed positive.
import pandas as pd
df = pd.read_csv('heart.csv')
df.head()
Split the data first into X
and y
, and then into training and test sets. Assign 25% to the test set and set the random_state
to 0.
# Import train_test_split
# Split data into X and y
y = None
X = None
# Split the data into a training and a test set
X_train, X_test, y_train, y_test = None
- Import and instantiate
LogisticRegression
- Make sure you do not use an intercept term and use the
'liblinear'
solver - Fit the model to training data
# Import LogisticRegression
# Instantiate LogisticRegression
logreg = None
# Fit to training data
model_log = None
model_log
def precision(y, y_hat):
# Your code here
pass
def recall(y, y_hat):
# Your code here
pass
def accuracy(y, y_hat):
# Your code here
pass
def f1_score(y, y_hat):
# Your code here
pass
Do this for both the training and test sets.
# Your code here
y_hat_train = None
y_hat_test = None
Great job! Now it's time to check your work with sklearn
.
Each of the metrics we calculated above is also available inside the sklearn.metrics
module.
In the cell below, import the following functions:
precision_score
recall_score
accuracy_score
f1_score
Compare the results of your performance metrics functions above with the sklearn
functions. Calculate these values for both your train and test set.
# Your code here
Nicely done! Did the results from sklearn
match that of your own?
Calculate and then plot the precision, recall, accuracy, and F1 score for the test and training splits using different training set sizes. What do you notice?
import matplotlib.pyplot as plt
%matplotlib inline
training_precision = []
testing_precision = []
training_recall = []
testing_recall = []
training_accuracy = []
testing_accuracy = []
training_f1 = []
testing_f1 = []
for i in range(10, 95):
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= None) # replace the "None" here
logreg = LogisticRegression(fit_intercept=False, C=1e20, solver='liblinear')
model_log = None
y_hat_test = None
y_hat_train = None
# Your code here
Create four scatter plots looking at the train and test precision in the first one, train and test recall in the second one, train and test accuracy in the third one, and train and test F1 score in the fourth one.
We already created the scatter plot for precision:
# Train and test precision
plt.scatter(list(range(10, 95)), training_precision, label='training_precision')
plt.scatter(list(range(10, 95)), testing_precision, label='testing_precision')
plt.legend()
plt.show()
# Train and test recall
# Train and test accuracy
# Train and test F1 score
Nice! In this lab, you calculated evaluation metrics for classification algorithms from scratch in Python. Going forward, continue to think about scenarios in which you might prefer to optimize one of these metrics over another.