In the previous lesson, we learned the mathematical definition of a gradient. We saw that the gradient of a function was a combination of our partial derivatives with respect to each variable of that function. We saw the direction of gradient descent was simply to move in the negative direction of the gradient. For example, if the direction of ascent of a function is a move up and to the right, the descent is down and to the left. In this lesson we will apply gradient descent to our cost function to see how we can move towards a best fit regression line by changing variables of
Think about why gradient descent applies so well to a cost function. Initially, we said that the cost of our function, meaning the difference between what our regression line predicted and the dataset, changed as we altered the y-intercept or the slope of the function.
Remember that mathematically, when we say cost function, we use the residual sum of squares where $$ RSS = \sum_{i=1}^n(actual - expected)^2 = \sum_{i=1}^n(y_i - \hat{y})^2 = \sum_{i=1}^n(y_i - mx_i + b)^2$$ for all
And RSS takes the difference between
And we when we just plotted how RSS changes as we change one variable of our regression line,
import plotly
from plotly.offline import init_notebook_mode, iplot
from graph import m_b_trace, trace_values, plot
init_notebook_mode(connected=True)
b_values = list(range(70, 150, 10))
rss = [10852, 9690, 9128, 9166, 9804, 11042, 12880, 15318]
cost_curve_trace = trace_values(b_values, rss, mode="lines", name = 'RSS with changes to y-intercept')
plot([cost_curve_trace])
In two dimensions, we decrease our RSS simply by moving forwards or backwards along the cost curve which is the equivalent of changing our variable, in this case y-intercept. So the cost curve above indicates that changing the regression line from having a y-intercept of 70 to 80 decreases our cost, the RSS.
Allowing us to change both variables,
Because the RSS is a function of how we change our values of
In the function above,
Just our other multivariable functions we have seen thus far, we can display it in three dimensions, and it looks like the following.
The three-dimensional graph above shows how the cost associated with our regression line changes as the slope and y-intercept values are changed.
Let's explore using gradient descent to determine how to change our regression line when we can alter both
Now that we are applying gradient descent to our cost curve
$J(m, b)$ , the technique should answer how much to move the$m$ variable and the$b$ variable to produce the greatest decrease in cost, or RSS. In other words, when altering our regression line, we want to know how much of this change should be derived from a move in the slope versus how much should be derived from a change in the y-intercept.
As we know, the gradient of a function is simply the partial derivatives with respect to each of the variables, so:
In calculating the partial derivatives of our function
Ok, so let's take our partial derivatives of the following:
Let's start with taking the partial derivative with respect to
Now this is a tricky function to take the derivative of. So we can use functional composition followed by the chain rule to make it easier. Using functional composition, we can rewrite our function
Now using the chain rule to find the partial derivative with respect to a change in the slope, gives us:
Our next step is to solve these derivatives individually:
Each of the terms are treated as constants, except for the middle term.
Now plugging these back into our chain rule we have:
$\frac{dJ}{dg}J(g(m,b))\frac{dg}{dm}g(m,b) = (2g(m,b))-x = 2(y - (mx + b))*-x $
So
$$\frac{\delta J}{\delta m}J(m, b) = 2*(y - (mx + b))-x = -2x(y - (mx + b )) $$
Ok, now let's calculate the partial derivative with respect to a change in the y-intercept. We express this mathematically with the following:
Then once again, we use functional composition following by the chain rule. So we view our cost function as the same two functions
So applying the chain rule, to this same function composition, we get:
Now, our next step is to calculate these partial derivatives individually.
From our earlier calculation of the partial derivative, we know that
Now we plug our terms into our chain rule and get:
$$ \frac{dJ}{dg}J(g)\frac{dg}{db}g(m,b) = 2g(m,b)-1 = -2(y - (mx + b)) $$
Ok, so now we have our two partial derivatives for
$$ \frac{dJ}{dm}J(m,b) = -2x(y - (mx + b )) $$ $$ \frac{dJ}{db}J(m,b) = -2(y - (mx + b)) $$
And as
$$ \frac{dJ}{dm}J(m,b) = -2x(y - \hat{y}) = -2x\epsilon$$ $$ \frac{dJ}{db}J(m,b) = -2*(y - \hat{y}) = -2\epsilon$$
Remember, error
= actual
- expected
, so we can replace
$$ \frac{dJ}{dm}J(m,b) = -2*\sum_{i=1}^n x(y_i - \hat{y}i) = -2*\sum{i=1}^n x_i*\epsilon_i$$ $$ \frac{dJ}{db}J(m,b) = -2*\sum_{i=1}^n(y_i - \hat{y}i) = -2*\sum{i=1}^n \epsilon_i$$
So that is what what we'll do to find the "best fit regression line." We'll start with an initial regression line with values of
In the context of gradient descent, we use these partial derivatives to take a step size. Remember that our step should be in the opposite direction of our partial derivatives as we are descending towards the minimum. So to take a step towards gradient descent we use the general formula of:
current_m
= old_m
$ - \frac{dJ}{dm}J(m,b)$
current_b
= old_b
$ - \frac{dJ}{db}J(m,b) $
or in the code that we just calculated:
current_m
= old_m
$ - (-2*\sum_{i=1}^n x_i*\epsilon_i )$
current_b
= old_b
$ - ( -2*\sum_{i=1}^n \epsilon_i )$
In the next lesson, we'll work through translating this technique, with use of our
In this section, we developed some intuition for why the gradient of a function is the direction of steepest ascent and the negative gradient of a function is the direction of steepest decent. Essentially, the gradient uses the partial derivatives to see what change will result from changes in the function's dimensions, and then moves in that direction weighted more towards the partial derivative with the larger magnitude.
We also practiced calculating some gradients, and ultimately calculated the gradient for our cost function. This gave us two formulas which tell us how to update our regression line so that it descends along our cost function and approaches a "best fit line".