Machine Learning - 2.1 - Multiple Features

Posted on Edited on In Disqus:

Week 2 of professor Andrew Ng’s Machine Learning course on Coursera. In this part you’ll see linear regression extended to multiple input features, along with the best practices for doing linear regression.

See the other posts at Machine Learning Course Structure

1. Mutiple Features

Linear Regression with multiple features is also known as multivariate linear regression.

1.1. Notation

  • $latex x^{(i)}_j$ = the value of feature j in the i-th training example.
  • $latex x^{(i)}$ = the i-th input (feature) of the training example.
  • m = the number of training examples
  • n = the number of features

1.2. Hypothesis

So our hypothesis function is rewritten as:

$latex h_0(x) = \theta_0x_0 + \theta_1x_1 + … + \theta_nx_n$

with $latex x_0 = 1$.

1.3. Trick

Applying the matrix multiplication knowledge from the previous post, we get

$latex h_0(x) = \begin{bmatrix} \theta_0 & \theta_1 & … & \theta_n \end{bmatrix} \begin{bmatrix} x_0 \\ x_1 \\ … \\ x_n \end{bmatrix} = \theta^Tx$

That is the hypothesis function’s formula condensed into a matrix-by-vector multiplication.

2. Gradient Descent for Multiple Variables

The formula for the Gradient Descent algorithm is exactly the same as before. We just repeat it for n features.

Repeat until convergence:

$latex \theta_0 := \theta_0 - \alpha \frac{1}{m} \sum\limits_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)}) \cdot x_0^{(i)} \\ \theta_1 := \theta_1 - \alpha \frac{1}{m} \sum\limits_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)}) \cdot x_1^{(i)} \\ \theta_2 := \theta_2 - \alpha \frac{1}{m} \sum\limits_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)}) \cdot x_2^{(i)} \\ …$

or written another way:

$latex \theta_j := \theta_j - \alpha \frac{1}{m} \sum\limits_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)}) \cdot x_j^{(i)}$

with j:= 0…n

For $latex \theta_0$, $latex x^i_0 = 1$

3. Gradient Descent in Practice

3.1. Feature Scaling and Mean Normalization

When the features range over widely different intervals, the Gradient Descent algorithm usually takes a lot of time to find the result.

For example, say we have 2 features, house area and number of bedrooms:

[code lang=text] 200 < House area < 2000 1 < Number of bedrooms < 5 [/code]

If you plot the hypothesis function predicting the house price, you’ll see it is a bowl-shaped graph with a very sharp but flat bottom. This makes each gradient descent step stretch sideways without moving quickly toward the convergence point, increasing the total run time of the gradient descent algorithm.

plot

the picture is for illustration only ;)

We can speed gradient descent up by transforming the feature values so they sit in roughly the same range. The reason is that $latex \theta$ moves faster in a smaller range and, conversely, slower in a larger one.

Overall, we transform so that:

$latex -1 \leq x_{(i)} \leq 1$

or

$latex -0.5 \leq x_{(i)} \leq 0.5$

These are only examples; the goal is to make the features’ ranges as close to each other as possible.

The 2 techniques for doing this are Feature Scaling and Mean Normalization.

Feature Scaling is dividing the input by the range (max - min). Mean Normalization is input - the average of the input.

$latex x_i := \frac{x_i - \mu_i}{\delta_i}$

where:

  • $latex \mu_i$: the average of feature i
  • $latex \delta_i$: max - min, or the standard deviation

Max - min gives a very different result from the standard deviation.

3.2. Learning Rate

To determine the parameter $latex \alpha$ we can apply a few techniques:

  • Plot a graph with the x axis = the number of gradient descent iterations and the y axis = the value of $latex J(\theta)$. If $latex J(\theta)$ increases, you have to lower the value of $latex \alpha$ and start over.
  • Automatic convergence test: declare convergence when the value of $latex J(\theta)$ doesn’t change by more than E in one iteration, where E is some very small value. In practice, however, this E value is usually very hard to determine.

It has been proven that if the learning rate $latex \alpha$ is small enough, then the value of $latex J(\theta)$ decreases after every iteration.

In short: * If $latex \alpha$ is too small: gradient descent runs for a long time. * If $latex \alpha$ is too large: $latex J(\theta)$ may not decrease after each iteration -> no convergence.

4. Features and Polynomial Regression

We can improve the features and the shape of the hypothesis function in several ways.

One of those ways is combining features together. For example, when we have 2 features, length and width, we can combine them into area = length * width.

On top of that, we can’t always use a hypothesis function that is a straight line, especially when it doesn’t “fit” the data set. In that case we can transform it a bit, or bend it by raising the features to a power or taking their square root (or any other form you like).

For example:

$latex h_{\theta}(x) = \theta_0 + \theta_1x_1$

We can add a new feature by raising x to a power:

$latex h_{\theta}(x) = \theta_0 + \theta_1x_1 + \theta_2x_1^2$

or taking its square root:

$latex h_{\theta}(x) = \theta_0 + \theta_1x_1 + \theta_2\sqrt{x_1}$

One important thing: when you transform the features like this, their value ranges become very different from the original feature. At that point you’ll have to apply optimizations like the Feature Scaling and Mean Normalization described above to optimize the Gradient Descent algorithm.