[Machine Learning] - 3.1 - Classification and Representation

Posted on Edited on In Disqus:

Week 3 of professor Andrew Ng’s Machine Learning course on Coursera.

1. Classification and Representation

1.1. Sigmoid Function or Logistic Function

In a classification problem we could approach it with the linear regression algorithms we already know, by temporarily forgetting that y can only be 0 or 1. That approach doesn’t seem very good. The value of $latex h_0(x)$ has to lie between 0 and 1.

To solve this we transform the hypothesis function $latex h_0(x)$ so it satisfies $latex 0<=h_\theta(x)<=1$. We stuff $latex \theta^Tx$ into the Logistic Function:

$latex h_\theta (x) = g ( \theta^T x )$
$latex z = \theta^T x$
$latex g(z) = \dfrac{1}{1 + e^{-z}}$

the expression above has this graph:

The function g(z) can represent any real number between 0 and 1. $latex h_\theta(x)$ gives us the probability that the result = 1. For example, if $latex h_\theta(x) = 0.7$ it means the probability that the result = 1 is 70%. Conversely, the probability that the result = 0 is 30% (since the output can only take the 2 values 0 and 1).

$latex h_\theta(x) = P(y=1 | x ; \theta) = 1 - P(y=0 | x ; \theta)$
$latex P(y = 0 | x;\theta) + P(y = 1 | x ; \theta) = 1$

1.2. Decision Boundary

Based on the Logistic function above, we can `transform` our hypothesis function into this:

$latex h_\theta(x) \geq 0.5 \Rightarrow y = 1$
$latex h_\theta(x) < 0.5 \Rightarrow y = 0$

The function g(z) behaves like this:

$latex g(z) \geq 0.5$
when
$latex z \geq 0$

Remember that:

$latex z=0, e^{0}=1 \Rightarrow g(z)=1/2$
$latex z \to \infty, e^{-\infty} \to 0 \Rightarrow g(z)=1$
$latex z \to -\infty, e^{\infty}\to \infty \Rightarrow g(z)=0$

So we can write:

$latex h_\theta(x) = g(\theta^T x) \geq 0.5$
when
$latex \theta^T x \geq 0$

From the statements above we can write

$latex \theta^T x \geq 0 \Rightarrow y = 1$
$latex \theta^T x < 0 \Rightarrow y = 0$

The Decision Boundary is exactly the line separating the y = 0 region from the y = 1 region, produced by our hypothesis function

2. An example

Take this example:

$latex \theta = \begin{bmatrix}5\\ -1\\ 0\end{bmatrix}$
$latex y = 1 \; if \; 5 + (-1) x_1 + 0 x_2 \geq 0$
$latex 5 - x_1 \geq 0$
$latex - x_1 \geq -5$
$latex x_1 \leq 5$

so our graph looks like this

Note that depending on the hypothesis function and the theta parameters, the shape of the boundary line changes accordingly