Machine Learning - 1.4 - Matrices and Vectors

Posted on Edited on In Disqus:

The 4th post in the series on self-studying professor Andrew Ng’s Machine Learning course on Coursera.

In this post we’ll talk about Matrices and Vectors, along with their operations.

See the other posts at Machine Learning Course Structure

1. The notation

1.1. Matrix

A matrix is a 2-dimensional array (extendable to n dimensions).

$latex \begin{bmatrix}a & b & c \\d & e & f \\g & h & i \\j & k & l \end{bmatrix}$

The matrix above has 4 rows and 3 columns, denoted $latex R^{4x3}$.

1.2. Vector

A vector is a matrix with only 1 column and many rows.

$latex \begin{bmatrix} w \\ x \\ y \\ z \end{bmatrix}$

The vector above is a 4x1 matrix.

1.3. Notation

  • $latex A_{ij}$ is the element at row i and column j of the matrix.
  • A Vector A with ‘n’ rows is an n-dimensional Vector.
  • $latex v_{i}$ is the element at row i of the vector.

Normally vectors and matrices use indexes starting at 1. In most programming languages arrays usually start at element 0.

  • Matrices are usually denoted with capital letters, vectors with lowercase letters.
  • Scalar means an object that is a single value, not a vector or a matrix.
  • $latex \mathbb{R}$ denotes the set of real numbers.
  • $latex \mathbb{R}^n$ denotes the set of n-dimensional real-valued vectors.

2. The software

To quickly try out the math-related hypotheses and algorithms in Machine Learning, we can use a piece of software named Octave

You can run the code below in Octave or Matlab

[code lang=matlab] % The ; denotes we are going back to a new row. A = [1, 2, 3; 4, 5, 6; 7, 8, 9; 10, 11, 12]

% Initialize a vector v = [1;2;3]

% Get the dimension of the matrix A where m = rows and n = columns [m,n] = size(A)

% You could also store it this way dim_A = size(A)

% Get the dimension of the vector v dim_v = size(v)

% Now let’s index into the 2nd row 3rd column of matrix A A_23 = A(2,3) [/code]

3. The operations

3.1. Adding and multiplying a matrix by a real number

For addition and multiplication you simply add and multiply each element of the matrix.

$latex \begin{bmatrix} a & b \\ c & d \\ \end{bmatrix} +\begin{bmatrix} w & x \\ y & z \\ \end{bmatrix} =\begin{bmatrix} a+w & b+x \\ c+y & d+z \\ \end{bmatrix}$

Likewise for multiplication and division:

$latex \begin{bmatrix} a & b \\ c & d \\ \end{bmatrix} * x =\begin{bmatrix} a*x & b*x \\ c*x & d*x \\ \end{bmatrix}$

$latex \begin{bmatrix} a & b \\ c & d \\ \end{bmatrix} / x =\begin{bmatrix} a /x & b/x \\ c /x & d /x \\ \end{bmatrix}$

3.2. Multiplying a matrix by a vector

To multiply a matrix by a vector, multiply each element then add the results:

$latex \begin{bmatrix} a & b \\ c & d \\ e & f \end{bmatrix} *\begin{bmatrix} x \\ y \\ \end{bmatrix} =\begin{bmatrix} a*x + b*y \\ c*x + d*y \\ e*x + f*y\end{bmatrix}$

An m x n matrix multiplied by an n x 1 vector gives an m x 1 vector.

This matrix knowledge can be applied to the hypothesis function of linear regression:

We have the following set of input x values:

[code lang=text] x1 = 21 x2 = 30 x3 = 25 x4 = 22 [/code]

the hypothesis function:

$latex h_{0} = -40 + 0.25x$

To compute the predicted y values quickly, we can build the x matrix and the hypothesis vector and set up this operation:

$latex \begin{bmatrix} 1 & 21 \\ 1 & 30 \\ 1 & 25 \\ 1 & 22 \end{bmatrix} * \begin{bmatrix} -40 \\ 0.25 \end{bmatrix}$

This operation is easily computed in Octave

[code lang=matlab] A = [1,21;1,30;1,25;1,22] B = [-40;0.25]

mul_AB = A * B [/code]

The result:

[code lang=matlab] A = 1 21 1 30 1 25 1 22

B = -40.00000 0.25000

mul_AB = -34.750 -32.500 -33.750 -34.500 [/code]

3.3. Multiplying 2 matrices

Same as multiplying a matrix by a vector: you just split multiplying 2 matrices into several matrix-by-vector multiplications.

$latex \begin{bmatrix} a & b \\ c & d \\ e & f \end{bmatrix} *\begin{bmatrix} w & x \\ y & z \\ \end{bmatrix} =\begin{bmatrix} a*w + b*y & a*x + b*z \\ c*w + d*y & c*x + d*z \\ e*w + f*y & e*x + f*z\end{bmatrix}$

An m x n matrix multiplied by an n x o matrix gives an m x o matrix.

3.4. Properties of matrix multiplication

Multiplying 2 real numbers has some properties that don’t carry over to multiplying 2 matrices.

Let $latex A$ and $latex B$ be 2 matrices; then:

  • $latex A \times B \neq B \times A$
  • $latex A \times B \times C = A \times (B \times C) = (A \times B) \times C$

3.5. Identity Matrix

Any matrix multiplied by a matching Identity matrix gives itself back.

Notation: $latex I$ or $latex I_{n \times n}$

For example:

$latex I_{1 \times 1} = \begin{bmatrix} 1 \end{bmatrix}$

$latex I_{2 \times 2} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$

$latex I_{3 \times 3} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$

Note: $latex A \times I = I \times A$

4. Inverse and Transpose

4.1. Inverse

An example with real numbers

3 is a real number. We have: $latex 3 \times (3^{-1}) = 1$

In general: $latex a \times (a^{-1}) = 1$

For a matrix A:

$latex A \times A^{-1} = A^{-1} \times A = I$

A matrix of all 0 elements has no inverse

4.2. Transpose

Let B be the transpose matrix of A; then:

$latex A = \begin{bmatrix} a & b \\ c & d \\ e & f \end{bmatrix}$

$latex A^T = \begin{bmatrix} a & c & e \\ b & d & f \end{bmatrix}$

Or in other words:

$latex A_{ij} = A^T_{ji}$