Eigenvalues and Eigenvectors

Before we talk about eigenvalues and eigenvectors let us just remind ourselves that vectors can be transformed using matrices. For example we can rotate a vector using the rotation matrix:

Or we can use a matrix to scale a vector:

Now let us go back to eigenvalues and eigenvectors. An eigenvector $\boldsymbol{v}$ of a square matrix $\boldsymbol{A}$ is defined as a non-zero vector such that the multiplication with $\boldsymbol{A}$ only changes the scale of the vector it does not change the direction. The scalar $\lambda$ is called the eigenvalue.

Because there would be an infinite amount of solutions we limit the magnitude of the vector to $\parallel\boldsymbol{v}\parallel_2=1$.

Let us look at an example of how to calculate the eigenvector and eigenvalue of

For this we can rewrite the problem and solve the following equations:

For there to be a solution where $\boldsymbol{v}$ is non-zero then the following must be true and which then must lead to the characteristic polynomial of $\boldsymbol{A}$. Solving the characteristic polynomial equaling 0 we can get between 0 and $n$ eigenvalues with $n$ being the number of dimensions of $\boldsymbol{A} \in \mathbb{R}^{n \times n}$:

Now that we have the eigenvalues all we need to do is calculate the eigenvectors corresponding to each eigenvalue.

So we know $v_1 = -v_2$ since we constrict ourselves to vectors with a magnitude of 1 so $\sqrt{v_1^2 + (-v_1)^2}=1$ we get for eigenvalue $\lambda_1=-1$ the eigenvector

We can also calculate this using the following numpy code:

import numpy as np
 
A = np.array([[0, 1], [-2, -3]])
e_values, e_vectors = np.linalg.eig(A)
print(f"Eigenvalues: {e_values}")
print(f"Eigenvectors: {e_vectors}")

    Eigenvalues: [-1. -2.]
    Eigenvectors: [[ 0.70710678 -0.4472136 ]
     [-0.70710678  0.89442719]]

Properties

We can use the eigenvalues and eigenvectors of the matrix $\boldsymbol{A}$ to find out a lot about it

• The trace of $\boldsymbol{A}$ is the sum of its eigenvalues $tr(\boldsymbol{A})=\sum_{i=1}^{n}{\lambda_i}$.
• The determinant of $\boldsymbol{A}$ is the product of its eigenvalues $det(\boldsymbol{A})=\prod_{i=1}^{n}{\lambda_i}$.
• The rank of $\boldsymbol{A}$ is amount of non-zero eigenvalues.

print(f"Trace: {np.trace(A)}")
print(f"Determinant: {np.linalg.det(A)}")
print(f"Rank: {np.linalg.matrix_rank(A)}")

    Trace: -3
    Determinant: 2.0
    Rank: 2

If $\boldsymbol{A}$ is a diagonal matrix then the eigenvalues are just the diagonal elements.

D = np.diag([1, 2, 3])
e_values, e_vectors = np.linalg.eig(D)
print(f"Eigenvalues: {e_values}")
print(f"Eigenvectors: {e_vectors}")

    Eigenvalues: [1. 2. 3.]
    Eigenvectors: [[1. 0. 0.]
     [0. 1. 0.]
     [0. 0. 1.]]

Trick for 2 by 2 Matrices

As presented in this video by 3Blue1Brown (opens in a new tab) there is a cool formula that can be used to calculate the eigenvalues of a $2 \times 2$ matrix such as $\boldsymbol{A}=\begin{bmatrix}a & b \\ c & d\end{bmatrix}$. It rests upon two properties that have already been mentioned above:

• The trace of $\boldsymbol{A}$ is the sum of its eigenvalues $tr(\boldsymbol{A})=\sum_{i=1}^{n}{\lambda_i}$. So in other words $a + d = \lambda_1 + \lambda_2$. We can also reform this to get the mean value of the two eigenvalues: $\frac{1}{2}tr(\boldsymbol{A})=\frac{a+d}{2}=\frac{\lambda_1 + \lambda_2}{2}=m$
• The determinant of $\boldsymbol{A}$ is the product of its eigenvalues $det(\boldsymbol{A})=\prod_{i=1}^{n}{\lambda_i}$. So in other words $ad - bc = \lambda_1 \cdot \lambda_2 = p$.

Eigendecomposition

The eigendecomposition is a way to split up square matrices into 3 matrices which can be useful in many applications. Eigendecomposition can be pretty easily derived from the above since it lead to the following equations:

Instead of holding this information in three separate equations we can combine them to one equation using matrices. We combine the eigenvectors to a matrix where each column is a eigenvector and we create a diagonal matrix with the eigenvalues (by convention in order of small to large):

If $\boldsymbol{A}$ is a symmetric matrix then $\boldsymbol{Q}$ is guaranteed to be an orthogonal matrix because it is the eigenvectors of $\boldsymbol{A}$ concatenated. Because $\boldsymbol{Q}$ is orthogonal $\boldsymbol{Q}^{-1} = \boldsymbol{Q}^T$ which leads to the formula being simplified to

A = np.array([[5, 2, 0], [2, 5, 0], [4, -1, 4]])
A

    array([[ 5,  2,  0],
           [ 2,  5,  0],
           [ 4, -1,  4]])

X = np.array([[1, 0, -1], [1, 0, 1], [1, 1, 5]])
Lambda = np.diag([7, 4, 3])
inverse = np.linalg.inv(X)
np.matmul(np.matmul(X, Lambda), inverse)

    array([[ 5.,  2.,  0.],
           [ 2.,  5.,  0.],
           [ 4., -1.,  4.]])