Hadamard Product

The hadamard product is how some people might first think matrix multiplication works, which is wrong.

The hadamard product is denoted using $\boldsymbol{A} \odot \boldsymbol{B}$ and is also commonly known as the element-wise matrix multiplication which perfectly describes how it works. It is defined for two matrices $\boldsymbol{A}$ and $\boldsymbol{B}$ of the same dimensions and results in another matrix $\boldsymbol{C}$ that has the same dimension as the two input matrices so $\boldsymbol{A, B, C} \in \mathbb{R}^{N \times M}$ . So each element at $i,j$ in the resulting matrix is the product of the elements at $i,j$ of the two input matrices.

\boldsymbol{A} \odot \boldsymbol{B}= \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \odot \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \\ b_{31} & b_{32} & b_{33} \end{bmatrix} = \begin{bmatrix} a_{11}b_{11} & a_{12}b_{12} & a_{13}b_{13} \\ a_{21}b_{21} & a_{22}b_{22} & a_{23}b_{23} \\ a_{31}b_{31} & a_{32}b_{32} & a_{33}b_{33} \end{bmatrix}

Example

\begin{bmatrix} 1 & 4 & 7 \\ 2 & 3 & 8 \\ 9 & 5 & 6 \end{bmatrix} \odot \begin{bmatrix} 6 & 5 & 8 \\ 3 & 2 & 9 \\ 7 & 4 & 1 \end{bmatrix} = \begin{bmatrix} 6 & 20 & 56 \\ 6 & 6 & 72 \\ 63 & 20 & 6 \end{bmatrix}

Beginners in numpy often write the following lines of code to multiply two matrices

import numpy as np
 
A = np.array([[1,4,7],[2,3,8],[9,5,6]])
B = np.array([[6,5,8],[3,2,9],[7,4,1]])
A * B

    array([[ 6, 20, 56],
           [ 6,  6, 72],
           [63, 20,  6]])

But as might notice this is the hadamard product/element-wise multiplication. You would get the same result if you used the np.multiply(A,B) (opens in a new tab) function as this is the function that is called under the hood when using the * operator. If you really did want to multiply $\boldsymbol{A}$ with $\boldsymbol{B}$ you would need to use the @ operator or np.matmul(A,B) (opens in a new tab) function.

If you try to calculate a hadamard product in numpy where the matrices do not have the same shape you will probably get the following error:

ValueError: operands could not be broadcast together with shapes (2,2) (3,3)

So you can see that the hadamard product only works if A and B are of the same dimension/shape. However, from the error you can also see that there is an exception to this definition "if the matrices can be broadcast together". To find out more about broadcasting and how it works check out this page.

PCA - Principal Component Analysis Cross Product