How to Differentiate a Hadamard Product

The Hadamard product, also called the elementwise product, multiplies corresponding entries rather than performing ordinary matrix multiplication. If $A$ and $B$ have the same dimensions, their Hadamard product is written

$$A \odot B,$$

with entries

$$(A \odot B)_{ij}=A_{ij}B_{ij}.$$

Its derivative follows the ordinary product rule entry by entry. The only extra issue is notation: for vectors, the derivative is usually a Jacobian matrix, while for matrices it is often most naturally written as a differential.

Product rule for vectors

Let $a(x),b(x)\in\mathbb{R}^n$ be differentiable vector-valued functions, and define

$$f(x)=a(x)\odot b(x).$$

The $i$th component is

$$f_i(x)=a_i(x)b_i(x).$$

Applying the scalar product rule to each component gives

$$df=da\odot b+a\odot db.$$

Equivalently, using diagonal matrices,

$$df=\operatorname{diag}(b)\,da+\operatorname{diag}(a)\,db.$$

This is the most useful general formula. It says that the change in the Hadamard product has two contributions: the change in the first factor, weighted by the second, and the change in the second factor, weighted by the first.

If $a$ and $b$ depend on $x\in\mathbb{R}^m$, let $J_a$ and $J_b$ denote their Jacobian matrices. Then the Jacobian of $f$ is

$$J_f(x)=\operatorname{diag}(b(x))J_a(x)+\operatorname{diag}(a(x))J_b(x).$$

This formula has dimensions $n\times m$, as it should.

One factor fixed

If $b$ is a constant vector and only $a(x)$ depends on $x$, then $db=0$. Therefore, since $db=0$,

$$df=da\odot b=\operatorname{diag}(b)\,da,$$

and hence

$$J_f(x)=\operatorname{diag}(b)J_a(x).$$

In particular, if $f(x)=c\odot x$ for a fixed vector $c$, then

$$J_f=\operatorname{diag}(c).$$

The diagonal matrix appears because each output component depends only on the corresponding input component.

Worked vector example

Let

$$a(x,y)=\begin{bmatrix}x^2\\ \sin y\\ x+y\end{bmatrix}, \qquad b(x,y)=\begin{bmatrix}e^x\\ y^2\\ x-y\end{bmatrix}.$$

Define $f=a\odot b$. Its components are

$$f(x,y)=\begin{bmatrix} x^2e^x\\ y^2\sin y\\ (x+y)(x-y) \end{bmatrix}.$$

Differentiate directly:

$$J_f(x,y)= \begin{bmatrix} e^x(x^2+2x) & 0\\ 0 & 2y\sin y+y^2\cos y\\ 2x & -2y \end{bmatrix}.$$

The same result follows from the Hadamard product rule. First,

$$J_a= \begin{bmatrix} 2x&0\\ 0&\cos y\\ 1&1 \end{bmatrix}, \qquad J_b= \begin{bmatrix} e^x&0\\ 0&2y\\ 1&-1 \end{bmatrix}.$$

Thus

$$J_f=\operatorname{diag}(b)J_a+\operatorname{diag}(a)J_b.$$

The first row, for example, is

$$e^x\begin{bmatrix}2x&0\end{bmatrix}+x^2\begin{bmatrix}e^x&0\end{bmatrix} =\begin{bmatrix}e^x(x^2+2x)&0\end{bmatrix}.$$

Matrix-valued functions

Let $A(t)$ and $B(t)$ be matrices of the same size, depending on a scalar $t$. Their Hadamard product satisfies

$$\frac{d}{dt}\bigl(A(t)\odot B(t)\bigr) =A'(t)\odot B(t)+A(t)\odot B'(t).$$

This is exactly the ordinary product rule applied to every matrix entry.

For example, let

$$A(t)=\begin{bmatrix}t^2&\sin t\\1&t\end{bmatrix}, \qquad B(t)=\begin{bmatrix}e^t&t\\t^2&\cos t\end{bmatrix}.$$

Then

$$A'(t)=\begin{bmatrix}2t&\cos t\\0&1\end{bmatrix}, \qquad B'(t)=\begin{bmatrix}e^t&1\\2t&-\sin t\end{bmatrix}.$$

Therefore,

$$\frac{d}{dt}(A\odot B) = \begin{bmatrix} e^t(t^2+2t)&t\cos t+\sin t\\ 2t&\cos t-t\sin t \end{bmatrix}.$$

For a matrix-valued function depending on a vector $x$, the same identity holds with differentials:

$$d(A\odot B)=dA\odot B+A\odot dB.$$

If individual partial derivatives are required, take the corresponding partial derivative of this expression.

Relation to ordinary matrix multiplication

The Hadamard product is not the same as ordinary matrix multiplication. For example,

$$\begin{bmatrix}1&2\\3&4\end{bmatrix} \odot \begin{bmatrix}5&6\\7&8\end{bmatrix} = \begin{bmatrix}5&12\\21&32\end{bmatrix},$$

whereas ordinary multiplication gives

$$\begin{bmatrix}1&2\\3&4\end{bmatrix} \begin{bmatrix}5&6\\7&8\end{bmatrix} = \begin{bmatrix}19&22\\43&50\end{bmatrix}.$$

Consequently, do not replace the Hadamard product rule with a rule for matrix multiplication. For ordinary matrix products, the derivative is $d(AB)=dA\,B+A\,dB$; for Hadamard products, it is $d(A\odot B)=dA\odot B+A\odot dB$.

Common mistakes

• Confusing $\odot$ with ordinary multiplication: The dimensions and derivative formulas are different.
• Dropping one product-rule term: If both factors depend on the variable, both derivatives must appear.
• Ignoring the Jacobian convention: Some authors use Jacobians with gradients arranged as rows rather than columns. The componentwise formula remains valid, but matrix orientations may be transposed.
• Using different-sized factors: A Hadamard product requires corresponding entries, so the factors must have the same shape unless a separate broadcasting convention has been specified.
• Treating a matrix derivative as one ordinary matrix: With several scalar variables, the derivative of a matrix-valued function is generally a third-order tensor, or is represented more conveniently by differentials or vectorization.