Matrix Calculus
In this page we introduce a differential based method for vector and matrix derivatives (matrix calculus), which only needs a few simple rules to derive most matrix derivatives. This method is useful and well established in mathematics, however few documents clearly or detailedly describe it. Therefore, we make this page aiming at comprehensive introduction of matrix calculus via differentials.
* If you want results only, there is an awesome online tool Matrix Calculus. If you want "how to", let's get started.
0. Notation
, 
and 
denote 
, 
and 
respectively.- The first half of the alphabet
denote constants, and the second half 
denote variables. 
denotes matrix transpose, 
is the trace, and 
is the determinant.
is the Kronecker product and 
is the Hadamard product.- Here we use numerator layout, while the online tool Matrix Calculus seems to use mixed layout. Please refer to Wiki - Matrix Calculus - Layout Conventions for the layout definitions, and keep in mind that different layouts lead to different results.
1. Matrix Calculus via Differentials
1.1 Differential Identities
- Identities 1
- Identities 2
- Identities 1.5 - total differential. Actually, all identities 1 are the matrix form of total differential in eq. (19).
1.2 Deriving Matrix Derivatives
To derive a matrix derivative, we repeat using the identities 1 (the process is actually a chain rule) assisted by identities 2.
1.2.1 Abstract examples: repeat identities 1
E.g. 1, 
,
finally from eq. (2), we get 
.
E.g. 2, 
,
finally from eq. (3), we get 
.
E.g. 3, 
,
finally from eq. (1), we get 
.
E.g. 4, 
,
finally from eq. (5), we get 
.
1.2.2 Actual examples: assisted by identities 2
E.g. 1, 
,
finally from eq. (2), we get 
.
E.g. 2, 
,
finally from eq. (3), we get 
. From line 3 to 4, we use the conclusion of 
, that is to say, we can derive more complicated matrix derivatives by properly utilizing the existing ones. From line 6 to 7, we use 
to introduce the 
in order to use eq. (3) later, which is common in scalar-by-matrix derivatives.
E.g. 3, 
,
finally from eq. (3), we get 
.
E.g. 4, 
,
finally from eq. (3), we get 
.
E.g. 5, prove 
,
since,
then,
therefore,
* See examples.md for more examples.
2. Conclusion
Now, if we fully understand the core mind of the above examples, I believe we can derive most matrix derivatives in Wiki - Matrix Calculus by ourself. Please correct me if there is any mistake, and raise issues to request the detailed steps of computing the matrix derivatives that you are interested in.