矩阵求导简明方法：元素方法和矩阵方法

矩阵求导方法：元素方法和矩阵方法

本篇运算定义：相同角标代表求和

$I\in\mathbb{R}^{n*n}$

$记I=\delta_{ij}=\left\{\begin{array}{left}1 \ \ \ i=j\\0\ \ \ else \end{array}\right.$

$a\in\mathbb{C}^{1\ast m}$$,b\in\mathbb{C}^{m \ast 1}$

$向量a的第k个元素:=(a)_k=a_k \\向量b的第k个元素:=(b)_k=b_k$

$A\in\mathbb{C}^{m*n}$

$A的第i行:=a_{i.}\\ A的第j列:=a_{.j}$

$A\in\mathbb{C}^{m*n}$

$(A^T)_{ij}=(A)_{ji}=a_{ji}$

$A\in\mathbb{C}^{n*n}$

$tr(A)=\sum_ia_{ii}\\ 为了简便我们省略求和号记为:tr(A)=a_{ii}$

$A\in\mathbb{C}^{m\ast l}$$,B\in\mathbb{C}^{l \ast n}$

$(AB)_{ij}=\sum_k a_{ik}b_{kj}\\ 同样为了简便我们记为:AB=a_{ik}b_{kj}$

$x\in\mathbb{R}^{m \ast 1}$$,y\in\mathbb{R}^{m \ast 1}$

$\\内积:=x\cdot y=x^Ty=\sum_i x_iy_i(记为x_iy_i)\\ 外积:=x\circ y=xy^T,(x\circ y)_{ij}=x_iy_j$

$A\in\mathbb{R}^{m\ast m}$$,B\in\mathbb{R}^{n\ast n }$

$\\直和:=A\oplus B=\left[ \matrix{ A&O\\ O&B\\ } \right]\in\mathbb{R}^{(m+n)*(m+n)}$

$A,B\in\mathbb{R}^{m*n}$

$Hadamard积:=(A*B)_{ij}=a_{ij}b_{ij}\in \mathbb{R}^{m*n}(即对应元素相乘)$

$A\in \mathbb{R}^{m \ast n}$$,B\in \mathbb{R}^{p \ast q}$

$\\Kronecker积:=A\otimes B=\left[\matrix{ a_1B,a_2B,...,a_nB\\ } \right]=\left[\matrix{ a_{ij}B\\ } \right]_{i=1,j=1}^{m,n}=\left[\matrix{ a_{11}B&a_{12}B&\cdots&a_{1n}B\\a_{21}B&a_{22}B&\cdots&a_{2n}B\\\vdots&\vdots&\ddots&\vdots\\a_{m1}B&a_{m2}B&\cdots&a_{mn}B} \right]\in \mathbb{R^{mp*nq}}\\下面简便起见记为[a_{ij}B]$

$G\in\mathbb{R}^{p \ast n}$$,F\in\mathbb{R}^{q \ast n}$

$Khatri-Rao 积:=F\odot G=\left[\matrix{ f_1\otimes g_1,f_2\otimes g_2,...,f_n\otimes g_n\\ } \right]\in \mathbb{R}^{pq*n}$

向量化：

$A\in \mathbb{R}^{m*n}$

$列向量化:=vec(A)=[a_{11},\cdots a_{m1},\cdots,a_{1n},\cdots,a_{mn}]^T\in \mathbb{R}^{mn*1} \\简便起见记为\left[\matrix{ a_{.1}\\a_{.2}\\\vdots\\a_{.n}} \right](A的第j列:=a_{.j})\\ 第j列的第k个元素:=(a_{.j})_k=a_{kj}(也就是第k行第j列的元素)$

$K_{mn}\in \mathbb{R}^{mn*mn}$

$交换矩阵:=K_{mn}\\ 有K_{mn}vec(A)=vec(A^T)\\ K_{nm}vec(A^T)=vec(A)$

去向量化：

$unvec_{m,n}(a)=A_{m*n}\Longleftrightarrow vec(A_{m*n})=a_{mn*1}$

一些常用性质:(注意证明思路即可)

$tr(A^TB)=tr((a_{ik})^Tb_{kj})=tr(a_{ki}b_{kj})=a_{ki}b_{ki}$

$最后一步令i=j即可$

$(A+B)\oplus(C+D)=A\oplus B+C\oplus D$

$证明:左=\left[ \matrix{ A+B&O\\ O&C+D\\ } \right]=\left[ \matrix{ A&O\\ O&B\\ } \right]+\left[ \matrix{ C&O\\ O&D\\ } \right]=右$

$AB\oplus CD=(A\oplus C)(B\oplus D)$

$证明:左=\left[ \matrix{ AB&O\\ O&CD\\ } \right]=\left[ \matrix{ A&O\\ O&C\\ } \right]\left[ \matrix{ B&O\\ O&D\\ } \right]=右$

$A\oplus B\oplus C=(A\oplus B)\oplus C=A\oplus (B\oplus C)$

$证明:略$

$ (A\oplus B) \ast (C\oplus D)=(A \ast C)\oplus(B \ast D)—其中A,C\in \mathbb{R}^{m \ast m},B,D\in \mathbb{R}^{n \ast n}$

$证明:左=\left[ \matrix{ A&O\\ O&B\\ } \right]*\left[ \matrix{ C&O\\ O&D\\ } \right]=\left[ \matrix{ A*C&O\\ O&C*D\\ } \right]=右$

$(A+B) \ast (C+D)=A \ast C+A \ast D+B \ast C+B \ast D$

$证明：略$

$tr(A^T(B \ast C))=tr((A^T \ast B^T)C)$

$证明：首先(A^T(B*C))_{ij}=(A^T)_{ik}(B*C)_{kj}=a_{ki}(b_{kj}c_{kj})=a_{ki}b_{kj}c_{kj}\\同理我们有：((A^T*B^T)C)_{ij}=(A^T*B^T)_{ik}C_{kj}=(A^T)_{ik}(B^T)_{ik}C_{kj}=a_{ki}b_{ki}c_{kj}\\ 则左边=tr(A^T(B*C))=a_{ki}b_{ki}c_{ki}(在上面得到的式子里面令j=i即可）\\右边=tr((A^T*B^T)C)=a_{ki}b_{ki}c_{ki}(同理令j=i即可)，比较左边和右边证毕$

$AB\otimes CD=(A\otimes C)(B\otimes D)——其中A\in \mathbb{R}^{m \ast n},B\in \mathbb{R}^{n \ast p},C\in \mathbb{R}^{s \ast r},D\in \mathbb{R}^{r \ast t}$

$证明：左边=[(AB)_{ij}CD]=[(A_{ik}B_{kj})CD]=[A_{ik}C][B_{kj}D]=右边$

$A\otimes (B+C)=A\otimes B+A\otimes C$

$证明：左边=[a_{ij}(B+C)]=[a_{ij}B]+[a_{ij}C]$

$(A+ B)\otimes(C+D)=A\otimes C+A\otimes D+B\otimes C+B\otimes D$

$证明：左边=(A+B)\otimes C+(A+B)\otimes D=[(a_{ij}+b_{ij})C]+[(a_{ij}+b_{ij})D]=[a_{ij}C]+...=右边$

$(A\otimes B)^T=A^T\otimes B^T$

$证明：左边=[a_{ij}B]^T=[a_{ji}B^T]=右边$

$A\otimes (B\otimes C)=(A\otimes B)\otimes C$

$证明：按定义证明即可$

$K_{mn}^T=K_{mn}^{-1}=K_{nm}$

$证明：K_{nm}K_{mn}vec(A)=K_{nm}vec(A^T)=vec(A)\Rightarrow K_{mn}^{-1}=K_{nm}\\转置从定义考虑即可$

$K_{n1}=K_{1n}=I_n$

$K_{pm}(A\otimes B)K_{nq}=B\otimes A——其中A\in \mathbb{R}^{m\ast n},B\in \mathbb{R}^{p\ast q}$

$证明：较为麻烦，此处从略，主要目的是使A\otimes B交换位置$

$tr(A^TB)=vec(A)^Tvec(B)$

$证明：右边=vec(A)^Tvec(B)=[a_{.1}^T,a_{.2}^T,...,a_{.n}^T]\left[\matrix{ b_{.1}\\b_{.2}\\\vdots\\b_{.n}} \right]=a_{.i}^Tb_{.i}=(a_{.i})_k(b_{i})_k=a_{ki}b_{ki}=左边$

$tr(ABC)=vec(A^T)^T(I_p\otimes B)vec(C)$

$证明：右边=[a_{1.},a_{2.},...,a_{n.}][\delta_{ij}B]\left[\matrix{ c_{.1}\\c_{.2}\\\vdots\\c_{.n}} \right]=(a_{k.})([\delta_{ij}B])_{i=k,j=t}(c_{.t})=\delta_{kt}(a_{k.})B(c_{.t})\\=(a_{t.})B(c_{.t})=(a_{t.})_iB_{ij}(c_{.t})_j=a_{ti}b_{ij}c_{jt}=左边$

$tr(ABCD)=vec(D)^T(A\otimes C^T)vec(B^T)$

$证明：右边=[d_{.1}^T,d_{.2}^T,...,d_{.n}^T][a_{ij}C^T]\left[\matrix{ b_{1.}^T\\b_{2.}^T\\\vdots\\b_{n.}^T} \right]=(d_{.k}^T)([a_{ij}C^T])_{i=k,j=t}(b_{t.}^T)=a_{kt}(d_{.k}^T)(C^T)(b_{t.}^T)\\=a_{kt}(d_{.k}^T)_i(C^T)_{ij}(b_{t.}^T)_j=a_{kt}d_{ik}c_{ji}b_{tj}=a_{kt}b_{tj}c_{ji}d_{ik}=左边$

$vec(A\ast B)=vec(A)\ast vec(B)=diag(vec(A))vec(B)—-diag(a)表示向量以a每个元素为对角元素的对角阵$

$证明：由定义易得$

$\bigstar向量化操作，乘积矩阵到向量的纽带$

$vec(ABC)=(C^T\otimes A)vec(B)$

$证明：(ABC)_{ij}=a_{ik}b_{kt}c_{tj}\Rightarrow(ABC)_{.j}=a_{.k}b_{kt}c_{tj}\\ 右边=[(C^T)_{ij}A]\left[\matrix{ b_{.1}\\b_{.2}\\\vdots\\b_{.n}} \right]=[c_{ji}A]\left[\matrix{ b_{.1}\\b_{.2}\\\vdots\\b_{.n}} \right]=\left[\matrix{ c_{j1}Ab_{.j}\\c_{j2}Ab_{.j}\\\vdots\\c_{ji}Ab_{.j}\\\vdots} \right]=\left[\matrix{ c_{j1}a_{.k}b_{kj}\\c_{j2}a_{.k}b_{kj}\\\vdots\\c_{ji}a_{.k}b_{kj}\\\vdots} \right]=\left[\matrix{ (ABC)_{.1}\\(ABC)_{.2}\\\vdots\\(ABC)_{.i}\\\vdots} \right]=vec(ABC)=左边$

$vec(AC)=(I_p\otimes A)vec(C)=(C^T\otimes I_m)vec(A)——其中A\in \mathbb{R}^{m\ast p},B\in \mathbb{R}^{p\ast q},C\in \mathbb{R}^{q\ast n}$

$证明：同上$

矩阵求导部分

对元素求导方法

与我们学过的求导定义类似,求导是要求出目标量随着一个变量的变化，我们这边把变量看成元素

举个例子

$Ax=\left[ \matrix{ a_{11}&a_{12}\\ a_{21}&a_{22}\\ } \right]x=b,我们咋么定义b相对于A的变化呢$

回想我们所知道的f(x)对x求导的定义

很自然想到

$\left[ \matrix{ a_{11}+\vartriangle a_{11}&a_{12}\\ a_{21}&a_{22}\\ } \right]x=b+\vartriangle b,更细致地\left[ \matrix{ a_{11}+\vartriangle a_{11}&a_{12}\\ a_{21}&a_{22}\\ } \right]\left[ \matrix{ x_1\\ x_2} \right]=\left[ \matrix{ b_1+\vartriangle b_1\\ b_2+\vartriangle b_2} \right]\\计算知：\vartriangle b_1=(\vartriangle a_{11})x_1,\vartriangle b_2=0\\将这一个小的元素扰动引起的量变化定义为\frac{\partial b_1}{\partial a_{11}}\\则\frac{\partial b_1}{\partial a_{11}}=lim_{\vartriangle a_{11}->0}\frac{\vartriangle b_1}{\vartriangle a_{11}}=x_1 \\$

而实际上

$b_1=a_{11}x_1+a_{12}x_2,\frac{\partial b_1}{\partial a_{11}}=x_1,计算结果与上面一致！\\当然我们要求此处\frac{\partial a_{12}}{\partial a_{11}}=0,其实这也是非常自然的，因为元素变化之间并没有关联性$

由上面的结果，我们发现只要我们把定义放在元素对元素上，就与我们之前学过的求导定义一致了！

我们之前所学的关于求导的技术都可以移植过来，比如链式法则等

计算时我们要求

$\frac{\partial x_{ijk...}}{\partial x_{rst...}}=\delta_{ir}\delta_{js}\delta_{kt}即可$

例1、$f(x)=x^TAx$

$f(x)=x_iA_{ij}x_j\\ \frac{\partial f(x)}{\partial x_{r}}=\frac{\partial x_{i}}{\partial x_{r}}a_{ij}x_{j}+x_ia_{ij}\frac{\partial x_{j}}{\partial x_{r}}=\delta_{ir}a_{ij}x_j+x_ia_{ij}\delta_{jr}=a_{rj}x_j+x_ia_{ir}$

例2、$X\in \mathbb{R}^{m\ast n},a,b\in \mathbb{R}^{n\ast 1},f(X)=a^TXX^Tb$

$f(X)=a_ix_{ij}(x^T)_{jk}b_k=a_ix_{ij}x_{kj}b_k \\\frac{\partial f(X)}{\partial x_{rs}}=a_i\frac{\partial x_{ij}}{\partial x_{rs}}x_{kj}b_k+a_ix_{ij}\frac{\partial x_{kj}}{\partial x_{rs}}b_k=a_i\delta_{ir}\delta_{js}x_{kj}b_k+a_ix_{ij}\delta_{kr}\delta_{js}b_k=a_rx_{ks}b_k+a_ix_{is}b_{r}$

接下来的问题是我们能不能定义一套东西，能将这些元素变回矩阵形式

对矩阵定义求导

$紧接着我们上面得到的,排列如下\\\left[ \matrix{ \frac{\partial b_1}{\partial a_{11}}&\frac{\partial b_1}{\partial a_{12}}& \frac{\partial b_1}{\partial a_{21}}&\frac{\partial b_1}{\partial a_{22}}\\ } \right],此时则\frac{\partial b_1}{\partial a_{11}}= lim_{\vartriangle a_{11}->0}\frac{\vartriangle b_1}{\vartriangle a_{11}}= x_1 \\ 此时我们发现\vartriangle b_1=\left[ \matrix{ \frac{\partial b_1}{\partial a_{11}}&\frac{\partial b_1}{\partial a_{12}}& \frac{\partial b_1}{\partial a_{21}}&\frac{\partial b_1}{\partial a_{22}}\\ } \right]\left[ \matrix{ \vartriangle a_{11}\\ \vartriangle a_{12}\\ \vartriangle a_{21}\\ \vartriangle a_{22}\\ } \right] \\ \left[ \matrix{ \vartriangle b_1\\ \vartriangle b_2 } \right]=\left[ \matrix{ \frac{\partial b_1}{\partial a_{11}}&\frac{\partial b_1}{\partial a_{12}}& \frac{\partial b_1}{\partial a_{21}}&\frac{\partial b_1}{\partial a_{22}}\\ \frac{\partial b_2}{\partial a_{11}}&\frac{\partial b_2}{\partial a_{12}}& \frac{\partial b_1}{\partial a_{21}}&\frac{\partial b_1}{\partial a_{22}} } \right]\left[ \matrix{ \vartriangle a_{11}\\ \vartriangle a_{12}\\ \vartriangle a_{21}\\ \vartriangle a_{22}\\ } \right]\\矩阵\left[ \matrix{ \frac{\partial b_1}{\partial a_{11}}&\frac{\partial b_1}{\partial a_{12}}& \frac{\partial b_1}{\partial a_{21}}&\frac{\partial b_1}{\partial a_{22}}\\ \frac{\partial b_2}{\partial a_{11}}&\frac{\partial b_2}{\partial a_{12}}& \frac{\partial b_1}{\partial a_{21}}&\frac{\partial b_1}{\partial a_{22}} } \right]正是我们想要的，定义其为(\bigtriangledown_Ab)^T \\采用这一套定义我们可以很好地与之前我们元素求导方法对应$

定义：$X\in \mathbb{R}^{m\ast n},F(X)\in \mathbb{R}^{p\ast q},\bigtriangledown_XF(X)\in \mathbb{R}^{pq\ast mn}$

$d(vec(F(X)))=(\bigtriangledown_XF(X))^Td(vec(X))$

对上面的例子我们恢复成矩阵形式

例1、$f(x)=x^TAx$

但对一般情况，变回矩阵较为麻烦且没有必要，我们可以通过矩阵的方法直接对矩阵求导

矩阵求导方法

1.取微元 2.向量化

取微元常见性质：

$\left\{ \begin{array}{} d(trX)=tr(dX)\\ d(UVW)=(dU)VW+U(dV)W+UV(dW)\\ d|X|=|X|tr(X^{-1}dX)\\ d(U\otimes V)=(dU)\otimes V+U\otimes (dV)\\ d(U*V)=(dU)*V+U*(dV)\\ d(vec(X))=vec(dX)\\dlogX=X^{-1}dX\\d(X^{-1})=-X^{-1}(dX)X^{-1}\\d(F(X))=F'(X)*(dX)\end{array}\right.$

例2、$X\in \mathbb{R}^{m\ast n},a,b\in \mathbb{R}^{n\ast 1},f(X)=a^TXX^Tb$（利用$vec(ABC)=(C^T\otimes A)vec(B)$）

$取微元:df(X)=d(a^TXX^Tb)=a^T(dX)X^Tb+a^TX(dX^T)b\\ 向量化:vec(df(X))=vec(a^T(dX)X^Tb)+vec(a^TX(dX)^Tb)=((X^Tb)^T\otimes a^T)vec(dX)+(b^T\otimes a^TX)vec(dX)^T\\=((b^TX)\otimes a^T)d(vec(X))+(b^T\otimes a^TX)K_{mn}vec(dX)\\\bigtriangledown_Xf(X)=(X^Tb)\otimes a+K_{nm}(b\otimes X^Ta)$

例3、logistic回归

$l=-yx^Tw+log(1+exp(x^Tw)),y=\{0,1\},x,w\in \mathbb{R}^{n*1}$

$dl=-yx^Tdw+\frac{exp(x^Tw)*(x^Tdw)}{1+exp(x^Tw)}=-yx^Tdw+\sigma(x^Tw)x^Tdw---\sigma(a)=\frac{exp(a)}{1+exp(a)}\\ \bigtriangledown_wl=-xy+\sigma(x^Tw)x\\d(\bigtriangledown_wl)=\sigma’(x^Tw)*(x^Tdw)x=(\sigma'(x^Tw)x^T(dw))x=\sigma'(x^Tw)xx^T(dw)\\vec(d(\bigtriangledown_wl))=\sigma'(x^Tw)xx^Tvec(dw)\\\bigtriangledown_w^2l=\sigma'(x^Tw)xx^T$

例4、多元logistic回归

$l=-y^Tlog(softmax(Wx))=-y^TWx+log(1^Texp(Wx)),y\in\mathbb{R}^{m\ast 1},x\in \mathbb{R}^{n\ast 1},W\in \mathbb{R}^{m\ast n}-softmax(a)=\frac{exp(a)}{1^Texp(a)}$

$dl=-y^T(dW)x+\frac{1^T(exp(Wx)*((dW)x))}{1^Texp(Wx)}=-y^T(dW)x+\frac{exp(Wx)^T(dW)x}{1^Texp(Wx)}\\vec(dl)=-(x^T\otimes y^T)d(vec(W))+\frac{(x^T\otimes exp(Wx)^T)(dvec(W))}{1^Texp(Wx)}\\ \bigtriangledown_Wl=-(x\otimes y)+\frac{x\otimes exp(Wx)}{1^Texp(Wx)}\\d(\bigtriangledown_Wl)=\frac{x\otimes (exp(Wx)*((dW)x))}{1^Texp(Wx)}+\frac{(x\otimes exp(Wx))(exp(Wx)^T(dW)x)}{(1^Texp(Wx))^2}\\ d(vec(\bigtriangledown_Wl))=\frac{x\otimes (diag(exp(Wx))vec((dW)x))}{1^Texp(Wx)}+\frac{(x\otimes exp(Wx))((x^T\otimes exp(Wx)^T)(dvec(W))}{(1^Texp(Wx))^2}\\=\frac{x\otimes (diag(exp(Wx))(x^T\otimes I_m)dvec(W)}{1^Texp(Wx)}+\frac{(xx^T\otimes exp(Wx)exp(Wx)^T)(dvec(W))}{(1^Texp(Wx))^2}\\=\frac{xx^T\otimes (diag(exp(Wx))dvec(W)}{1^Texp(Wx)}+\frac{(xx^T\otimes exp(Wx)exp(Wx)^T)dvec(W)}{(1^Texp(Wx))^2}\\\bigtriangledown_W^2l=xx^T\otimes(diag(softmax(Wx))-softmax(Wx)softmax(Wx)^T)=xx^T\otimes D(x)\\注vec(A*B)=vec(A)*vec(B)=diag(vec(A))vec(B)---diag(a)表示向量以a每个元素为对角元素的对角阵$