I was looking at the derivation of the normal equation from here.
Now, the author has used the fact that $A^{\mathrm T} B = B^{\mathrm T} A$ to reach the step shown in the below image. Can anyone provide some information like, when is it true, or how we can prove it?
$\endgroup$ 1$$J(\theta) = ((X\theta)^{\mathrm T} -y^{\mathrm T})(X\theta -y)$$ $$J(\theta) = (X\theta)^{\mathrm T} X\theta -\color{blue}{(X\theta)^{\mathrm T} y \color{black}{-} y^{\mathrm T} (X\theta)} +y^{\mathrm T} y$$ Note that $X\theta$ is a vector, and so is $y$. So when we multiply one by another, it doesn't matter what the order is (as long as the dimensions work out). So we can further simplify: $$J(\theta) = \theta^{\mathrm T} X^{\mathrm T} X \theta -\color{blue}{2(X\theta)^{\mathrm T} y} +y^{\mathrm T} y$$
2 Answers
$\begingroup$Note that $(A^{\rm T} B)^{\rm T} = B^{\rm T} (A^{\rm T})^{\rm T} = B^{\rm T} A$, so if you are constraining $A^{\rm T} B = B^{\rm T} A$, that implies $(A^{\rm T} B)^{\rm T} = A^{\rm T} B$, meanining $A^{\rm T} B$ is symmetric.
$\endgroup$ 2 $\begingroup$Let's call $$C=AB$$ Then we have that $$C_{ij}=A_{ik}B_{kj}$$ We know that $$(M^T)_{ab}=M_{ba}$$ So we have that: $$(C^T)_{ij}=C_{ji}$$ $$(C^T)_{ij}=A_{jk}B_{ki}$$ $$(C^T)_{ij}=B_{ki}A_{jk}$$ $$(C^T)_{ij}=(B^T)_{ik}(A^T)_{kj}$$ And finally: $$C^T=B^TA^T$$ $$(AB)^T=B^TA^T$$
$\endgroup$ 1
