Optimization criteria

• Let $M^{*}$ denote the Hermitian (conjugate) transpose of a matrix $M$. We choose $\left\{A_{k}\right\}$ such that the sum-of-squares error:
  

  $$J(\left\{A_{k}\right\}) = \sum{e_{n}e^{*}_{n}}$$ (2)

  
  is minimal, in the sense indicated by positive semidefinite (PSD) matrices: If $J$ is the cost due to an optimal set of model parameters and $J'$ is the cost due to another set then $J' - J$ is PSD; i.e. all quadratic forms $x(J'-J)x^{*}$ are nonnegative. Practical consequences of this ``PSD-minimal'' criterion are:
  1. We minimize the sum-of-squares of any linear combination of error components: Let $w$ be a weight vector, and define the cost due to $\left\{A_{k}\right\}$:
     

     $$j(w) = \sum \vert w^{*}e_{n}\vert^{2}$$

     $$= \sum w^{*} e_{n}e^{*}_{n} w$$

     $$= w^{*} \left(\sum e_{n}e^{*}_{n}\right) w$$

     $$= w^{*} J w$$ (3)

     
     

     Similarly, let $j'(w) = w^{*} J' w$ be the cost due to $\left\{{A'}_{k}\right\}$.

     Since $J' - J$ is PSD, it follows that $w^{*}(J'-J) w \ge 0$, and by (3) it follows: $j'(w) \ge j(w)$.

  2. We minimize the sum of all weighted error norms, provided the weight matrix is Hermitian and PSD. Let $W$ be that $(m \times m)$ weight matrix, and define the cost due to $\left\{A_{k}\right\}$:
     

     $$j(W) = \sum e^{*}_{n}W e_{n}$$

     $$= \sum \vert\vert W^{1/2}e_{n}\vert\vert^{2}$$

     $$= \mathrm{Tr}\left[W^{1/2}\left(\sum{e_{n}e^{*}_{n}}\right)W^{*/2}\right]$$

     $$= \mathrm{Tr}(W^{1/2}JW^{*/2})$$ (4)

     
     Similarly $j'(W) = \mathrm{Tr}(W^{1/2}J'W^{*/2})$ is the cost due to $\left\{{A'}_{k}\right\}$.

     The existence of $W^{1/2}$ such that $W = W^{*/2}W^{1/2}$ is justified by the Hermitian/PSD properties of $W$. If $J' - J$ is PSD, $x^{*}(J'-J)x \ge 0$ for any vector $x$. Given a vector $y$, choose $x = W^{1/2}y$. Substituting, we find $y^{*}(W^{1/2} J' W^{*/2} - W^{1/2} J W^{*/2}) y \ge 0$; because $y$ is arbitrary, $(W^{1/2} J' W^{*/2} - W^{1/2} J W^{*/2})$ is PSD. So we have shown if $J$ is PSD-minimal then so is $W^{1/2}JW^{*/2}$. Finally, a PSD matrix has a nonnegative trace. This follows because the trace of a matrix is the sum of its eigenvalues and all eigenvalues of a PSD matrix are nonnegative. Using this fact and (4) it follows that $j'(W) - j(W) \ge 0$ for all $W$, as was to be shown.

• The first criterion (3) follows from the second (4) by setting $W = ww^{*}$, but it's easier to prove directly.