Linear Algebra
Contents
Doesnt belong here
$\mathbf{A}$ is the variance/covariance structure on the original axes; the eigenvectors define a new set of axes chosen so that covariance between them is eliminated entirely, and the eigenvalues are what's left — the variance along each of those now-uncorrelated axes.
Eigenvectors
Eigenvectors - vectors that, when acted on by a given matrix (linear transformation), only get scaled, not rotated: $\mathbf{A}\vec{v} = \lambda \vec{v}$, where $\lambda$ is the corresponding eigenvalue.
Dimensions: $\mathbf{A} \in \mathbb{R}^{n \times n}$ (square — only square matrices have eigenvectors in this sense), $\vec{v} \in \mathbb{R}^{n}$, and $\lambda \in \mathbb{R}$ is a scalar, so both sides of the equation are $n \times 1$ vectors.
Note: $\vec{v}$ is a column vector ($n \times 1$), not a row vector.
This isn't obvious from "$\vec{v} \in \mathbb{R}^n$" alone — it follows from the product $\mathbf{A}\vec{v}$: for a matrix-vector product to be defined with $\mathbf{A}$ being $n \times n$, $\vec{v}$ must be $n \times 1$.
Note: $\vec{v}$ here is a single eigenvector paired with its own eigenvalue $\lambda$. Since $\mathbf{A}$ here is a covariance matrix (real, symmetric), the spectral theorem guarantees it has exactly $n$ such pairs, with real eigenvalues and mutually orthogonal eigenvectors.
The single-vector form $\mathbf{A}\vec{v} = \lambda \vec{v}$ is called the eigenvalue equation. The stacked form below, which gathers all $n$ pairs together, is called the eigendecomposition (solved as $\mathbf{A} = \mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^{-1}$, it's also called diagonalization).
Stacked together, these are written as $\mathbf{A}\mathbf{V} = \mathbf{V}\boldsymbol{\Lambda}$, where $\mathbf{V} \in \mathbb{R}^{n \times n}$ has the eigenvectors as columns and $\boldsymbol{\Lambda} \in \mathbb{R}^{n \times n}$ is diagonal with the corresponding eigenvalues.
Note: $\mathbf{V}$ comes before $\boldsymbol{\Lambda}$ here, whereas $\lambda$ came after $\vec{v}$ in the eigenvalue equation.
This isn't arbitrary — $\lambda$ is a scalar, so $\lambda \vec{v} = \vec{v}\lambda$ either way. $\boldsymbol{\Lambda}$ is a matrix, though, and matrix multiplication doesn't commute. $\mathbf{V}\boldsymbol{\Lambda}$ scales each column of $\mathbf{V}$ by its own $\lambda_i$ (column $i$ of $\mathbf{V}\boldsymbol{\Lambda}$ works out to $\lambda_i \vec{v}_i$), matching column $i$ of $\mathbf{A}\mathbf{V}$, which is $\mathbf{A}\vec{v}_i$. Writing $\boldsymbol{\Lambda}\mathbf{V}$ instead would scale the rows of $\mathbf{V}$, which is not what's needed.
In PCA, $\mathbf{A}$ is the $N \times N$ covariance matrix of $\widetilde{\mathbf{Y}}_c$, and its eigenvectors point along the directions of maximum variance in the data, becoming the principal component axes.
Orthogonality
Orthogonal means perpendicular. Two vectors meet at a 90 degree angle.
Algebraically two vectors are orthogonal if their dot product equals zero.
$ \vec{u} = (u_1, u_2, \ldots, u_n), \qquad \vec{v} = (v_1, v_2, \ldots, v_n) $
Orthonormal
Orthonormal means orthogonal and unit length: a set of vectors is orthonormal if every pair is orthogonal (dot product zero) and every vector has length (norm) 1.
For a matrix $\mathbf{W}$ whose columns are orthonormal, this gives $\mathbf{W}^\top\mathbf{W} = \mathbf{I}$ — and if $\mathbf{W}$ is square, also $\mathbf{W}\mathbf{W}^\top = \mathbf{I}$, i.e. $\mathbf{W}^{-1} = \mathbf{W}^\top$ (the transpose serves as the inverse, no separate matrix inversion needed). This is the property invoked throughout — e.g. why the eigenvectors of a symmetric matrix like $\mathbf{A}$ are orthonormal (spectral theorem), and why $\widetilde{\mathbf{Y}}_c\mathbf{W}\mathbf{W}^\top = \widetilde{\mathbf{Y}}_c$ holds exactly when all $N$ components are kept.
Unsorted Information
Eigenvectors of a matrix don't have to be orthogonal to each other. However if a matrix is symmetric eigenvectors are orthogonal. This is a consequence of the spectral theorem.
This matters for PCA because covariance matrix is symmetric. Consequently eigenvectors give a clean, perpendicular set of orthogonal axes to work from.
?? pretentious way of saying symmetrix is Hermitian ??