Bayesian PCA (Latent) — a second track
Contents
Why a second track
The existing Bayesian PCA (Bolt-On) track deliberately keeps the classical eigenvectors ($\mathbf{W}$) fixed and puts the prior on each component's score distribution instead — a decision left open on the Q&A page ("Should we put a prior on the loadings?").
This track explores the alternative: a prior directly on $\mathbf{W}$, in the spirit of Tipping & Bishop's Probabilistic PCA and Bishop's Bayesian PCA. Both tracks are being kept side by side rather than replacing one with the other, so they can be compared before settling on either. See What are we modeling? for why this is more than a difference in where the prior sits.
What changes vs. the first track
- Likelihood target: the raw demeaned yield matrix $\widetilde{\mathbf{Y}}_c$ (from Classical PCA's Step 5, $(T-12)\times N$), not the classical component scores.
- $\mathbf{W}$: a free parameter with its own prior ($N \times 3$, $k=3$ fixed to match classical), not a fixed input.
- $\boldsymbol{\mu}$: fixed at $\mathbf{0}$, not inferred — $\widetilde{\mathbf{Y}}_c$ is already column-demeaned, so there's no free mean left to estimate.
- No automatic component selection: unlike Bishop's Bayesian PCA, no ARD prior is used to prune components automatically — $k$ stays fixed at 3, same as classical.
Where the covariance matrix fits in
Unlike the classical track, there's no explicit "compute the covariance matrix" step here.
Classical Step 6 computes a free, full-rank empirical $\mathbf{A} = \widetilde{\mathbf{Y}}_c^\top\widetilde{\mathbf{Y}}_c/(n-1)$ and then eigendecomposes it.
In this track, the covariance matrix isn't computed directly — it's implied once a likelihood is attached in Step 2.
A likelihood says how probable the observed data would be for a given candidate value of the parameters, so it will take the form $\widetilde{\mathbf{Y}}_c \sim \text{Normal}(\mathbf{0}, \mathbf{W}\mathbf{W}^\top + \sigma^2\mathbf{I})$: the real, already-observed data $\widetilde{\mathbf{Y}}_c$ sits on the left, and a candidate $\mathbf{W}$ and $\sigma$ on the right imply a covariance matrix that the real data then gets scored against.
$\mathbf{W}\mathbf{W}^\top+\sigma^2\mathbf{I}$ therefore takes over the role $\mathbf{A}$ played classically, but instead of being computed once from the data (as $\mathbf{A}$ is), it's recomputed for every candidate $(\mathbf{W},\sigma)$ tried during inference — different candidates get judged on how plausible they make the real data look, rather than the data being generated from any single one of them. The difference from $\mathbf{A}$ is also structural: $\mathbf{A}$ could be any $N \times N$ matrix, whereas $\mathbf{W}\mathbf{W}^\top+\sigma^2\mathbf{I}$ is restricted to whatever a rank-3 loading matrix plus flat noise can produce.
At Step 1 (priors only), no covariance matrix exists yet — just priors on the pieces that will combine into one.
Once sampling exists, each posterior draw of $(\mathbf{W},\sigma)$ implies its own covariance matrix, so instead of classical's single $\mathbf{A}$ you'd get a distribution over $\mathbf{W}\mathbf{W}^\top+\sigma^2\mathbf{I}$ — the natural comparison point against classical Step 6, playing the same validation role Steps 4–5 play in the existing track.
Notes
This track is just getting started (Step 1: prior specification only, no likelihood or sampling yet). One open question is deliberately not being resolved yet: the likelihood only depends on $\mathbf{W}\mathbf{W}^\top$, so $\mathbf{W}$ is non-identified up to rotation (and reflection) — raw MCMC over its entries would need some identifiability fix (e.g. a constrained parameterization, or comparing only rotation-invariant summaries) before individual columns of $\mathbf{W}$ could be compared to classical PC1/PC2/PC3. That decision is deferred to whichever step first attaches a likelihood and samples the model.