Bayes Theorem
Contents
The core idea
$$p(\theta \mid \text{data}) = \frac{p(\text{data} \mid \theta) \times p(\theta)}{p(\text{data})}$$
In words: posterior ∝ prior × likelihood.
- Prior $p(\theta)$: how plausible each possible parameter value $\theta$ is, before looking at any data.
- Likelihood $p(\text{data} \mid \theta)$: for a specific candidate value of $\theta$, how probable it is that we'd have seen the data we actually observed. A candidate that makes the observed data look "typical" gets a high number; one that makes the data look bizarre gets a tiny number.
- Posterior $p(\theta \mid \text{data})$: multiply prior × likelihood for every candidate $\theta$, then rescale so the result is a valid probability distribution again — that rescaling constant, $p(\text{data})$, is the marginal likelihood. Candidates that are both plausible upfront and explain the data well end up with high posterior probability; everything else gets down-weighted.
So "sampling the posterior" isn't computing a formula and then drawing from it — for MCMC in particular, it's a search process that, by construction, spends time in each region of parameter space in proportion to how probable that region is (see MCMC). The collection of visited points is the posterior.
Prior, likelihood, and posterior in this project
For one component, $\theta = (\mu_i, \sigma_i)$. The prior (see Priors) says every candidate $\mu_i$ and $\sigma_i>0$ is equally plausible before data. The likelihood: pick a candidate $(\mu_i, \sigma_i)$, plug the ~93 observed monthly scores into the Normal density for that $\mu_i, \sigma_i$, multiply those 93 numbers together — a candidate that makes the observed scores look typical under $\text{Normal}(\mu_i, \sigma_i)$ scores highly; one that doesn't (e.g. $\mu_i = 1000$ when all the scores are near 0) scores almost zero. The posterior combines the two, and is what Steps 3–5 of the Bayesian PCA process actually work with.
There are two routes to turning prior × likelihood into an actual posterior: solving the algebra directly (below), or MCMC (see MCMC) when the algebra doesn't simplify — which is the route this project takes.
Conjugate / closed-form posterior
When the prior and likelihood are chosen from matching families (e.g. Normal-Inverse-Gamma prior with a Normal likelihood), the posterior falls out algebraically as an update to the prior's parameters — no simulation required, and the posterior predictive distribution (a Student-t, in the per-component-score case) is exact.