GitHub profile

Priors

Contents

Non-informative (uninformative) prior

A prior chosen to add as little information as possible beyond what's in the data, e.g. a flat prior on $\mu_i$ and a Jeffreys prior $p(\sigma_i^2) \propto 1/\sigma_i^2$ on the variance of a component's scores. Used as the first Bayesian PCA pass so that the comparison against classical PCA isn't confounded by any expert judgement — differences between the two should come from the framework, not from added information.

Proper vs improper priors

A proper prior is an actual probability distribution — its density integrates to exactly 1 over its support, so it can be sampled from directly (e.g. $\text{Normal}(0, 1000)$). An improper prior is a non-negative function that does not integrate to a finite value — a flat prior stretched over all of $(-\infty,\infty)$, or Jeffreys' $p(\sigma_i^2)\propto 1/\sigma_i^2$ over $(0,\infty)$, both have infinite area under them, so neither is a valid distribution on its own, and there's no well-defined way to draw a sample from either in isolation.

Improper priors are still used routinely because once combined with a likelihood (actual data), the resulting posterior is typically proper again — the data supplies the constraint the prior lacks. MCMC sampling of that posterior (see MCMC) works fine with an improper prior.

The practical consequence for this project: a prior predictive check — simulating from the prior alone, before including any data, to sanity-check it's reasonable — is undefined for a true improper prior, since there's nothing to sample. Producing one requires substituting a proper but very spread-out stand-in instead (e.g. $\text{Normal}(0, 1000)$ for $\mu_i$, $\text{HalfNormal}(1000)$ for $\sigma_i$), which is no longer strictly non-informative but is close enough in practice to be a reasonable approximation.

Open decision, not yet settled: whether Step 1 of the Bayesian PCA process (specifying the prior) uses the true improper Jeffreys/flat prior — matching the Non-informative prior definition above exactly, but with no prior predictive chart possible — or the proper weakly-informative stand-in, which allows an actual prior predictive chart at the cost of encoding a small amount of information the true non-informative prior wouldn't.

Prior predictive (sampling)

Simulating hypothetical data from the model using only the prior: draw parameter values from the prior (e.g. $\mu_i \sim \text{Normal}(0,1000)$, $\sigma_i \sim \text{HalfNormal}(1000)$), then draw a simulated score from the likelihood using those draws — repeated many times to build up a distribution of what data the model considers plausible, entirely before looking at the actual component scores.

It's a sanity check on the prior/model specification, not an analysis of real data: if the prior predictive puts most of its mass on absurd values (e.g. component scores of ±10,000), the prior is too wide or mis-specified; if it's suspiciously narrow, a supposedly non-informative prior is secretly informative. This is why it requires a proper prior (see Proper vs improper priors, above) — it involves literally drawing random numbers from the prior, which is undefined for an improper one.

Contrast with posterior predictive sampling, which comes later (after Step 3's MCMC run): the same idea, but drawing parameters from the posterior instead of the prior, used to check whether the fitted model reproduces data resembling what was actually observed.

Is prior predictive sampling needed for this project?

For the uninformative-prior pass (Steps 1–7 above), not really. The prior is deliberately vague, and Step 7's comparison against the classical output already serves as the sanity check for this phase: if the uninformative Bayesian posterior doesn't closely match the classical normal fit, something's wrong — a more direct test than eyeballing a prior predictive plot of a prior designed to say almost nothing. With roughly 93 monthly observations, the likelihood dominates any reasonably wide prior anyway, so a prior predictive check here would mostly confirm "yes, this allows almost anything," which isn't very informative on its own.

It becomes genuinely important once the informative/hierarchical prior (aim 4) is on the table. A principal component is a vector spanning many original variables, so a prior placed directly on $\mu_i, \sigma_i$ for a component is not intuitively interpretable as a belief about yield curve dynamics (see the open challenge noted on the Bayesian PCA page). Prior predictive sampling is the mechanism that closes that gap: simulate from the candidate prior, project back through the loadings, and check the implied yield curve shocks actually resemble the expert view that was meant to be encoded — before the model ever sees the data.

Conclusion: skip it for the uninformative-prior pass; build it once informative priors are introduced.

Rough Notes

Intuitive Interpretation of priors

Classical PCA maps PC1 - 3 broadly to:

but not precisely.

Priors should work with this and not fight it.

Market and Structural Priors

The 3 principal components map well on to macroeconomic realities:

Intuitive Economic Priors