Dimensionality Reduction

Notation

Each eigenvalue is the absolute variance captured by its component, not a percentage. To get the percentage of total variance explained by component $i$:

$$\text{\% explained}_i = \frac{\lambda_i}{\sum_j \lambda_j} \times 100$$

The denominator $\sum_j \lambda_j$ is the sum of all eigenvalues from Step 7, equal to $\text{trace}(\mathbf{A})$ — the total variance across all original variables.

Running total across components gives the cumulative percentage explained, which is what drives the decision of how many components to keep.

step 1 - calculate percentage and cumulative percentage explained

import subprocess  
%run ./dataframes/step8_variance_explained.py  
%run ./dataframes/truncate_with_ellipsis.py  
df = variance_explained()  
subprocess.run(['wl-copy'], input=truncate_with_ellipsis(df, 3).to_html(index=False).encode())  

Python

filename: step8_variance_explained.py

from dataframes.step7_eigendecomposition import *

def variance_explained():
    df = eigenvalues()
    total = df["eigenvalue"].sum()
    df = df.copy()
    df["pct_explained"] = df["eigenvalue"] / total * 100
    df["cumulative_pct_explained"] = df["pct_explained"].cumsum()
    return df