De-mean the dataset

Notation

Let $\mathbf{Y} \in \mathbb{R}^{T \times N}$ denote the raw spot yields (as defined earlier), before logging or differencing.

The dataset arriving at this step is not $\mathbf{Y}$ — it's the log-differenced series produced in Step 4: $\widetilde{\mathbf{Y}} \in \mathbb{R}^{(T-12)\times N}$, entries $\widetilde{y}_{s,n}$, $s=1,\dots,T-12$. To keep $\mathbf{Y}$ reserved for the raw yields, this step's input is written as $\widetilde{\mathbf{Y}}$ throughout, never plain $\mathbf{Y}$.

Demeaning proceeds in two operations:

1. Column means — for each maturity $n$: $\bar{\widetilde{y}}_n = \dfrac{1}{T-12}\sum_{s=1}^{T-12} \widetilde{y}_{s,n}$, giving a mean vector $\vec{\widetilde{y}} \in \mathbb{R}^N$.
2. Demean — subtract each column's mean from every entry in that column: $\widetilde{y}^{\,c}_{s,n} = \widetilde{y}_{s,n} - \bar{\widetilde{y}}_n$, producing the demeaned matrix $\widetilde{\mathbf{Y}}_c$.

step 1 - calculate the column means

import subprocess  
%run ./dataframes/step5_column_of_means.py  
%run ./dataframes/truncate_with_ellipsis.py  
df = column_of_means()  
subprocess.run(['wl-copy'], input=truncate_with_ellipsis(df, 3).to_html(index=False).encode())  

step 2 - calculate demeaned dataset

import subprocess  
%run ./dataframes/step5_demeaning.py  
%run ./dataframes/truncate_with_ellipsis.py  
df = demeaning()  
subprocess.run(['wl-copy'], input=truncate_with_ellipsis(df, 3).to_html(index=False).encode())  

Python

filename: step5_column_of_means.py

from dataframes.step4_differencing import *
import pandas as pd

def column_of_means():
    df = differencing()
    value_cols = df.columns[1:]
    means = df[value_cols].mean()
    df_means = pd.DataFrame({"maturity": means.index, "mean": means.values})
    return df_means

filename: step5_demeaning.py

from dataframes.step5_column_of_means import *

def demeaning():
    df = differencing()
    means = column_of_means().set_index("maturity")["mean"]
    value_cols = df.columns[1:]
    df_demeaned = df.copy()
    df_demeaned[value_cols] = df[value_cols] - means
    return df_demeaned