Transform the dataset

We perform the following transformations:

a. differencing
b. de-meaning

Differencing the dataset

import subprocess  
%run ./dataframes/step4_differencing.py  
df = differencing()  

so we are up to a point where we want to deduct from 1 year ahead the value from 1 year previous
and the way i suggest we do this is to pick out some specific values in random places
and do it by hand... and then look at the result otherwise...

proof that calculations are correct

In [9]: df.iloc[50,0] Out[9]: '2020-05-31'

In [10]: df.iloc[50,5] Out[10]: -3.506557897319982

In [11]: df.iloc[62,0] Out[11]: '2021-05-31'

In [12]: df.iloc[62,5] Out[12]: -0.527632742082372

In [2]: df_diff.iloc[50,0] Out[2]: '2021-05-31'

In [9]: df_diff.iloc[50,5] Out[9]: 2.9789251552376097

Second Check

An even better proof is to sum all the values in the dataframe.. and sum all the values in the first year. sum of the differences should be

We calculate 2 scalars. The difference should equate to the sum of the first twelve rows in the full dataset:

$ S_Y = \sum_{t=1}^{T}\sum_{n=1}^{N} y_{t,n} $

$ S_{\widetilde{Y}} = \sum_{s=1}^{T-12}\sum_{n=1}^{N} \widetilde{y}_{s,n} = \sum_{t=13}^{T}\sum_{n=1}^{N} y_{t,n} $

$ D = S_Y - S_{\widetilde{Y}} = \sum_{t=1}^{12}\sum_{n=1}^{N} y_{t,n} $

Python

filename: step4_differencing.py

from dataframes.step3_taking_logs import *
def differencing():
    df = taking_logs()  
    df_diff = df.copy()
    df_diff.iloc[:, 1:] = df.iloc[:, 1:].diff(12)
    # drop the first 12 rows since they'll be NaN
    df_diff = df_diff.dropna()
    return df_diff