Exogenous Regressors

Forecast with exogenous regressors in StatsForecast, pass future values through X_df, access fitted AutoARIMA coefficients, and fix common errors.

In this notebook, we’ll incorporate exogenous regressors to a StatsForecast model.

Prerequisites

This tutorial assumes basic familiarity with StatsForecast. For a minimal example visit the Quick Start

Introduction

Exogenous regressors are variables that can affect the values of a time series. They may not be directly related to the variable that is being forecasted, but they can still have an impact on it. Examples of exogenous regressors are weather data, economic indicators, or promotional sales. They are typically collected from external sources and by incorporating them into a forecasting model, they can improve the accuracy of our predictions.

By the end of this tutorial, you’ll have a good understanding of how to incorporate exogenous regressors into StatsForecast’s models. Furthermore, you’ll see how to evaluate their performance and decide whether or not they can help enhance the forecast.

Outline

1. Install libraries
2. Load and explore the data
3. Split train/test set
4. Add exogenous regressors
5. Create future exogenous regressors
6. Train model
7. Evaluate results
8. Access the fitted coefficients
9. Troubleshooting

Tip

You can use Colab to run this Notebook interactively

Install libraries

We assume that you have StatsForecast already installed. If not, check this guide for instructions on how to install StatsForecast.

# uncomment the following line to install the library
# %pip install statsforecast

import pandas as pd

Load and explore the data

In this example, we’ll use a single time series from the M5 Competition dataset. This series represents the daily sales of a product in a Walmart store. The product-store combination that we’ll use in this notebook has unique_id = FOODS_3_586_CA_3. This time series was chosen because it is not intermittent and has exogenous regressors that will be useful for forecasting.

We’ll load the following dataframes:

• Y_ts: (pandas DataFrame) The target time series with columns [unique_id, ds, y].
• X_ts: (pandas DataFrame) Exogenous time series with columns [unique_id, ds, exogenous regressors].

base_url = 'https://datasets-nixtla.s3.amazonaws.com'
filters = [('unique_id', '=', 'FOODS_3_586_CA_3')]
Y_ts = pd.read_parquet(f'{base_url}/m5_y.parquet', filters=filters)
X_ts = pd.read_parquet(f'{base_url}/m5_x.parquet', filters=filters)

We can plot the sales of this product-store combination with the statsforecast.plot method from the StatsForecast class. This method has multiple parameters, and the required ones to generate the plots in this notebook are explained below.

• df: A pandas dataframe with columns [unique_id, ds, y].
• forecasts_df: A pandas dataframe with columns [unique_id, ds] and models.
• engine: str = matplotlib. It can also be plotly. plotly generates interactive plots, while matplotlib generates static plots.

from statsforecast import StatsForecast 

StatsForecast.plot(Y_ts)

The M5 Competition included several exogenous regressors. Here we’ll use the following two.

• sell_price: The price of the product for the given store. The price is provided per week.
• snap_CA: A binary variable indicating whether the store allows SNAP purchases (1 if yes, 0 otherwise). SNAP stands for Supplement Nutrition Assitance Program, and it gives individuals and families money to help them purchase food products.

X_ts = X_ts[['unique_id', 'ds', 'sell_price', 'snap_CA']]
X_ts.head()

	unique_id	ds	sell_price	snap_CA
0	FOODS_3_586_CA_3	2011-01-29	1.48	0
1	FOODS_3_586_CA_3	2011-01-30	1.48	0
2	FOODS_3_586_CA_3	2011-01-31	1.48	0
3	FOODS_3_586_CA_3	2011-02-01	1.48	1
4	FOODS_3_586_CA_3	2011-02-02	1.48	1

Here the unique_id is a category, but for the exogenous regressors it needs to be a string.

X_ts['unique_id'] = X_ts.unique_id.astype(str)

We can plot the exogenous regressors using plotly. We could use statsforecast.plot, but then one of the regressors must be renamed y, and the name must be changed back to the original before generating the forecast.

StatsForecast.plot(Y_ts, X_ts, max_insample_length=0)

From this plot, we can conclude that price has increased twice and that SNAP occurs at regular intervals.

Split train/test set

In the M5 Competition, participants had to forecast sales for the last 28 days in the dataset. We’ll use the same forecast horizon and create the train and test sets accordingly.

# Extract dates for train and test set 
dates = Y_ts['ds'].unique()
dtrain = dates[:-28]
dtest = dates[-28:]

Y_train = Y_ts.query('ds in @dtrain')
Y_test = Y_ts.query('ds in @dtest') 

X_train = X_ts.query('ds in @dtrain') 
X_test = X_ts.query('ds in @dtest')

Add exogenous regressors

The exogenous regressors need to be place after the target variable y.

train = Y_train.merge(X_ts, how = 'left', on = ['unique_id', 'ds']) 
train.head()

	unique_id	ds	y	sell_price	snap_CA
0	FOODS_3_586_CA_3	2011-01-29	56.0	1.48	0
1	FOODS_3_586_CA_3	2011-01-30	55.0	1.48	0
2	FOODS_3_586_CA_3	2011-01-31	45.0	1.48	0
3	FOODS_3_586_CA_3	2011-02-01	57.0	1.48	1
4	FOODS_3_586_CA_3	2011-02-02	54.0	1.48	1

Create future exogenous regressors

We need to include the future values of the exogenous regressors so that we can produce the forecasts. Notice that we already have this information in X_test.

X_test.head()

	unique_id	ds	sell_price	snap_CA
1941	FOODS_3_586_CA_3	2016-05-23	1.68	0
1942	FOODS_3_586_CA_3	2016-05-24	1.68	0
1943	FOODS_3_586_CA_3	2016-05-25	1.68	0
1944	FOODS_3_586_CA_3	2016-05-26	1.68	0
1945	FOODS_3_586_CA_3	2016-05-27	1.68	0

Important

If the future values of the exogenous regressors are not available, then they must be forecasted or the regressors need to be eliminated from the model. Without them, it is not possible to generate the forecast.

Train model

To generate the forecast, we’ll use AutoARIMA, which is one of the models available in StatsForecast that allows exogenous regressors. To use this model, we first need to import it from statsforecast.models and then we need to instantiate it. Given that we’re working with daily data, we need to set season_length = 7.

from statsforecast.models import AutoARIMA

# Create a list with the model and its instantiation parameters 
models = [AutoARIMA(season_length=7)]

Next, we need to instantiate a new StatsForecast object, which has the following parameters.

• df: The dataframe with the training data.
• models: The list of models defined in the previous step.
• freq: A string indicating the frequency of the data. See pandas’ available frequencies.
• n_jobs: An integer that indicates the number of jobs used in parallel processing. Use -1 to select all cores.

sf = StatsForecast(
    models=models, 
    freq='D', 
    n_jobs=1,
)

Now we’re ready to generate the forecast. To do this, we’ll use the forecast method, which takes the following arguments.

• h: An integer that represents the forecast horizon. In this case, we’ll forecast the next 28 days.
• X_df: A pandas dataframe with the future values of the exogenous regressors.
• level: A list of floats with the confidence levels of the prediction intervals. For example, level=[95] means that the range of values should include the actual future value with probability 95%.

horizon = 28
level = [95]

fcst = sf.forecast(df=train, h=horizon, X_df=X_test, level=level)
fcst.head()

	unique_id	ds	AutoARIMA	AutoARIMA-lo-95	AutoARIMA-hi-95
0	FOODS_3_586_CA_3	2016-05-23	72.956276	44.109070	101.803482
1	FOODS_3_586_CA_3	2016-05-24	71.138611	40.761467	101.515747
2	FOODS_3_586_CA_3	2016-05-25	68.140945	37.550083	98.731804
3	FOODS_3_586_CA_3	2016-05-26	65.485588	34.841637	96.129539
4	FOODS_3_586_CA_3	2016-05-27	64.961441	34.291973	95.630905

We can plot the forecasts with the statsforecast.plot method described above.

StatsForecast.plot(Y_ts, fcst, max_insample_length=28*2)

Evaluate results

We’ll merge the test set and the forecast to evaluate the accuracy using the mean absolute error (MAE).

res = Y_test.merge(fcst, how='left', on=['unique_id', 'ds'])
res.head()

	unique_id	ds	y	AutoARIMA	AutoARIMA-lo-95	AutoARIMA-hi-95
0	FOODS_3_586_CA_3	2016-05-23	66.0	72.956276	44.109070	101.803482
1	FOODS_3_586_CA_3	2016-05-24	62.0	71.138611	40.761467	101.515747
2	FOODS_3_586_CA_3	2016-05-25	40.0	68.140945	37.550083	98.731804
3	FOODS_3_586_CA_3	2016-05-26	72.0	65.485588	34.841637	96.129539
4	FOODS_3_586_CA_3	2016-05-27	69.0	64.961441	34.291973	95.630905

mae = abs(res['y']-res['AutoARIMA']).mean()
print('The MAE with exogenous regressors is '+str(round(mae,2)))

The MAE with exogenous regressors is 11.42

To check whether the exogenous regressors were useful or not, we need to generate the forecast again, now without them. To do this, we simply pass the dataframe without exogenous variables to the forecast method. Notice that the data only includes unique_id, ds, and y. The forecast method no longer requires the future values of the exogenous regressors X_df.

# univariate model 
fcst_u = sf.forecast(df=train[['unique_id', 'ds', 'y']], h=28)

res_u = Y_test.merge(fcst_u, how='left', on=['unique_id', 'ds'])
mae_u = abs(res_u['y']-res_u['AutoARIMA']).mean()

print('The MAE without exogenous regressors is '+str(round(mae_u,2)))

The MAE without exogenous regressors is 12.18

Hence, we can conclude that using sell_price and snap_CA as external regressors helped improve the forecast.

Access the fitted coefficients

ARIMA-family models with exogenous regressors are regressions with ARIMA errors. The exogenous regressors enter the model linearly and additively:

$\hat{y}_t = \text{ARMA part} + C_1 x_{1,t} + C_2 x_{2,t} + \dots + C_N x_{N,t}$

This means the contribution of each regressor to a given prediction is its coefficient multiplied by its value at that point.

To inspect the coefficients, train the model with the fit method (rather than forecast) and access the fitted model of each series through the fitted_ attribute. fitted_ is an array of shape (number of series, number of models).

sf.fit(df=train)

result = sf.fitted_[0, 0].model_  # first series, first model
print(result['coef'])

result['coef'] is a dictionary. The AR and MA terms use keys like ar1, ma1, sar1, and sma1. The exogenous regressors use the keys ex_1, ex_2, and so on, in the same order as the columns of the training dataframe. When the model includes a constant, the dictionary also contains an intercept or drift key.

Troubleshooting

Common errors when forecasting with exogenous regressors:

ValueError: xreg is rank deficient. AutoARIMA checks that the training exogenous regressors are linearly independent before fitting. The check drops constant columns and then stacks the remaining columns with a linear trend, so it fails when one column is a linear combination of the others or of a linear time trend. To fix it, remove redundant columns. For example, drop one category from a full set of one-hot encoded dummies, or remove a column that increases linearly with time. This check only applies to the training data. The future values in X_df are only multiplied by the fitted coefficients, so linear dependence that appears only in the forecast horizon does not raise this error.

ValueError: Expected X to have shape (a, b), but got (c, d). X_df must contain the unique_id and ds columns plus exactly the same exogenous columns used in training, with one row per series for each timestamp in the forecast horizon. An extra column (such as a leftover y) or a missing column changes the shape and raises this error.

Models require the following exogenous features .... If the training data contains exogenous columns and a model that supports them is used, the forecast method requires their future values through X_df.