Best procedure to select predictors and ARIMA model terms

[maxdevblock]

I'm searching for a valid procedure to select predictors and $\text{ARIMA}(p,d,q)(P,D,Q)[S]$ model in a forecasting framework.

I have a dataset from $t=1$ to $t=T$, with a target variable and $p$ predictors. I want to find the best selection of $k \leq p$ predictors and $(p,d,q)(P,D,Q)$ ARIMA terms minimizing the prediction error up to $t=T+h$ (with $S=12$).

The first procedure I thought about is:

1. for each of the $2^p$ predictors' combinations
   a. find the best $(p,d,q)(P,D,Q)$ ARIMA model by minimizing the corrected Akaike Information Criterion (AICc) over the full dataset $t=1,\ldots,T$ using the Hyndman-Khandakar algorithm
   b. train-test split the dataset, so that the train set span $t=1,\ldots,\tau$ and the test set span $t=\tau+1,\ldots,T$ with $\tau < T$, for $u$ values of $\tau$
   c. for each train-test split, refit the model (re-estimating the parameters) and calculate the test-RMSE
   d. average the test-RMSE
2. among the $2^p$ models resulting from point (1), choose the one minimizing the average test-RMSE

A second procedure (more difficult to implement) could be:

1. for each of the $2^p$ predictors' combinations
   a. train-test split the dataset, so that the train set span $t=1,\ldots,\tau$ and the test set span $t=\tau+1,\ldots,T$ with $\tau < T$, for $u$ values of $\tau$
   b. use an ad-hoc algorithm similar to the Hyndman-Khandakar but minimizing the average test-RMSE to find the best $(p,d,q)(P,D,Q)$ ARIMA terms
2. among the $2^p$ models resulting from point (1), choose the one minimizing the average test-RMSE

Which of these two procedures could be better? Is there a best practice and/or a more rigorous and tested approach?

Thank you.

[robjhyndman]

Great question! I've done something similar to the first procedure before, and it works pretty well. Note that minimizing the AICc is asymptotically equivalent to minimizing one-step RMSE on cross-validated test sets, so there is no guarantee that the ARIMA model will be optimal for multi-step forecasting. On the other hand, if the data truly come from the fitted model, then optimizing for one-step RMSE will also give the optimal model for multi-step RMSE.

The second approach focuses more directly on the multi-step RMSE, but is less efficient in choosing the ARIMA model as there are a limited number of training/test splits that you can average over. It will also be much slower as the model search space is much bigger (including all the possible ARIMA models in each iteration).

I don't know of any literature on this, but I would guess that it won't make that much difference to forecast accuracy, and so I'd go for the faster approach, namely the first procedure.

[maxdevblock]

Hi Rob. Many thanks.

The only paper I've found so far about a rigorous approach for selecting the best SARIMAX model is

Xie, J. (2023). Identifying optimal indicators and lag terms for nowcasting models. International Monetary Fund. https://doi.org/10.5089/9798400235177.001

The selection procedure is "simpler" (doesn't involve the average test-RMSE) because it's for nowcasting. Xie suggests to check for significance also, and it makes definitely sense to me because, in this case, the model would also be valid for explanation (and, maybe, reduce the prediction interval?).
Thus, I'm thinking to change the point (2) of the first procedure to

2. among the $2^p$ models resulting from point (1), choose the one minimizing the average test-RMSE and whose parameters p-values are all less than $\alpha$ (Xie suggests 5% significance level)

so that the selected model would finally be:

• the model minimizing the average test-RMSE (over $u$ train-test splits)
• with significant estimates at a chosen $\alpha$ level
• and the predictors combination minimizing AICc (and, asymptotically, the one-step RMSE) over the full dataset

What do you think about it?

Thank you.

[robjhyndman]

Statistical significance is a completely different problem from selecting predictors. I would not confuse them.

[maxdevblock]

Thank you @robjhyndman

Indeed, upon closer reading of Xie's paper, no specific reason is adduced for this choice, nor are there references to Scholars who made the same choice or proof that this choice could improve the model.

I will therefore proceed with the first approach, as it is fully defensible and does not conflate different theoretical frameworks.