Mutual Information for Time Series Forecasting in Python
When analysing time series and building forecasting systems, it is essential to understand the predictive strength of endogenous and exogenous features and their effects on the target variable. Would it not be great to have some ranking of feature utility in terms of their predictive power? How could one achieve such an objective?
Of course, there are tools like correlation. However, correlation is not only limited to explaining linear relationships, but also suffers from the inability to correctly reflect the predictive power between features and the target variable. You probably have heard of Anscombe’s quartet, but there is also a more fun way to illustrate the concept. In the Datasaurus Dozen, all these datasets have the exact statistics, including the same correlation between feature X and the target y.
But let’s not get distracted; now that we know that correlation is far from ideal? How do we really measure the predictive power of features (both internal and external) in terms of their effect on target y?
For time series, there are tools like ACF that can help to identify the impact of lags on the target variable — but ACF is … again computed using linear correlations.
Can we do better in the era of machine learning that can deal with non-linear relationships between features and the target?
It turns out that we can, using the concept of ‘mutual information.’ Mutual information is a measure of the shared information between two random variables. Mutual information quantifies the amount of information that one variable provides about the other variable, and it does so without relying on any assumptions like linearity.
Formally, if X and Y are two random variables, the mutual information I(X; Y) between X and Y is defined as the expected value of the logarithmic likelihood ratio between the joint distribution of X and Y and the product of their marginal distributions:
I(X; Y) = E[log (p(X, Y) / (p(X) * p(Y)))]
where p(X, Y) is the joint probability distribution of X and Y and p(X) and p(Y) is the marginal probability distributions of X and Y.
The mutual information is non-negative and zero if and only if X and Y are independent. Mutual information can be seen as a measure of reduction in uncertainty about one variable given the knowledge of the other variable.
Mutual information is widely used in various fields, including information theory, machine learning, statistics, and neuroscience, to quantify the relationships between variables, select relevant features, cluster data, and study information processing in biological systems.
Given two-time series X and Y, the mutual information I(X; Y) can be used to determine the strength and direction of the relationship between X and Y, such as positive or negative dependence, and the degree of non-linearity in the relationship. This information can then inform decision-making or develop models to forecast future values of one series given values of another series.
Another application of mutual information in time series analysis is feature selection. Given a large number of time series variables, mutual information can be used to determine which variables are the most informative about the target variable, allowing data scientists to select a subset of relevant variables for further analysis or modelling.
Mutual information provides a flexible and powerful tool for analysing the relationships between time series variables and extracting meaningful information from time series data.
In this example, we will look at the electricity dataset I have used in several of my Medium articles (if you have not read them, check them out!).
Imagine we are interested in finding out which lags are more critical in modelling the time series of electricity consumption. We first create lags using .shift(n) in Pandas, where n is the lag order. The resulting data frame looks like this.
We use mutual_info_regression from Scikit_learn to compute mutual information between each lag and the target and plot the resulting mutual information scores.
Mutual information has identified a decaying pattern of lag order impact on the target; perhaps more importantly, it has also identified several local maxima every seven days. This is interesting and logical as electricity price consumption has a particular seasonality pattern.
Finally, let’s compare this with the results from the ACF. Whilst ACF also shows a declining pattern, it is more smooth than suggested by mutual information and does not have local maximum peaks at every seventh’s lag. Other tools from the time series toolkit can identify such seasonality, but this is not the subject of this article.
As we have seen, mutual information can be used as an effective tool in time series analysis and forecasting by measuring the dependence between a time series and its lags, which can provide valuable information about the underlying structure and relationships in the data.
Mutual information can be used for model-free feature selection to reduce the data’s dimensionality and improve the models’ accuracy.
Mutual information can also be used for model selection to compare the performance of different time series models and to select the best-performing model. You can determine which model best fits the data by comparing the mutual information between the observed time series and the predicted time series for different models.
Enjoyed reading this article?
You can read my other articles here -> https://medium.com/@valeman and follow me on Twitter, LinkedIn or ResearchGate
Check out my book ‘Practical Guide to Applied Conformal Prediction: Learn and apply the best uncertainty frameworks to your industry applications’ -> https://a.co/d/dArQW07
