Replacing Judgmental Forecasting with Statistical Methods

Although machine learning and data science is on everyone's lips in recent years, and with that the promise of an AI revolution, the methodologies and technologies currently available are often not taken advantage of. In the following blog I will show a case of how, with relative low resources, current business forecasting methods can be replaced with more modern ones, giving better results and allowing for a higher degree of automatization.

What is Judgmental Forecasting?

Accurate prediction of aggregated business data is of great interest in various contexts. Reliable predictions open the possibility of planning for the future, which in turn lead to cost-savings. Examples of this are sales data, revenue, demand for a certain product, or as in our case described below, treasury cash-flow. Often, the demand for this type of forecasting is fulfilled by what is known as judgmental forecasts, which means that the prognostication is left to the individual forecaster’s skill at judging previous behavior, anecdotal evidence, domain knowledge and his or her previous experience, but doesn’t make use of statistical modelling. Although this type of forecasting can be effective, it has numerous disadvantages. Foremost, it is impossible for outsiders to determine the reason for systematic forecasting errors. Secondly, since the forecasts are subjective, changes in personnel can lead to drastic changes in systematic errors as well as performance, which can only fully be determined after a track record has been established.

Model Based Forecasting

An alternative to judgmental based forecasting is model based forecasting. This type of forecasting offers solutions to some of the disadvantages mentioned previously. Since the models are not a black-box to outsiders, external forecasters are able to investigate systematic errors and successful models can often be easily applied to related areas. Furthermore, model based forecasts are also able to give a probabilistic statement about the range of forecasts provided, such as, “the probability of lower sales figures than last year is 10%”.

Although a model based forecasting procedure is generally preferable, under certain conditions judgmental based forecasting must be used:

  • If there is no historical data
  • If due to changing conditions, the historical data is no longer relevant (e.g. due to new competitors on the market)

Treasury Cash-Flow Use Case: Replacing Judgemental Methods

An example of replacing judgmental forecasting with model based forecasting is a project we at Swisscom Analytics have conducted since spring 2018, where we are predicting daily cashflows from different debtor accounts. Having more accurate predictions in this area provides additional benefits. In this case, more money can be placed in different savings accounts which generate interests. These predictions have for the last 10 years been created using a judgmental framework, and from experience gathered during those years, have become quite accurate. Nevertheless, we were convinced that we could do it better and more efficiently by using a model based approach.

Feature Engineering and Tools

Due to the nature of time series data, where fluctuations are depending on the day of the month, we applied an R-script which creates features for months, holidays, working day of the month, working day of the month backwards, etc. All in all, it creates a large number of features which are able to capture those fluctuations well.

After the feature engineering step, we made use of the open source tool “Prophet”, which has been created by Facebook research. Behind Prophet is an additive model with components for linear or logistic growth trend, yearly seasonality, weekly seasonality, holidays and additional regressors. In our case we added the features created previously as additional regressors. Using this framework, we are then able to provide predictions for the daily cashflow one month ahead of time.

25% Lower Average Prediction Error

Evaluating the performance of the model based approach for 2018, we were very happy to see that, on average, the daily prediction error has been lowered by 25% when compared to the previous method! However, for 2018 the feature engineering and data extraction were all done manually, therefore the next steps of this project in 2019 will be to automate the entire process chain to be able to provide equally accurate and high-quality predictions while minimizing the work load.