A few years ago, I was tasked with forecasting student enrollment counts across various courses at a university. Initially, the problem seemed daunting due to numerous potential factors affecting the forecasts, such as Covid-19 impacts, course suspensions, and the introduction of innovative technologies. However, the primary concern for business stakeholders was not just the forecasted values themselves but rather the aggregated demand across different disciplines (Humanities, Social & Natural Sciences, Engineering, Medical and Health Sciences, Business and Economics, Law, Teaching, Fine Arts, Computer Science and Information Technology).
This information was crucial for several purposes:
- Recruitment and capacity planning.
- Implementing new topics, courses, or educational content to an existing educational program.
- Decision of whether few courses should be discontinued or not.
- Check the demand of a particular course or at an aggregated discipline.
We were interested to see how applied machine learning and other statistical methods can help us solve this problem.
List of techniques that were ideated to help achieve the above outcomes were
1. Statistical Methods
- ARIMA, SARIMA, Naive method or ETS.
2. Machine Learning
- Supervised Learning
- Regression
- Predicting enrolment count of individual courses or respective discipline (Business and Economics, Law, Teaching, or ICT)
- Regression
- Un-supervised Learning
- Clustering
- Clustering college students into cohorts based on their grades/ATAR scores and the subjects of their interest undertaken during their college days and the course expression of interest/registrations for international students.
- Clustering
Hierarchical or grouped time-series forecasting a supervised learning algorithm helps us solve this problem as it helps and is used to predict future values for various levels of a hierarchical structure, typically within an organisation or a system. This approach involves forecasting at distinct levels of aggregation and ensuring that forecasts at lower levels (e.g., individual courses) are consistent with forecasts at higher levels (e.g., total enrolment count or count of Semi-Aggregate level i.e., individual discipline)
This functioned as an intuitive and user-friendly tool, aiding them in making informed decisions and providing a clearer understanding of how different external factors could impact the organisation.
(where C_1,…C_n are different courses under each discipline)
Ex: A two-level hierarchical tree diagram
Key Features of HF
- Levels of Hierarchy
- Temporal hierarchies: The structural connection across the different levels of aggregation.
- Forecasting models (Statistical, Machine Learning or deep learning techniques)
- Classical reconciliation methods:
- Top-Down Approach
- Bottom-Up Approach
- Ordinary least squares (OLS)
- Alternative reconciliation methods:
- Middle-Out Approach
- Min-Trace
- ERM
- Accuracy and Consistency
- MAPE (Mean absolute percentage error) or MAE (Mean Absolute Error)
- RMSE (Root mean squared error)
- SMAPE (Symmetric mean absolute percentage error)
Steps to generate Hierarchical forecasts and Reconciliation for your dataset
- Data Preparation
- Data cleaning (Handle Null values, etc….)
- EDA (Exploratory data analysis of the data)
- Feature Engineering (Inclusion or exclusion of extraneous variables that may affect the forecast values) & Hyper parameter tuning
- Convert categorical variables into a long format (Allows data to be stored more densely/sparsely)
- Normalise seasonality and trend
- Define Data Hierarchy
- Split Data into Train, Validation, and test dataset
- Model Selection (Machine Learning or Statistical Methods)
4.1 Statistical Methods: ARIMA, SARIMA, Exponential Smoothing.
4.2 Machine Learning: LightGBM, RandomForests, SVM, ANN or XgBoost- Amazon Forecast: DeepAR(hts), ARIMA, SARIMA, Naive & ETS
- SOA Nixtla : HierarchicalForecast
- Facebook Prophet
- GluonTs : HierarchicalTimeseries
- Scikit-hts
- Generate & Visualise Forecasts
- Reconciliation of the forecasts
- Classical reconciliation
- Alternative reconciliation
- Evaluate Forecasts
- Deployment & fine-tuning
Challenges with Hierarchical or grouped time series forecasting
- Lack of available historical data and irregular patterns
- Complex structures arise when attributes of interest are both nested and crossed
- Top-Down forecasts do not align with bottom-up forecasts i.e., forecasts that are coherent across the entire aggregation
- Inaccurate or improper feature engineering can lead to error or biases.
- Seasonality and trends in the dataset
- External factors: accounting for external factors (e.g., economic conditions, competitor actions) that impact various levels of the hierarchy adds to the complexity.
Use Cases and Applications
- Retail Product & Catalogue Hierarchy Forecasting
- Applications
- Demand Forecasting for products/catalogue at a (Store/State/ catalogue) level for stock availability
- Merchandising: Forecast prices for various levels of hierarchy
- System Network Hierarchy
- Applications
- Detect usage demand peaks.
- Forecasting enrolment count of university courses
- Applications
- Applications
Choosing a forecasting model
Choosing a model can be confusing as there are several tools and open-sourced packages out there that help us achieve the required outcome, below a set of features to consider when deciding on a forecast model.
Availability of hierarchical reconciliation methods in various package
Minimal Example of Hierarchical Forecasting & Reconciliation
To give a high-level glimpse of conducting HF, we are using the darts package. We will see how reconciling forecasts can be done in a few lines of code.
In this instance, we are utilising the Australian domestic tourism which has the count of tourists for 35 months (about 3 years).
1. Import the required packages
- Using the models we import the LinearRegressionModel.
- For metrics we use the MAE (Mean Absolute Error).
- For reconciliation we use the Min-Trace Reconciliation.
2. Data Preparation
The loaded dataset has several components:
- One component named “Total”
- One component per “region” (“NSW”, “VIC”, etc)
- One component per tourism reason (“Hol” for holiday, “Bus” for business, etc)
- One component per (region, reason) pair (named “NSW – hol”, “NSW-bus”)
- One component per (region, reason, <city/non-city>) pair
Let’s visualise the dataset for the NSW state for each hierarchy or name-pair.
(time is measured in months on the x-axis, totalCount of tourists on the y-axis)
3. Defining data hierarchy
Some of these components add up in certain ways. For instance, the sum of the holiday tourism done in New South Wales can be decomposed as the sum of “city” and “noncity” holiday New South Wales tourism:
(time is measured in months on the x-axis)
Similarly, both the region and reason can be grouped together
So, the overall hierarchy looks something like this:
4. Feature Engineering
Encoding the Hierarchy the way dart expects
- Fill in grouping by reason, region, (reason, region) and (reason, region, city/non-city)
In Darts, hierarchies are a property of TimeSeries objects so that we convert the dataset from dict to the time-series format. We can now embed our hierarchy in our TimeSeries. We do this using the with_hierarchy() method, but we could also specify the hierarchy at the series construction; for instance, providing a hierarchy to the TimeSeries.from_dataframe() factory method.
5. Generating the forecasts
- Split the dataset into train and validation.
(For validation we use the last 12 months of the dataset)
- Visualising the forecasted values
6. Model Metrics
From the above metrics we can see the model forecasts had the least error value when forecasted on (region, reason, city) compared to other hierarchies.
MAE works in a simple way, yet is a powerful metric used to evaluate the accuracy of regression models. It measures the average absolute difference between the predicted value and the actual value. Let us see the predicted_value and actual_value of a single month/record.
We see the model forecasted 5k less records to the actual total, we can say the model is 100% accurate if the MAE value is 0. To improve the MAE, one should focus on the feature engineering steps to see if there are any correlating variables, null values, etc….
7. Reconciliation of the forecasts
First, let us see if our forecast values add up.
They do not add up. So, let’s reconcile them.
We use the darts.dataprocesssing.transformers to do that. These transformers can perform posthoc reconciliation (i.e., reconcile forecasts that already have been generated) We use the Min-Trace method for this example.
Darts package provides
- BottomUpReconciliator : performs bottom-up reconciliation, simply resetting each component in the hierarchy as the sum of its children components
- TopDownReconciliator : performs top-down reconciliation, which breaks down the aggregate forecasts down the hierarchy using historical proportions. This transformer requires calling fit() with the historical values (i.e., the training series) in order to learn these proportions.
- MinTReconciliator: Technique to perform “optimal” reconciliation, as detailed here. This transformer can work in a few different ways.
After running the above commands, we see if the forecasted values for all hierarchies add up. Let us visualise them.
Now let us see the mean MAE values after running the reconciliation step.
In contrast to the expected initial values, there are marginal improvements observed across all levels of the hierarchy. This is typical of reconciliation; it can increase the error, but also decrease it in some cases as reconciliation only adds up the forecasted values across the hierarchy. It is suggested to be reconciling the generated forecasted using various forecasting techniques to see how the MAE values vary for different techniques.
Once the desired MAE error value is achieved, we can use Sagemaker Experiments & Models to finetune and deploy the model of our choice to a desired inference instance.
Conclusion
In summary, hierarchical or grouped time series forecasting is a powerful technique to see how the model performs on distinct levels of hierarchies and the steps one can take to ensure the forecasted values across all levels of hierarchy add up. By following these steps, organisations can leverage hierarchical or grouped time series forecasting to gain insights into trends, make informed decisions, and optimise resource allocation effectively. This approach not only enhances forecasting accuracy but also provides a structured framework for managing complex data relationships in predictive analytics.
