Evaluating a Revenue Management solution these days is a lot like purchasing a new car. It’s easy to be distracted by the “bells and whistles” and nice-to-have features. But at the end of the day, you probably care most about how well that car serves its basic purpose – to get you from point A to point B effectively and reliably.
If a car’s engine is the most integral factor that determines your success in reaching your destination, then you can think of a revenue management solution’s forecast as its “engine.” It’s the core underlying component that powers the rest of the system and its outputs. If you are a car aficionado, you might care to know how the engine is built, what gasoline fuels it, and how fast and powerful it is. Likewise, if you are keenly interested in the science of revenue management, you might be interested in how your system’s forecast is constructed, what types of data feed into it, and what kind of processing speed your system is capable of.
However, you hopefully care most about the main purpose of your revenue management system, which is to help maximize profit for your property or company. If so, you should primarily be concerned with the most important indicator of performance for your system’s “engine” – forecast accuracy. The more accurate its forecast, the more appropriate the system’s pricing and yielding recommendations will be, and the greater the revenue that will result. This is a proven linkage – in the airline industry, a 10% improvement in forecast accuracy was found to generate a 0.5% to 3.0% increase in revenue, and due to the high degree of flow-through, an even greater percent increase in profit.
So, when evaluating your current system or a new revenue management solution, it’s probably a good idea to ask about how accurate its forecast is. But what specific metrics should you look at?
Consider the following forecast and actual booking data for the past five Thursdays:
If I simply average the difference between my forecasted and actual observations, otherwise known as the “error,” I’d get a result of -0.2. That small number suggests a very high degree of accuracy – or does it?
We can deviate from our actuals in either direction. But, netting the “ups and downs” against each other gives us a false sense of security, since we care about the magnitude of the error and (most of the time) not the direction. So to correct this, we should take the absolute value of each forecast error, and average those. This is called the Mean Absolute Error (MAE):
A MAE of 4.2 might be good for a larger hotel but perhaps not a smaller one; it also might be good for a weeknight with strong demand but not a night with low demand. In order to normalize this metric across different situations, we should convert it to a percentage. This is called the Mean Absolute Percent Error (MAPE) and can be illustrated as follows:
|Actual||Forecast||Difference/Error||Absolute Difference||% of Actual|
MAPE is simple to calculate and easy to understand, so it is an ideal metric for measuring and benchmarking forecast accuracy. Still, you’ll have a number of questions to answer as you decide to incorporate MAPE into your Revenue Management solution’s evaluation and ongoing monitoring:
- What level of granularity do I want to use for measuring my forecast error? For instance, do I separate days of week and business segments, or roll them together?
- What is my standard for “days out?” In other words, given that I update my forecast daily for every future date for my business, how far in advance of my transaction dates am I concerned with my forecast accuracy?
- Do I take my capacity constraints into account when looking at forecasts and actuals, or do I evaluate accuracy on an unconstrained basis? While the former is more straightforward, the latter may be a better indicator of where your business had the best yielding opportunities.
In summary, when evaluating any revenue management solution, it’s likely that you’ll want to know how it will make your job easier, about the slick ways that it can draw a booking curve, and how it seemingly leverages “big data.” After all, big data is trendy, right? But if you truly do care about maximizing profit for your organization, then hone in on the performance of the “engine” that will get you there, and ask about forecast accuracy.