Judgmental Adjustments of Algorithmic Hotel Occupancy Forecasts: Does User Override Frequency Impact Accuracy at Different Time Horizons?

Abstract

Judgmental adjustments of algorithmic predictions with the aim of improving demand forecast accuracy are a common revenue management practice. While empirical evidence on the impact of these user overrides is growing, little research attention has been given to the time horizon and the frequency in which these adjustments take place. Utilizing a multilevel regression model for repeated measures, 20,081,973 forecasts comprising seven different time horizons were analyzed. Data were collected from 1752 hotels of different hotel types belonging to 232 hotel chains in seven geographical regions. We find that the accuracy of algorithmic computer forecasts improves as time nears the date of stay and that the number of user overrides impacts this accuracy. The effect of the override frequency depends on the type of the forecasted demand and on the presence of special events. A higher number of user overrides is beneficial for group segment, but damaging for the transient segment. During special events periods, override frequency enhances accuracy.

Keywords

Hotel revenue management system forecasting demand occupancy overrides judgmental adjustment time horizon error accuracy

Introduction

Occupancy forecasts are an essential element of hotel revenue management (RM) practice (Koupriouchina et al., 2014). The effectiveness of daily decisions, such as the dynamic setting of room rates, the allocation of room inventory to price levels and to distribution channels, and the determination of overbooking levels and cancelation fees, to a great extent, relies on accurate forecasts (Talluri and Van Ryzin, 2004). A common procedure, designed to improve forecast accuracy is for hotel revenue managers to override the computer-generated forecasts. That is, managers evaluate the algorithmic occupancy predictions, and then modify and adjust the numbers to reflect their knowledge and beliefs. These overrides are the most common way by which managers incorporate expert knowledge (i.e., information not observable to the algorithm), and it is therefore a valuable user option in computerized RM systems (Alvarado-Valencia et al., 2017).

In this context, time ahead of a date of stay—the forecasting horizon—is an important element of hotel RM optimization because both hotels and their guests make time-dependent decisions (Chen and Schwartz, 2008; Arenoe and Van der Rest, 2020). A thorough understanding of how demand is distributed over time in order to accurately forecast hotel occupancy along the forecasting horizon is necessary for optimal hotel RM, as daily occupancy reforecasts are used to update room allocation policies and price optimization decisions (Schwartz and Hiemstra, 1997). And yet, few studies have explored the role of time in subjective user overrides of these algorithmic predictions.

In this paper, we examine the effect of the time horizon and the number of judgmental adjustments on forecast accuracy. We first investigate whether algorithmic predictions improve over the time horizon at all. That is, we seek to determine whether for each future date of stay, the accuracy of machine-generated reforecasts—that is, forecast updates at narrowing forecast horizons—improves as the date of stay nears. We then examine the effect of the number of user overrides on forecasting accuracy for each date of stay, given the different forecasting horizons.

Literature review and hypotheses

Time horizon

Hotel occupancy forecasting is a complex and dynamic process where reforecasting over the time horizon is a continuous process. That is, the occupancy forecasts for each single day of the year are constantly updated and regenerated over the forecasting horizon as the date of stay approaches. The literature on forecasting shows that the shorter the forecasting horizon, the more accurate the forecast. In his seminal paper, summarizing the findings of decades of research on forecasting, Makridakis (1986: 34) provides an intuitive explanation:

“Inertia in business and economic trends makes short-term forecasting easier…As the time horizon of forecasting increases so do the chances of systematic changes that can affect the future.”

We contend that in the context of hotel forecasting, there is another “technical” explanation beyond Makridakis’ “systematic changes” argument, in particular the common use of booking curves. A booking curve for a certain date of stay “starts” when the first reservation is recorded and it then accumulates reservations as they arrive. In a very long forecasting horizon (that is, many days ahead of the date of stay) the booking curve could be practically empty, that is, no reservations are recorded that early. It follows that the longer the horizon, the less the contribution from this important data resource of reservations, and consequently a less accurate forecast.

Unlike in the general forecasting literature, empirical evidence regarding the relationship between hotel demand forecasting horizon and accuracy is limited and inconclusive. Kimes (1999) investigated group forecasting accuracy at various days prior to arrival and found that the forecast error (MAPE and MAD) declined as the day of arrival approached. “In general, the MAPE averaged 40% at two months before arrival, dropped to about 30% at one month before arrival, and decreased to 10-15% on the day of arrival” (Kimes, 1999: 1106). While the findings provide support for the notion that the longer the forecasting horizon, the less accurate the forecasts, the generalizability of these results is subject to certain limitations. The study examined the group segment and excluded transient segment, used a single hotel chain’s data over a period of four months, and was carried out 20 years ago, prior to significant technological advancements in RM systems. Most importantly, the accuracy is calculated on aggregated monthly figures and not for an individual arrival date, while in reality, RM optimization decisions are made at the more granular arrival date level.

Schwartz (1998) applied the Curve Similarity model to forecast the occupancies for one hotel for 45 sample days for each of the following forecasting horizons: 1, 2, 3, 7, 14, 21, 30, 45, 60, and 99 days in advance. He found that accuracy declined with the forecasting horizon in a non-linear manner. Again, caution needs to be exercised when generalizing this study’s findings, as it was performed using a single hotel for a period of 45 days, more than 20 years ago.

Rajopadhye et al. (2001) evaluated the Holt-Winters procedure. They report that the MAD and MAPE values fluctuate as the arrival day nears, and that at 2–4 days before arrival the forecast accuracy is lower than at 14 days. Weatherford and Kimes (2003) tested seven forecasting methods on data from Choice Hotels and Marriott Hotels. Their findings indicate that the impact of the forecasting horizon on accuracy varies across the tested forecasting methods. For example, pickup and regression forecasting methods had fairly similar forecast errors (MAE) for all days to arrival, while “the performance of the multiplicative booking method deteriorated when used more than 7 days before arrival” (Weatherford and Kimes, 2003: 409). Most recently, Pereira (2016) compared the performance of a new time-series method (TBAT) with five more traditional time-series methods when forecasting daily hotel room demand, using six forecast accuracy measures (RMSE, MAPE, MdAPE, sMAPE, RelMAE, and RelRMSE) for multi-step forecasts from 1 to 56 days ahead. The findings suggest complex patterns, where a shorter horizon was not always associated with a lower forecast error.

To summarize, the hospitality literature provides limited support for the notion that longer horizons of hotel demand forecasts are less accurate, and that the relationship is non-linear. It is unsurprising that to date, this support has been only partial. Very few hotel-specific published studies are directly relevant, and most were not designed specifically to answer this question. With the exception of Kimes (1999), it appears that all of these studies are based on observations using predictions generated by the researchers as part of a test of newly proposed forecasting models, and not on “real” predictions generated by the hotels as part of their ongoing operation (e.g., Chen and Kachani, 2007; El Gayar et al., 2011; Haensel and Koole, 2011; Law, 2004).

This limitation is of significance in the context of the horizon/accuracy issue, as most of the reported results used only one of the two main sources of information: historical stays or reservations on hand. It has been established that the forecast combinations often used by RM systems are superior (e.g., Andrawis et al., 2011; Claeskens et al., 2016; Clemen, 1989; Hsiao and Wan, 2014; Jaganathan and Prakash, 2020; Schwartz et al., 2021) and that it is, at least partly, due to the fact that the sources’ effectiveness, or accuracy, differ across the time horizon. Each source can potentially “balance” out relative weakness (inaccuracy) of the other source to various degrees as time nears the date of stay. Specifically, reservation-data based forecasting models, from the simple pickup methods to the more advanced KNN (K nearest neighbor) and ANN, (artificial neural networks) are theoretically and practically bound to generate more accurate forecasts the closer it is to the date of stay. The intuition is rather straightforward. The closer it is to the date of stay the more that number on the books is closer to the “actual” and the shape of the booking curve is more informative because it has more data points. Perhaps the best illustration of this difference is to consider predicting occupancy a day ahead of the date of stay. On that day (the shortest forecasting horizon) the number of reserved rooms is most often very close to the actual since very few rooms are typically added on the date of stay.

As such, previous work that tested the time/accuracy question using a single source instead of the best practice approach of multiple combined sources, and most often looked at time-series models applied to actual stays, may have led to biased inconclusive results. Therefore, we formulated the following hypothesis:

H₁: The longer the forecasting horizon, the less accurate the computer-generated forecast.

User overrides

Integrating subjective human judgment in the RM system is common practice in the hotel industry (e.g., Koupriouchina et al., 2014; Schwartz, 1998; Schwartz and Cohen, 2004), though the literature on the effect of these subjective forecast adjustments is scarce. Borrowed from airline forecasting, the best practice of hotel occupancy forecasts combines three sources of information: a forecast based on data from booking curves (i.e., Reservation on Hand), time-series predictions based on historical stays, and exogenous information (e.g., Belobaba, 1987; Haensel and Koole, 2011; Schwartz and Hiemstra, 1997). User overrides of algorithmic predictions, that is, humans subjectively adjusting computer-generated predictions, is the third prong of the forecast combinations approach to hotel daily occupancy forecasting. In this way, exogenous and relevant information, that is the knowledge beyond “what the system knows,” can be added. Schwartz et al. (2016: 270) explain that the trigger to override often includes the presence of additional knowledge, exogenous to the system, such as:

“a large new convention in the destination, a large marketing campaign, the opening of a new tourism attraction, a reduction in the attractiveness of competing tourism attractions due to new circumstances, changes in the carrying capacity of airlines flying to the destination, ongoing or anticipated health/political/climate crisis, … an abrupt change in hotel reputation or hotel capacity.”

One approach to incorporating this additional information is to “inform the computer system,” by adding (flagging) a special event in the system and indicating the anticipated impact of this event (e.g., an increase of 3%). Another common practice, though not necessarily optimal, is to judgmentally adjust the system forecast. Several studies have found support for the practice of subjective “intervention,” noting that statistical forecasts can be made more accurately if experts judgmentally adjust them (Donihue, 1993; Fildes et al., 2009; McNees, 1990; Turner, 1990). The underlying cognitive mechanisms behind judgmental adjustments are, however, complex, and it is argued that research so far has only “touched the tip of the iceberg” (Gönül, 2013). Moreover, a number of studies have demonstrated that forecasters often make unnecessary judgmental adjustments (Lawrence et al., 2006). Interestingly, training in forecasting methods does not seem to improve forecasters’ ability to extrapolate, and forecasters continued to make adjustments even when informed they were reducing accuracy (Goodwin and Wright, 1994; Lim and Oconnor, 1995). Research on “big losses” (i.e., judgmental adjustments that significantly reduced the forecasting accuracy) also identified an interesting phenomenon—after the first big loss, forecasters were still more likely to make another large adjustment, even though it increased the probability of a second big loss (Petropoulos et al., 2016).

Many of these findings are in the context of Stock-Keeping Unit (SKU) forecast adjustments so their applicability to hotel occupancy forecasting should be interpreted with caution; there is a substantial difference between hotel RM forecasting and SKU-based forecasting. For example, users of a forecast sometimes adjust the forecast in order to “control” the forecast (Gönül, 2013). In this regard, the SKU forecast creator and the forecast user are different, while in hotel RM, managers are usually co-creators of the forecast as they are involved in the setting up of forecast parameters by mapping special events in the system and indicating their impact on the forecast. Should substantial and consistent discrepancies be found in forecasting, revenue managers are encouraged to contact the system provider, “open a case,” and work directly with the system provider on investigating inconsistencies and re-calibrating the model (IDeaS, personal communication, July 2016). Thus, in contrast to managers in an SKU forecasting context, hotel revenue managers may be less inclined to make forecast adjustments solely in a pursuit of gaining the sense of control or ownership.

Several publications have explored user interventions in airline forecasting, a business operational context which has many similarities with hotel forecasting. An experiment at Northwest Airlines where a subset of markets was put on autopilot (not allowing human interventions to the forecasts) found that it led to a decrease in revenue. Bach (1999) reported that analysts increased revenue by 7.9%. At America West airline, a study demonstrated the possibility to identify and correct flights with strong negative bias (Loew, 2000). At US Airways, Zeni (2003) estimated that revenue analysts’ interventions added up to 3% incremental revenue. A study by Mukhopadhyay et al. (2007) demonstrated that user interventions were valuable, especially when competitors added flights to certain routes. Finally, based on a comprehensive review of RM forecasting models, Weatherford (2016) concluded that human judgment in airline RM forecasting is indispensable.

Multiple tourism studies explored judgmental forecasting in general, and the more particular question of subjective adjustments of algorithmic forecasts. Recently, Lin (2019) demonstrated that expert adjustments of forecast of tourist arrivals to Hong Kong proved more beneficial when the algorithmic predictions had large variability. Similarly, earlier studies by Song et al. (2013), Lin (2013) and Lin et al. (2014) provide support to the notion of increased in accuracy when time-series algorithmic predictions of tourists’ arrivals are adjusted by expert judgments through a Delphi survey.

Despite its importance, little attention has been paid to judgmental adjustments in the context of hotel RM forecasting. Kimes (1999) investigated the effect of forecast updates, using four months of group forecast data of a single hotel chain in North America. The study found that the “frequency of updates was associated with increased forecast accuracy from 0-11 days before arrival” (Kimes, 1999: 1109). Other studies (Koupriouchina et al., 2014; Schwartz and Cohen, 2004) demonstrated that managerial overrides are beneficial though prone to biases, akin to claims in the generic forecasting literature (e.g., Bearden et al., 2008; Fildes et al., 2009). Most recently, a study by Schwartz et al. (2021) indicates that as learning occurs—as the computer learns and improves its forecasts—the effectiveness of overrides declines. In other words, the ability of subjective adjustments to improve the machine’s predictions diminishes as the accuracy of the machine predictions improves over time.

In this context, it is important to consider the typical characteristics of effective hotel RM practice where the demand for the same date of stay is re-forecasted as time passes. These repeated forecasts are necessary because they support decisions on how to optimally adjust RM controls in dynamic markets. It follows that hotel revenue managers repeatedly adjust, or override, these repeated forecasts as they deem necessary. This leads to the question, is repetition—the frequency of overrides—associated with accuracy? In other words, is an override likely to be more (or less) accurate because it was performed multiple times in the past for the same date of stay?

There are two opposing arguments in this regard. On the one hand, if the forecast for the same date of stay has been overridden multiple times, it might indicate a higher perceived volatility, a higher uncertainty, or an inability of the algorithm to adjust to new information and dynamic market conditions, hence the need to repeat overriding. The more difficult and challenging the situation, the less likely the override will succeed in fully correcting the algorithmic prediction and generate an accurate forecast. As such, it is reasonable to expect that a high frequency of overrides might be associated with a less accurate computer forecast. On the other hand, a higher frequency of revenue managers’ interventions might also indicate that the revenue manager is more competent, as they continuously monitor and assess the computer output and are on “standby,” ready to adjust things as necessary. This might be a signal of “high quality” and therefore indicate that a higher number of overrides (superior RM practices) is associated with more accurate forecasts. In lieu of prior research or empirical evidence on the topic, it is, however, difficult to assess which argument prevails, and consequently our second hypothesis is non-directional:

H₂: The frequency of judgmental forecast adjustments is associated with different forecast accuracy (higher or lower).

Method

Data

A major Revenue Management System (RMS) provider, serving over 1.6 million hotel rooms in 124 countries, provided the anonymized data. A protocol, logical checks, and detailed code books were established to ensure correct interpretation and understanding of the extracted data as well as their underlying forecasting processes. From the 196 variables available, 15 were found to be applicable for testing the hypotheses. From the 25,003,446 cases obtained, 1,718,856 (6.8%) were incomplete. A further 3,202,617 (12.8%) cases were removed during data checks, bringing the final sample to 20,081,973 (80.3%) cases. Observations were excluded from the study for two reasons: missing data and time-horizon data incompatibility. The IT department of the RMS provider couldn’t clarify the reasons for the missing data. Possible reasons include corrupt files, software glitches, system upgrades and user errors. The second reason was time-horizon incompatibility. Not all subjective predictions’ horizons, that is, how far in advance they were made, matched the algorithmic predictions’ horizon. However, the measures of the hypothesized accuracy improvement due to user override must be horizon specific. That is, both the algorithmic prediction and the user override must be done at the same day and for the same future date of stay. Accordingly, these observations with no “counterpart” had to be removed.

The dataset contained forecast data from 24 March 2014 to 7 April 2017. These data were extracted at the business type level, separate for group and transient business, at seven forecast horizons (i.e., 0, 1, 3, 7, 21, 84, and 252 days prior to arrival). For each hotel at each occupancy date, there were fourteen cases (7 measurements × 2 segments). Taking short and long-term perspectives into consideration, the forecast horizons were selected in consultation with the RMS provider’s senior data experts as the existing literature did not provide indications, guidelines or justifications for choosing specific forecast horizons. The dataset covered forecasting data of 1752 hotels belonging to 232 hotel chains with a variety of hotel types, from seven geographical regions (see Table 1). The hotels ranged in size from 6 to 4028 rooms, with an average size of 242 (see Table 2). The majority of hotels were city hotels.

Table 1.

Number of hotels by region and type.

	Asia		Australasia		Central America		Europe		Middle East & Africa		North America		South America
	n	Column%	n	Column%	n	Column%	N	Column%	n	Column%	n	Column%	n	Column%
Airport	2	0.7	1	2.4	0	0.0	21	2.1	0	0.0	2	0.8	0	0.0
Casino	1	0.4	7	16.7	0	0.0	0	0.0	4	2.5	15	5.7	0	0.0
Casino—Native American	0	0.0	0	0.0	0	0.0	0	0.0	0	0.0	2	0.8	0	0.0
Casino—Resort	2	0.7	0	0.0	0	0.0	0	0.0	4	2.5	5	1.9	0	0.0
City—Capital	91	33.3	4	9.5	0	0.0	259	26.0	43	26.7	15	5.7	1	6.3
City—Major	87	31.9	14	33.3	0	0.0	350	35.2	64	39.8	155	59.2	15	39.8
City—Tertiary	49	17.9	6	14.3	0	0.0	293	29.4	5	3.1	11	4.2	0	0.0
Resort—Destination	20	7.3	9	21.4	1	33.3	18	1.8	15	9.3	15	5.7	0	0.0
Resort—Golf	0	0.0	0	0.0	0	0.0	8	0.8	3	1.9	4	1.5	0	0.0
Resort—Other	5	1.8	0	0.0	0	0.0	15	1.5	9	5.6	9	3.4	0	0.0
Resort—Seaside	16	5.9	1	2.4	2	66.7	24	2.4	14	8.7	23	8.8	0	0.0
Resort—Ski	0	0.0	0	0.0	0	0.0	7	0.7	0	0.0	6	2.3	0	0.0
Total	273	100	42	100	3	100	995	100	161	100	262	100	16	100

Table 2.

Hotel size by hotel type.

Hotel type	n	%	Size
Hotel type	n	%	Mean	Minimum	Maximum	Standard deviation
Airport	26	1.5	319	132	821	147
Casino	27	1.5	383	77	1960	353
Casino—Native American	2	0.1	358	301	416	81
Casino—Resort	11	0.6	1366	140	4028	1433
City—Capital	413	23.6	273	28	1261	194
City—Major	685	39.1	242	33	1590	173
City—Tertiary	364	20.8	162	9	1052	139
Resort—Destination	78	4.5	230	30	1483	233
Resort—Golf	15	0.9	218	6	401	117
Resort—Other	38	2.2	186	25	499	106
Resort—Seaside	80	4.6	249	30	789	161
Resort—Ski	13	0.7	271	59	766	220
Total	1752	100.0	242	6	4028	230

Dependent variables

Error measures are known to yield contradicting results (Davydenko and Fildes, 2013; Koupriouchina et al., 2014; Makridakis, 1993; Pereira, 2016; Trapero et al., 2011), so we followed recommendations by Koupriouchina et al. (2014) in selecting three scale-independent forecast accuracy measures as dependent variables. First, given the widespread use of percentage-based accuracy measures, the non-aggregated APE (Absolute Percentage Error) was selected to allow for comparison with existing work. Second, as APE is not symmetric and rewards under-forecasts (e.g., Armstrong and Collopy, 1992; Foss et al., 2003; Gneiting, 2011; Makridakis, 1993), its symmetric equivalent sAPE (symmetric Absolute Percentage Error) was chosen as a second measure even though Goodwin and Lawton (1999) and Koehler (2001) showed that it is not invariably symmetric. Third, the relative measure LnQ (log “forecast to actual” ratio) was selected as it is dimensionless, produces less skewed results than sAPE, and is thus considered superior to APE (Tofallis, 2015).

Independent variables

Two independent variables were selected. First, days to arrival (DTA) was selected at seven forecast horizons (values: 0, 1, 3, 7, 21, 84, 252 days prior to arrival). Second, the frequency of judgmental adjustments made for each specific day of arrival was selected. This continuous variable, which ranged from 0 to 28, was labeled as the number of overrides (numOVER).

In addition, five control variables were selected, two at within-hotel and three at hotel level. The within-hotel level variables included business type and special events, coded as BusinessType (0 = transient, 1 = group) and SpecialEvent (0 = no special event, 1 = special event). The business type control variable was included because of differences in the characteristics of transient and group demand. Special events were included due to their effect on the reliability of forecasts based on historical data, and, thus, accuracy. The hotel-level variables pertained to hotel size, type, and region. HotelSize is a hotel’s capacity in rooms ranging from 6 to 4028. As it does not have a meaningful zero, it was centered on its grand mean. HotelType comprised a variable with aggregated hotel type categories: airport hotels, casino hotels, city hotels, and resort hotels. This variable was coded as a dummy variable, with city hotels as the reference category. Both hotel size and hotel type were added as hotel-level control variables, as previous studies have shown these to be determinants of hotel performance (Claver-Cortés et al., 2007; Pine and Phillips, 2005). HotelRegion had the following categories: Asia, Australasia, Central America, Europe, Middle East and Africa, North America, and South America. It was also coded as a dummy variable, with Europe as the reference category. This variable was included as the development of revenue management might be different in geographical regions, as explained by Kimes (2008). She found that respondents from Asia are more likely to believe that the various technical aspects of RM are in need of improvement compared to North American respondents. She suggested that this difference could be explained by the slower development of revenue management in Asia compared to that in North America.

Analytical approach

A variation of multilevel analysis commonly described as repeated measures design or longitudinal data multilevel analysis was applied (Heck et al., 2014; Hox et al., 2018). Forecasting accuracy at DTA was considered at level 1, nested within hotels (level 2). Two subscripts to describe hotels (i) and occasions of measurement (t) were used. It was assumed that the observed status Y_ti at time t for hotel i is a function of a systematic change trajectory plus random error. We followed Hox et al. (2018) five-step multilevel modeling approach for repeated measures data.

First, an intercept-only (null) model was developed to test whether the grand-mean intercept for forecast error (APE, sAPE, LnQ) varied across hotels. The repeated measures (level 1) variance was estimated as 36,399.14, and the hotel-level (level 2) variance as 361.82 (Wald Z = 29.448, p < .001). The intra-class correlation (ICC) at the hotel level was estimated for APE as $ρ = 39.73 / (1,006.87 + 39.73) = 0.98 %$ . The ICC for sAPE was 3.8% and for LnQ, 3.3%. As these ICCs were relatively low despite the nested structure of the data, the ICC for Absolute Error (AE) was also determined: 16.1% ( $ρ = 127.84 / (665.36 + 127.84) = 0.16$ ), suggesting that multilevel analysis was appropriate as there was variation in intercepts across hotels.

Second, we added five level-1 (within-hotel) variables to the model, three explanatory (DTA, DTA quadratic, numOVER) and two control variables (BusinessType, SpecialEvent).¹ Starting with DTA, the predictors were added one at a time. Whereas DTA quadratic was found significant, its effect was very small. For reasons of parsimony, it was excluded. Moreover, models were formulated to explore interactions between the level-1 variables. For example, as demand patterns can differ during special events, it was verified whether the relationship between number of overrides and the improvement in forecasting accuracy depended on the presence or absence of a special event.

Third, three level-2 (hotel-level) control variables were added to the level-1 model of step 2. Except for a random intercept and the level-1 residual, all variables were still treated as fixed (i.e., coefficients were not allowed to differ across hotels).

Fourth, we assessed whether the intercept and linear component (DTA) varied randomly across hotels (i.e., forecasting accuracy and the coefficient of DTA varied across hotels). To assess whether forecast errors evolved differently over time for the different hotels, seventeen fixed effects, one random intercept ( $u_{0 i}$ ), the hotel-level residual of the slopes of the DTA ( $u_{1 i})$ , the level-1 residual ( $ℇ_{t i}$ ), and the covariance between the intercept and the slope $(u_{01}$ ) were estimated.

Finally, a cross-level interaction between HotelSize (level-2 control variable) and DTA (level-1 explanatory variable) was added. Scaled identity was selected as the level-1 covariance structure, and unstructured as level-2 covariance structure.

To evaluate all models from steps 1 to 5, improvements in the model fit were assessed with reference to a deviance statistic (−2 loglikelihood ratio), that is distributed as χ² with associated degrees of freedom, and by examining the Akaike and Bayesian Information Criteria (AIC and BIC).

Results

Descriptive statistics

Figure 1 illustrates the relationship between the mean forecast error and forecasting horizon for all forecasts. On average, the forecasting error decreases over time in a non-linear fashion.

Figure 1.

Relationship between mean forecast accuracy and days to arrival.

Table 3 reports the breakdown of forecast errors per days to arrival (DTA), with and without overrides. Standard deviation values indicate a large dispersion. It can be observed that errors are smaller for forecasts with overrides for all DTAs except DTAs 0 and 1.

Table 3.

Means and standard deviations of absolute percentage error, symmetric absolute percentage error and LnQ by days to arrival.

	DTA
	0	1	3	7	21	84	252
APE
No overrides	6.26 (40.79)	6.99 (39.59)	12.96 (62.44)	22.19(102.79)	39.79 (175.25)	68.82 (279.38)	85.52 (358.39)
Overrides	9.12 (63.61)	9.09 (64.9)	11.68 (68.06)	14.76 (72.71)	22.84 (96.15)	41.08 (176.02)	52.54 (235.95)
All	6.30 (41.22)	7.02 (40.08)	12.94 (62.53)	22.08 (102.42)	39.54 (174.36)	68.41 (278.15)	85.03 (356.91)
sAPE
No overrides	5.86 (16.62)	6.06 (12.14)	10.38 (17.76)	16.26 (24.18)	25.96 (32.36)	38.73 (40.43)	44.48 (43.35)
Overrides	7.23 (17)	6.62 (13.21)	9.07 (15.2)	11.78 (18.27)	17.81 (25.01)	27.48 (34.07)	32.16 (37.13)
All	5.88 (16.62)	6.07 (12.16)	10.36 (17.72)	16.19 (24.11)	25.84 (32.28)	38.57 (40.36)	44.29 (43.29)
LnQ
No overrides	0.06 (0.23)	0.06 (0.14)	0.11 (0.21)	0.17 (0.3)	0.28 (0.41)	0.44 (0.55)	0.51 (0.6)
Overrides	0.08 (0.21)	0.07 (0.16)	0.09 (0.18)	0.12 (0.22)	0.19 (0.32)	0.3 (0.45)	0.36 (0.51)
All	0.06 (0.23)	0.06 (0.14)	0.11 (0.21)	0.17 (0.3)	0.28 (0.41)	0.43 (0.55)	0.51 (0.6)

Note. Standard deviations are in parentheses.

On average, at forecasting horizons larger than one (DTA 3, 7, 21, 84, 252), forecasts with overrides are more accurate than those without. A breakdown of the forecast accuracies for each hotel type is shown in Table 4.

Table 4.

Forecast errors by hotel type.

Hotel type	APE				sAPE				LnQ
Hotel type	Mean	Min	Max	SD	Mean	Min	Max	SD	Mean	Min	Max	SD
Airport	30.33	0	15900	147.26	20.81	0	198.58	30.54	0.23	0	5.63	0.39
Casino	33.56	0	15400	183.54	19.84	0	197.71	30.81	0.22	0	5.16	0.40
Casino—Native American	33.83	0	4600	143.78	21.08	0	195.12	35.51	0.24	0	4.39	0.48
Casino—Resort	47.47	0	110400	525.92	22.02	0	199.68	31.87	0.24	0	7.13	0.41
City—Capital	31.45	0	36300	172.69	19.86	0	198.90	31.62	0.22	0	5.90	0.41
City—Major	35.54	0	48100	195.80	21.58	0	199.43	32.95	0.24	0	6.56	0.43
City—Tertiary	35.69	0	20800	182.55	21.77	0	198.90	32.26	0.24	0	5.89	0.42
Resort—Destination	30.76	0	17500	169.04	19.74	0	198.92	31.72	0.22	0	5.91	0.41
Resort—Golf	45.78	0	20600	306.76	21.31	0	198.08	33.66	0.24	0	5.33	0.45
Resort—Other	37.70	0	35600	221.88	22.40	0	198.88	32.59	0.25	0	5.88	0.42
Resort—Seaside	33.80	0	14500	203.67	18.72	0	198.94	31.58	0.21	0	5.93	0.41
Resort—Ski	31.76	0	20000	180.94	19.00	0	198.02	30.72	0.21	0	5.30	0.40

Multilevel analysis

Tables 5–7 provide overviews of the multilevel modeling steps per accuracy measure with APE, sAPE, and LnQ as the dependent variables. Models 1–7 assume fixed effects. Models 8 and 9 are random coefficient models which show whether forecast errors evolve differently over time for the different hotels (i.e., whether DTA has a significant variance component).

Table 5.

Results of the multilevel regression models for APE.

Model:	Step 1: Null model		Step 2: Level-1 predictors										Step 3: Level-2 predictors		Step 4: Random coefficient		Step 5: Cross-level interactions
Model:	M₁: Intercept only		M₂: DTA		M₃: DTA and DTA^2		M₄: number of overrides		M₅: Level-1 model with three predictors		M₆: Level -1 model with interactions		M₇: final level-1 and 2 mode		M₈: random DTA		M₉: Cross-level interaction of DTA and hotel size
Fixed part
Intercept	33.07***	(0.46)	17.48***	(0.45)	9.98***	(0.45)	17.59***	(0.45)	−0.77	(0.44)	−0.80	(0.4)	−3.31***	(0.59)	−2.22***	(0.41)	−2.13***	(0.41)
DTA			0.30***	(4.9 × 10⁻⁴)	0.94***	(2.3× 10⁻³)	0.30***	(4.9× 10⁻⁴)	0.30***	(4.9× 10⁻⁴)	0.30***	(4.9× 10⁻⁴)	0.30***	(4.9 × 10⁻⁴)	0.29***	(4.1× 10⁻³)	0.30***	(4.0× 10⁻³)
DTA quadratic					−0.003***	(8.9× 10⁻⁶)
numOVER							−2.41***	(0.09)	0.22*	(0.09)	0.53***	(0.10)	0.53***	(0.10)	0.53***	(0.10)	0.53**	(0.10)
SPECIAL									2.87***	(0.15)	2.90***	(0.15)	2.90***	(0.15)	2.89***	(0.15)	2.89***	(0.15)
BusinessType									44.45***	(0.09)	44.53***	(0.09)	44.53***	(0.09)	44.37***	(0.09)	44.37***	(0.09)
numOVER × SPECIAL											−0.17	(0.24)	−0.17	(0.24)	−0.17	(0.24)	−0.17	(0.24)
numOVER × BusinessType											−4.57***	0.32	−4.58***	0.32	−4.56***	0.32	−4.56***	0.32
Hotel size													−2.7× 10⁻³	(2.0× 10⁻³)	−0.02***	(1.3× 10⁻³)	−0.01***	(1.4× 10⁻³)
Hotel type
Airport vs city													−3.95	(3.42)	1.48	(2.28)	1.48	(2.28)
Casino vs city													5.66	(3.02)	7.40***	(2.09)	7.36***	(2.09)
Resort vs city													−0.69	(1.30)	−4.08***	(0.87)	−4.09***	(0.87)
Hotel Region
Asia vs Europe													12.71***	(1.25)	12.32***	(0.83)	12.32***	(0.84)
Australasia vs Europe													−8.83**	(2.78)	−7.60***	(1.85)	−7.59***	(1.85)
Central America vs Europe													−3.66	(10.06)	0.37	(6.74)	0.37	(6.74)
Middle East & Africa vs Europe													9.82***	(1.51)	8.15***	(1.01)	8.15***	(1.01)
North America vs Europe													−2.05	(1.28)	−4.59	(0.87)	−4.59***	(0.87)
South America vs Europe													9.49*	(4.37)	10.14**	(2.97)	10.16**	(2.97)
DTA × Hotel Size																	1.7× 10⁻⁴***	(1.7× 10⁻⁵)
Random part
$σ_{e}^{2}$	36,399.14***		35,736.94***		35,586.11***		35,735.66***		35,401.73***		35,401.38***		35,401.38***		35,190.14***		35,190.14***
$σ_{u 0}^{2}$	361.82***		355.92***		354.10***		355.49***		323.16***		323.17***		293.93***		148.81***		147.71***
$σ_{u 1}^{2}$															0.03***		0.03***
$σ_{u_{01}}$															0.79***		0.75***
−2*log likelihood	267,905,206.2		267,536,513.5		267,451,597.5		267,535,793.1		265,001,167.3		265,000,967.8		265,000,773.2		264,887,469.0		264,887,389.1
AIC	267,905,210.2		267,536,517.5		267,451,601.5		267,535,797.1		265,001,171.3		265,000,971.8		265,000,777.2		264,887,477.0		264,887,397.1
BIC	267,905,239.8		267,536,547.1		267,451,631.1		267,535,826.8		265,001,201.0		265,001,001.4		265,000,806.8		264,887,536.2		264,887,456.4

Note. Standard errors are in parentheses.

DTA: days to arrival; numOVER: number of overrides. BusinessType (0 = transient, 1 = group), SPECIAL (0 = no special event, 1 = special event)

*p < .05, ** p < .01, ***p < .001

Table 6.

Results of the multilevel regression models for sAPE.

Model	Step 1: Null model		Step 2: Level-1 predictors										Step 3: Level-2 predictors		Step 4: Random coefficient		Step 5: Cross-level interactions
Model	M₁: Intercept only		M₂: DTA		M₃: DTA and DTA^2		M₄: number of overrides		M₅: Level-1 model with three predictors		M₆: Level -1 model with interactions		M₇: final Level-1 and 2 model		M₈: random DTA		M₉: Cross-level interaction of DTA and hotel size
Fixed part
Intercept	20.50***	(0.15)	13.11***	(0.15)	8.95***	(0.15)	13.16***	(0.15)	6.46***	(0.14)	6.46***	(0.14)	5.36***	(0.18)	5.47***	(0.17)	5.47***	(0.17)
DTA			0.14***	(7.6 × 10⁻⁵)	0.50***	(3.4 × 10⁻⁴)	0.14***	(7.6 × 10⁻⁵)	0.14***	(7.4 × 10⁻⁵)	0.14***	(7.4 × 10⁻⁵)	0.14***	(7.4 × 10⁻⁵)	0.14***	(9.0 × 10⁻⁴)	0.14***	(9.0 × 10⁻⁴)
DTA quadratic					−1.44 × 10⁻³***	(1.3 ×10⁻⁶)
numOVER							−0.96***	(0.01)	0.04**	(0.01)	0.13***	(0.01)	0.13***	(0.01)	0.13**	(0.01)	0.13***	(0.01)
SPECIAL									0.09***	(0.02)	0.10***	(0.02)	0.10***	(0.02)	0.10***	(0.02)	0.10***	(0.02)
BusinessType									16.53***	(0.01)	16.56***	(0.01)	16.56***	(0.01)	16.53***	(0.01)	16.53***	(0.01)
numOVER × SPECIAL											−0.09**	(0.04)	−0.09**	(0.04)	−0.09**	(0.04)	−0.09**	(0.04)
numOVER × BusinessType											−1.31***	(0.05)	−1.31***	(0.05)	−1.31***	(0.05)	−1.31***	(0.05)
Hotel Size													−0.01***	(5.9 × 10⁻⁴)	−0.01***	(5.6 × 10⁻⁴)	-0.01***	(5.6 × 10⁻⁴)
Hotel type
Airport vs City													0.21	(1.02)	1.02	(0.96)	1.02	(0.96)
Casino vs City													2.03*	(0.89)	2.51**	(0.84)	2.51**	(0.84)
Resort vs City													−1.44***	(0.39)	−1.97***	(0.37)	−1.97***	(0.37)
Hotel region
Asia vs Europe													5.41***	(0.37)	5.62***	(0.35)	5.62***	(0.35)
Australasia vs Europe													−4.18***	(0.83)	−4.18***	(0.78)	−4.18***	(0.78)
Central America vs Europe													−0.88	(3.00)	−0.17	(2.83)	−0.17	(2.83)
Middle East & Africa vs Europe													5.16***	(0.45)	4.96***	(0.43)	4.96***	(0.43)
North America vs Europe													−1.07**	(0.38)	−1.11**	(0.36)	−1.11**	(0.36)
South America vs Europe													4.32**	(1.30)	4.61***	(1.22)	4.61***	(1.22)
DTA × hotel size																	5.7 × 10⁻⁶	(3.9 × 10⁻⁶)
Random part
$σ_{e}^{2}$	1,006.87***		857.98***		811.48***		857.78***		794.83***		794.80***		794.80***		786.13***		786.13***
$σ_{u 0}^{2}$	39.73***		37.98***		37.33***		37.88***		33.54***		33.54***		26.32***		23.41***		23.41***
$σ_{u 1}^{2}$															1.4 × 10⁻³***		1.4 × 10⁻³***
$σ_{u_{01}}$															−0.01		−0.01
−2*log likelihood	195,859,621.4		192,646,248.6		191,527,205.47		192,641,467.1		189,433,850.5		189,433,117.5		189,432,690.6		189,222,676.6		189,222,697.5
AIC	195,859,625.4		192,646,252.6		191,527,209.47		192,641,471.1		189,433,854.5		189,433,121.5		189,432,694.6		189,222,684.6		189,222,705.5
BIC	195,859,655.0		192,646,282.2		191,527,239.10		192,641,500.7		189,433,884.1		189,433,151.1		189,432,724.2		189,222,743.8		189,222,764.8

Note. Standard errors are in parentheses.

DTA: days to arrival; numOVER: number of overrides. BusinessType (0 = transient, 1 = group), SPECIAL (0 = no special event, 1 = special event)

*p < .05, ** p < .01, ***p < .001

Table 7.

Results of the multilevel regression models for LnQ.

Model:	Step 1: Null model		Step 2: Level-1 predictors										Step 3: Level-2 predictors		Step 4: Random coefficient		Step 5: Cross-level interactions
Model:	M₁: Intercept only		M₂: DTA		M₃: DTA and DTA^2		M₄: number of overrides		M₅: Level-1 model with three predictors		M₆: Level -1 model with interactions		M₇: final level-1 and 2 mode		M₈: random DTA		M₉: Cross-level interaction of DTA and hotel size
Fixed part
Intercept	2.3 × 10⁻¹***	(1.8 × 10^–3)	1.4 × 10^–1***	(1.8 × 10^–3)	9.3 × 10^–2***	(1.8 × 10^–3)	1.4 × 10^–1***	(1.8 × 10^–3)	5.7 × 10^–2***	(1.7 × 10^–3)	5.7 × 10^–2***	(1.7 × 10^–3)	4.3 × 10^–2***	(2.1 × 10^–3)	4.5 × 10^–2***	(1.9 × 10^–3)	4.5 × 10^–2***	(1.9 × 10^–3)
DTA			1.7 × 10^–3***	(1.0 × 10^–6)	5.7 × 10^–3***	(4.5 × 10^–6)	1.7 × 10^–3***	(1.0 × 10^–6)	1.7 × 10^–3***	(9.7 × 10^–7)	1.7 × 10^–3***	(9.7 × 10^–7)	1.7 × 10^–3***	(9.7 × 10^–7)	1.6 × 10^–3***	(1.2 × 10^–5)	1.6 × 10^–3***	(1.2 × 10^–5)
DTA quadratic					–1.6 × 10^–5***	(1.8 × 10^–8)
numOVER							–1.1 × 10***	(1.8 × 10^–4)	1.0 × 10^–3***	(1.8 × 10^–4)	2.1 × 10^–3***	(2.0 × 10^–4)	2.1 × 10^–3***	(2.0 × 10^–4)	2.2 × 10^–3***	(1.9 × 10^–4)	2.2 × 10^–3***	(1.9 × 10^–4)
SPECIAL									1.3 × 10^–3***	(3.0 × 10^–4)	1.5 × 10^–3***	(3.0 × 10^–4)	1.5 × 10^–3***	(3.0 × 10^–4)	1.4 × 10^–3***	(3.0 × 10^–4)	1.4 × 10^–3***	(3.0 × 10^–4)
BusinessType									2.1 × 10^–1***	(1.7 × 10^–4)	2.1 × 10^–1***	(1.7 × 10^–4)	2.1 × 10^–1***	(1.7 × 10^–4)	2.1 × 10^–1***	(1.7 × 10^–4)	2.1 × 10^–1***	(1.7 × 10^–4)
numOVER×SPECIAL											–1.0 × 10^–3*	(4.8 × 10^–4)	–1.0 × 10^–3*	(4.8 × 10^–4)	–1.0 × 10^–3*	(4.8 × 10^–4)	–1.0 × 10^–3*	(4.8 × 10^–4)
numOVER×BusinessType											–1.6 × 10^–2***	(6.4 × 10^–4)	–1.6 × 10^–2***	(6.4 × 10^–4)	–1.6 × 1^0–2***	(6.4 × 10^–4)	–1.6 × 1^0–2***	(6.4 × 10^–4)
Hotel size													–8.1 × 10^–5***	(7.3 × 10^–6)	–8.6 × 10^–5***	(6.6 × 10^–6)	–8.5 × 10^–5***	(6.6 × 10^–6)
Hotel type
Airport vs city													–4.6 × 10^–4	(1.2 × 10^–2)	1.1 × 10^–2	(1.1 × 10^–2)	1.1 × 10^–2	(1.1 × 10^–2)
Casino vs city													2.3 × 10^–2*	(1.1 × 10^–2)	2.9 × 10^–2**	(9.9 × 10^–3)	2.9 × 10^–2**	(9.9 × 10^–3)
Resort vs city													–1.5 × 10^–2**	(4.7 × 10^–3)	–2.3 × 10^–2***	(4.3 × 10^–3)	–2.3 × 10^–2***	(4.3 × 10^–3)
Hotel region
Asia vs Europe													6.6 × 10^–2***	(4.5 × 10^–3)	6.6 × 10^–2***	(4.1 × 10^–3)	6.6 × 10^–2***	(4.1 × 10^–3)
Australasia vs Europe													–4.8 × 10^–2***	(1.0 × 10^–2)	–4.8 × 10^–2***	(9.2 × 1^0–3)	–4.8 × 10^–2***	(9.2 × 10^–3)
Central America vs Europe													–1.1 × 10^–2	(3.7 × 10^–2)	1.6 × 10^–3	(3.3 × 10^–2)	1.6 × 10^–3	(3.3 × 10^–2)
Middle East & Africa vs Europe													6.2 × 10^–2***	(5.5 × 10^–3)	5.6 × 10^–2***	(5.0 × 10^–3)	5.6 × 10^–2***	(5.0 × 10^–3)
North America vs Europe													–1.0 × 10^–2*	(4.6 × 10^–3)	–1.1 × 10^–2*	(4.2 × 10^–3)	–1.1 × 10^–2*	(4.2 × 10^–3)
South America vs Europe													5.2 × 10^–2**	(1.6 × 10^–2)	5.4 × 10^–2***	(1.4 × 10^–2)	5.4 × 10^–2***	(1.4 × 10^–2)
DTA × hotel size																	8.0 × 10^–8	(5.1 × 10^–8)
Random part
$σ_{e}^{2}$	1.7 × 10^–1***		1.5 × 10^–1***		1.4 × 10^–1***		1.5 × 10^–1***		1.4 × 10^–1***		1.4 × 10^–1***		1.4 × 10^–1***		1.4 × 10^–1***		1.4 × 10^–1***
$σ_{u 0}^{2}$	5.7 × 10^–3***		5.5 × 10^–3***		5.4 × 10^–3***		5.5 × 10^–3***		4.9 × 10^–3***		4.9 × 10^–3***		3.9 × 10^–3***		3.2 × 10^–3***		3.2 × 10^–3***
$σ_{u 1}^{2}$															2.4 × 10^–7***		2.4 × 10^–7***
$σ_{u_{01}}$															6.5 × 10^–7		6.5 × 10^–7
−2*log likelihood	21,299,003.3		18,736,705.4		17,920,563.0		18,732,806.9		17,209,936.4		17,209,338.3		17,209,022.6		16,997,942.6		16,997,971.9
AIC	21,299,007.3		18,736,709.4		17,920,567.0		18,732,810.9		17,209,940.4		17,209,342.3		17,209,026.6		16,997,950.6		16,997,979.9
BIC	21,299,036.9		18,736,739.0		17,920,596.6		18,732,840.5		17,209,970.0		17,209,371.9		17,209,056.2		16,998,009.9		16,998,039.2

Note. Standard errors are in parentheses.

DTA: days to arrival; numOVER: number of overrides. BusinessType (0 = transient, 1 = group), SPECIAL (0 = no special event, 1 = special event)

*p < .05, ** p < .01, ***p < .001

Considering model 9 in Table 6, a value of 5.47 for DTA=0 is found, and this value increases by 0.14 with each additional day. In other words, forecast error increases when DTA increases, and vice versa. This supports Hypothesis 1 (forecast accuracy improves as date of arrival nears). Furthermore, to explore whether this change in DTA is impacted by hotel size, a cross-level interaction between hotel size and DTA was estimated. A small and insignificant effect can be observed, suggesting that the impact of DTA on forecast error is independent of hotel size.

Focusing on the effect of the number of user overrides on forecasting accuracy, it can be observed that for each additional override, forecast error increases by 0.13. However, when the within-hotel level control variable business type is considered, a more nuanced picture emerges. That is, the effect of override number on accuracy depends on (i.e., is moderated by) the business type. As the positive coefficient shows, the forecast error (sAPE) for group business is, on average, 16.53 higher than the forecast error for transient business. However, the negative coefficient of the interaction term (numOVER × BusinessType) indicates that for group business, the forecast error decreases by 1.31 with each additional user override. To a much lesser degree this also applies to special events, inasmuch that in the presence of a special event the forecast error decreases by 0.09 with each user override. This provides partial support for Hypothesis 2 (hotels with a higher frequency of forecast overrides have higher forecast accuracy). A higher number of user overrides is beneficial for group business, but damaging for the transient segment.

Analyzing the hotel-level control variables, hotel size is found to positively impact forecast accuracy. Moreover, resort hotels have higher and casino hotels have lower forecast accuracy than city hotels. Airport hotels and city hotels do not differ in forecast accuracy. Hotels in North America and Australasia have a higher forecast accuracy than hotels in Europe, which in turn have a higher accuracy than hotels in Asia, Middle East and Africa, and South America. There is no difference in the forecast accuracy of hotels in Europe and in Central America.

The findings are robust for all three accuracy measures (APE, sAPE, LnQ), as shown in Tables 5–7. Comparing the successive five modeling steps, each subsequent and more complex model (1–9) is preferred based on the χ²—difference test and the AIC and BIC statistics.

Discussion

The findings of this large-scale investigation suggest that the accuracy of hotel occupancy forecasts improves as time nears the date of stay, and that user override frequency impacts forecasting accuracy depending on the business segment. Frequent overrides to the group segment positively affect the forecasting accuracy at different forecasting time horizons. The opposite holds for the transient segment. The study thereby contributes to the growing interest in the impact of judgmental adjustments in forecasting, in particular across time horizons.

Similar to Franses and Legerstee (2011) who also examined SKU forecasting, Van den Broeke et al. (2019: 41) found that “the closest time horizon does not necessarily produce the most accurate adjusted forecast,” in particular when the statistical forecast is updated over the time horizon. The researchers argued that forecasters may put too much weight on their own judgment relative to the statistical model. They also found that “adjustments far away from the sales point contribute little to, or damage, forecast accuracy” (Van Den Broeke et al., 2019: 39). Our findings, investigating a hotel revenue management operating environment, provide new insights. We found a significant relationship between the frequency of user override and forecasting accuracy at seven different forecasting time horizons (0, 1, 3, 7, 21, 84, 252 DTA). The impact of overrides, however, is dependent on the business type. A higher frequency of judgmental adjustments improves the forecast accuracy of group business, but damages transient forecast accuracy. In other words, RMS users appear to face difficulties with improvements to computer-generated predictions of transient business, but add value when they override the RMS group business forecast. This is an important insight, as group business plays a key role in terms of occupancy and revenue forecasting. According to Anderson and Xie (2010: 57) “many city hotels have upwards of 50 percent of their reservations blocked far in advance by large groups at discounted rates for conferences or special events.” As such, group business decisions have a direct impact on the room availability for the transient segment. Once rooms are blocked for a group, they become unavailable for other segments. Decisions about group acceptance thus directly impact the revenue management optimization cycle for the entire hotel. For full-service hotels, the group segment may represent more than half of the total revenue (Cross et al., 2009; Hormby et al., 2010).

There are a number of reasons for why modeling group forecasts is difficult in hotel RM. The first is linked to the nature of the group business: “groups demand blocks of rooms, introducing a combinatorial aspect to the optimization problem” (Hormby et al., 2010: 49). A related reason is that the nature of group cancellations is more variable and statistically uncertain (Sierag et al., 2017). A third reason is that the information necessary to calculate the unconstrained demand is not always available and/or reliable. For transient demand, statistical methods can be used to infer unconstrained demand. However, for group demand, lost and turndown data need to be recorded; in practice these are not always logged or completed by the sales team (Hormby et al., 2010). This may explain why judgmental adjustments provide value to group forecasting.

These modeling challenges have significant consequences. El Gayar et al. (2011) point out that the majority of existing forecasting models in hotel revenue management ignore group reservations and assume a pre-defined probability distribution to represent group arrivals. They argue that ignoring the group is a critical limitation, as groups can sometimes constitute the main guest segment. Furthermore, assuming a probability distribution for group demand can lead to inaccurate revenue management decisions, as hotel arrival characteristics tend to have a dynamically changing and evolving nature. For example, in a single small hotel case study, (Sierag et al., 2017) found evidence for an inhomogeneous Poisson nature of the probability distribution function. Furthermore, in the absence of user overrides, there are indications that these modeling challenges result in higher forecasting errors for group forecasts. In one hotel chain study, Kimes (1999) reports substantial group forecasting errors, depending on the time horizon with MAPE, of up to 40% two months before arrival. Whereas no direct comparison was made with the transient segment, the errors Kimes (1999) found for group business were much higher than the forecast errors reported by other studies at the total hotel level (Schwartz et al., 2016; Yuksel, 2007), or by room type (Pereira, 2016).

It is known that revenue managers spend substantial time monitoring and adjusting group forecasts. Noone and Hultberg (2011) found evidence that hotel revenue managers use multiple sources of information for group forecasting, including information about specific group characteristics and external information. They also revealed that revenue managers explicitly refer to communication with the sales team as part of their group forecasting activities. Insights obtained from the sales team, who are in direct contact with the group, contain important and the most updated information about the final group size or any changes affecting the final group demand. Since group demand modeling has important limitations, revenue managers thus adjust group forecasts using multiple sources of information exogenous to the RMS. This third prong of the forecast combinations approach to hotel occupancy forecasting, acting on exogenous and relevant information (i.e., knowledge beyond “what the system knows”) is where judgmental adjustments seem to add value to the RMS (Schwartz et al., 2016).

Limitations and future research

Although subject to limitations, our study points to multiple promising venues for further research. The first is related to a key limitation of this study. Given the enormous size of the data set and constrained computational capacity, this study analyzed seven forecasting horizons. Future research with access to more powerful and advanced computing resources, as well as data storage capacity, could potentially extract more data and analyze a wider range of forecasting horizons. This could improve the model estimates and, in particular, their non-linear form as our findings appear to agree with previous research on the relationship between time and forecast accuracy, for both linear and non-linear components (e.g., Schwartz, 1998; Pereira, 2016; Rajopadhye et al., 2001).

Additional area for future improvement is related to the estimation of the polynomial functions. Given that the components of a polynomial function are often correlated, Heck et al. (2014) recommend using an orthogonal function if a trajectory estimation is involved. We did not adopt this approach because it requires the observations to be equally spaced. Future research might follow Hox (2010) suggestion regarding using orthogonal polynomials if repeated measurements are not exactly spaced, and contrast the results with the non-orthogonal findings.

Another limitation has to do with the aggregation levels. Our understanding of overrides performance and the implications on revenue management effectiveness could greatly benefit from a study that explores overrides at a more granular level. For example, extending the scope beyond the group/transient dichotomy addressed in this study and contrast performances across the various commonly-used sub-segments and distribution channels. Another related future improvement could be to examine the role of additional operational factors on the impact of time on the accuracy of judgmental adjustments. These include the size of overrides, their direction (positive or negative adjustment), and the experience of the expert overriding the forecast.

In periods of extreme crisis situations such as the recent covid-19 pandemic, historical patterns of demand on which algorithmic prediction rely, are effectively useless. It was demonstrated during 2020 and 2021 in all but a few markets globally. This could also be true for the period following the crisis until stable demand patterns are reestablished to enable the desirable level of algorithmic forecast accuracy. In these periods, human assessment proves invaluable. Future research could define and quantify the crisis-induced level of pattern disruption. It could then be beneficial to assess how the level of pattern disruption affects the relative importance contribution of user overrides to the forecast accuracy. In addition, in situations where the historical patterns prove useless, the importance of advanced booking data as the source of valuable information increases considerably. However, a related issue of shrinking booking curves makes it less likely to be that beneficial. What works remains to be explored in future research. In this context, this study did not explore the impact of the characteristics of the user overriding the computer forecast. Relevant experience skills and even personality traits such as self-confidence could affect the accuracy of the subjective adjustment. This is certainly an interesting topic to explore in future research.

Finally, and perhaps the most exciting beneficial research in this domain of subjective overrides of machine predictions, is to assess the overall effectiveness of overrides. In other words, it has yet to be established to what degree (if at all) overrides improve the accuracy of algorithmic predictions, and what are the circumstances in which subjective overrides are more likely to improve the forecast accuracy in hotel revenue management.

Footnotes

Acknowledgments

The authors gratefully acknowledge the insights and continuous support of Dr Helen Pluut and Dr Peter van der Zwan (Leiden University) who greatly assisted this work.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Netherlands Organisation for Scientific Research (NWO) (023.002.090).

Note

Author biographies

Larissa Koupriouchina, PhD, was a Senior Lecturer in Revenue Management and Research fellow at Hotelschool The Hague from 2009 to 2021. She received a prestigious research grant from the Netherlands Organization for Scientific Research (NWO) to study human-computer interactions between revenue managers and revenue management systems. Prior to starting her career in education, Larissa worked for over ten years as a senior consultant on revenue and profit recovery projects.

Jean-Pierre van der Rest, PhD, is a professor and dean at Leiden University. He previously held positions as chaired professor, research director and associate dean at Hotelschool The Hague. Recipient of research grants and awards, his research interest include a variety of topics in hotel revenue management, including behavioral pricing, forecasting, strategic pricing, ethical pricing, and turnaround and risk management.

Zvi Schwartz, PhD, is a professor in the Department of Hospitality Business Management at the Lerner College of Business and Economics at the University of Delaware. His scholarly research and industry consulting focus on the core technical and strategic elements of the revenue management cycle: forecasting, optimization and monitoring, as well as the closely related topics of strategic pricing and consumer and firm decisions in advanced reservation environments. Schwartz’s recent projects explore novel hotel forecasting approaches, occupancy forecasting accuracy measures, hotel competitive sets, overbooking optimization and revenue management performance measures.

References

Alvarado-Valencia

Barrero

Onkal

, et al. (2017) Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting. International Journal of Forecasting 33: 298–313.

Anderson

Xie

(2010) Improving hospitality industry sales. Cornell Hospitality Quarterly 51: 53–67.

Andrawis

Atiya

El-Shishiny

(2011) Combination of long term and short term forecasts, with application to tourism demand forecasting. International Journal of Forecasting 27: 870–886.

Arenoe

Van der Rest

J-PI

(2020) Does willingness to pay for rate conditions depend on the booking window? A novel time-dependent conjoint analysis approach. Cornell Hospitality Quarterly 61: 213–222.

Armstrong

Collopy

(1992) Error measures for generalizing about forecasting methods: empirical comparisons. International Journal of Forecasting 8: 69–80.

Bach

(1999) Maximizing the Value of Analyst Intervention. London, UK: AGIFORS Reservations and Yield Management Study Group Proceedings.

Bearden

Murphy

Rapoport

(2008) Decision biases in revenue management: some behavioral evidence. Manufacturing & Service Operations Management 10: 625–636.

Belobaba

(1987) Air Travel Demand and Airline Seat Inventory Management. Dept. of Aeronautics and Astronautics, Massachusetts Institute of Technology.

Chen

C-C

Schwartz

(2008) Timing matters: travelers’ advanced-booking expectations and decisions. Journal of Travel Research 47: 35–42.

10.

Chen

Kachani

(2007) Forecasting and optimisation for hotel revenue management. Journal of Revenue & Pricing Management 6: 163–174.

11.

Claeskens

Magnus

Vasnev

, et al. (2016) The forecast combination puzzle: a simple theoretical explanation. International Journal of Forecasting 32: 754–762.

12.

Claver Cortés

Molina Azorín José

Pereira Moliner

(2007) The impact of strategic behaviours on hotel performance. International Journal of Contemporary Hospitality Management 19: 6–20.

13.

Clemen

(1989) Combining forecasts: a review and annotated bibliography. International Journal of Forecasting 5: 559–583.

14.

Cross

Higbie

Cross

(2009) Revenue management’s renaissance. Cornell Hospitality Quarterly 50: 56–81.

15.

Davydenko

Fildes

(2013) Measuring forecasting accuracy: the case of judgmental adjustments to SKU-level demand forecasts. International Journal of Forecasting 29: 510–522.

16.

Donihue

(1993) Evaluating the role judgment plays in forecast accuracy. Journal of Forecasting 12: 81–92.

17.

El Gayar

Saleh

Atiya

, et al. (2011) An integrated framework for advanced hotel revenue management. International Journal of Contemporary Hospitality Management 23: 84–98.

18.

Fildes

Goodwin

Lawrence

, et al. (2009) Effective forecasting and judgmental adjustments: an empirical evaluation and strategies for improvement in supply-chain planning. International Journal of Forecasting 25: 3–23.

19.

Foss

Stensrud

Kitchenham

, et al. (2003) A simulation study of the model evaluation criterion MMRE. IEEE Transactions on Software Engineering 29: 985–995.

20.

Franses

Legerstee

(2011) Experts’ adjustment to model-based SKU-level forecasts: does the forecast horizon matter? Journal of the Operational Research Society 62: 537–543.

21.

Gneiting

(2011) Making and evaluating point forecasts. Journal of the American Statistical Association 106: 746–762.

22.

Gönül

(2013) A popular and important tool for forecast users: judgmental adjustments. Tahmin Kullanıcılarının Hizmetindeki Popüler ve Önemli Bir Araç: ‘Yargısal Düzeltmeler 24: 73–93.

23.

Goodwin

Lawton

(1999) On the asymmetry of the symmetric MAPE. International Journal of Forecasting 15: 405–408.

24.

Goodwin

Wright

(1994) Heuristics, biases and improvement strategies in judgmental time-series forecasting. Omega-International Journal of Management Science 22: 553–568.

25.

Haensel

Koole

(2011) Booking horizon forecasting with dynamic updating: a case study of hotel reservation data. International Journal of Forecasting 27: 942–960.

26.

Heck

Thomas

Tabata

(2014) Multilevel and Longitudinal Modelling with IBM SPSS. New York, NY: Routledge.

27.

Hormby

Morrison

Dave

, et al. (2010) Marriott International increases revenue by implementing a group pricing optimizer. Interfaces 40: 47–57.

28.

Hox

(2010) Multilevel Analysis: Techiques and Applications. New York, NY: Routledge Academic.

29.

Hox

Moerbeek

Schoot

(2018) Multilevel Analysis: Techiques and Applications. New York, NY: Routledge.

30.

Hsiao

Wan

(2014) Is there an optimal forecast combination? Journal of Econometrics 178: 294–309.

31.

Jaganathan

Prakash

PKS

(2020) A combination-based forecasting method for the M4-competition. International Journal of Forecasting 36: 98–104.

32.

Kimes

(1999) Group forecasting accuracy in hotels. Journal of the Operational Research Society 50: 1104–1110.

33.

Kimes

(2008) Hotel Revenue Management: Today and Tomorrow: The Center for Hospitality Research. Ithaca, NY: Cornell University.

34.

Koehler

(2001) The assymetry of the SAPE measure and other comments on the M3-Competition. International Journal of Forecasting 17: 570–574.

35.

Koupriouchina

Van der Rest

Schwartz

(2014) On revenue management and the use of occupancy forecasting error measures. International Journal of Hospitality Management 41: 104–114.

36.

Law

(2004) Initially testing an improved extrapolative hotel room occupancy rate forecasting technique. Journal of Travel & Tourism Marketing 16: 71–77.

37.

Lawrence

Goodwin

O’Connor

, et al. (2006) Judgmental forecasting: a review of progress over the last 25 years. International Journal of Forecasting 22: 493–518.

38.

Lim

Oconnor

(1995) Judgmental adjustment of final forecasts - its effectiveness and biases. Journal of Behavioral Decision Making 8: 149–168.

39.

Lin

(2013) Improving forecasting accuracy by combining statistical and judgmental forecasts in tourism. Journal of China Tourism Research 9(3): 325–352.

40.

Lin

(2019) Judgmental adjustments in tourism forecasting practice: how good are they? Tourism Economics 25(3): 402–424.

41.

Lin

Goodwin

Song

(2014) Accuracy and bias of experts’ adjusted forecasts. Annals of Tourism Research 48: 156–174.

42.

Loew

(2000) Forecast Versus Actual Analysis. New York, NY: AGIFORS Reservations and Yield Management Study Group Proceedings.

43.

Makridakis

(1986) The art and science of forecasting An assessment and future directions. International Journal of Forecasting 2: 15–39.

44.

Makridakis

(1993) Accuracy measures: theoretical and practical concerns. International Journal of Forecasting 9: 527–529.

45.

McNees

(1990) The role of judgment in macroeconomic forecasting accuracy. International Journal of Forecasting 6: 287–299.

46.

Mukhopadhyay

Samaddar

Colville

(2007) Improving revenue management decision making for airlines by evaluating analyst-adjusted passenger demand forecasts. Decision Sciences 38: 309–327.

47.

Noone

Hultberg

(2011) Profiting through teamwork: the role of the revenue management and sales functions in group revenue management. Cornell Hospitality Quarterly 52: 407–420.

48.

Pereira

(2016) An introduction to helpful forecasting methods for hotel revenue management. International Journal of Hospitality Management 58: 13–23.

49.

Petropoulos

Fildes

Goodwin

(2016) Do ‘big losses’ in judgmental adjustments to statistical forecasts affect experts’ behaviour? European Journal of Operational Research 249: 842–852.

50.

Pine

Phillips

(2005) Performance comparisons of hotels in China. International Journal of Hospitality Management 24: 57–73.

51.

Rajopadhye

Ben Ghalia

Wang

, et al. (2001) Forecasting uncertain hotel room demand. Information Sciences 132: 1–11.

52.

Schwartz

(1998) Convention and conference facilities: a framework of statistical predictions and judgmental adjustments for daily occupancy forecasts. Journal of Convention & Exhibition Management 1: 71–88.

53.

Schwartz

Cohen

(2004) Hotel revenue-management forecasting: evidence of expert-judgment bias. Cornell Hotel and Restaurant Administration Quarterly 45: 85–98.

54.

Schwartz

Hiemstra

(1997) Improving the accuracy of hotel reservations forecasting: curves similarity approach. Journal of Travel Research 36: 3–14.

55.

Schwartz

Uysal

Webb

, et al. (2016) Hotel daily occupancy forecasting with competitive sets: a recursive algorithm. International Journal of Contemporary Hospitality Management 28: 267–285.

56.

Schwartz

Webb

van der Rest

, et al. (2021) Enhancing the accuracy of revenue management system forecasts: the impact of machine and human learning on the effectiveness of hotel occupancy forecast combinations across multiple forecasting horizons. Tourism Economics 27(2): 273–291.

57.

Sierag

Van der Rest

Koole

, et al. (2017) A call for exploratory data analysis in revenue management forecasting: a case study of a small and independent hotel in The Netherlands. International Journal of Revenue Management 10: 28–51.

58.

Song

Gao

Lin

(2013) Combining statistical and judgmental forecasts via a web-based tourism demand forecasting system. International Journal of Forecasting 29(2): 295–310.

59.

Talluri

Van Ryzin

(2004) The theory and practice of revenue management. Hamburg: Springer.

60.

Tofallis

(2015) A better measure of relative prediction accuracy for model selection and model estimation. Journal of the Operational Research Society 66: 1352–1362.

61.

Trapero

Fildes

Davydenko

(2011) Nonlinear identification of judgmental forecasts effects at SKU level. Journal of Forecasting 30: 490–508.

62.

Turner

(1990) The role of judgment in macroeconomic forecasting. Journal of Forecasting 9: 315–345.

63.

Van den Broeke

De Baets

Vereecke

, et al. (2019) Judgmental forecast adjustments over different time horizons. Omega 87: 34–45.

64.

Weatherford

(2016) The history of forecasting models in revenue management. Journal of Revenue and Pricing Management 15: 212–221.

65.

Weatherford

Kimes

(2003) A comparison of forecasting methods for hotel revenue management. International Journal of Forecasting 19: 401–415.

66.

Yuksel

(2007) An integrated forecasting approach to hotel demand. Mathematical and Computer Modelling 46: 1063–1070.

67.

Zeni

(2003) The value of analyst interaction with revenue management systems. Journal of Revenue & Pricing Management 2: 37–46.