* Faculty of Economics, University of Porto, 4200 Porto, Portugal.

Tel: +351-2-5509720, Fax: +351-2-5505050, E-mail: framos@fep.up.pt

Abstract:

This paper compares the out-of-sample forecasting accuracy of five classes of time series models for market shares of the six most important Portuguese car market competitors over different horizons. As representative time series models I employ a random walk with drift (Naive), a univariate ARIMA, a near-VAR and a general BVAR. The out-of-sample forecasts are also compared against forecasts generated from structural econometric market share models (SEM). Using four accuracy measures I find the forecasts from the near-VAR and the BVAR models really more accurate. With regard to these models, I could say that the BVAR model is the best for longer forecasts (12-steps ahead), while the n-VAR is superior over the shorter horizon of one to six steps.

JEL Classification: C110, C320, M310.

Keywords: Accuracy measures; ARIMA; Automobile market; BVAR; Market share; Portugal; Random-walk; SEM; VAR

Introduction

Multiple time series models have been proposed, for some times, as alternatives to structural econometric models (SEM) in economic forecasting applications. In the marketing field, applications of multiple time series techniques (Transfer Functions, Intervention and VARMA) in the area of market response models is now well known (Hanssens, Parsons and Schultz (1990) give a good overview in this domain).

One class of multiple time series models which has received much attention recently is the class of Vector Autoregressive (VAR) models. These models constitute a special case of the more general class of VARMA models. Although VAR models have been used primary for macroeconomic forecasting, they offer an interesting alternative to either SEM of market shares or Box-Jenkins ARIMA models for problems in which simultaneous forecasts are required for a collection of related microeconomic variables, such as industry sales and firm's market shares forecasting. The use of VAR models for economic forecasting was proposed by Sims (1980), motivated in part by questions related to the validity of the way in which economic theory is used to provide a priori justification for the inclusion of a restricted subset of variables in the "structural" specification of each dependent variable. Sims (1980) questions the use of the so called "exclusionary and identification restrictions". Such time series models have the appealing property that, in order to forecast the endogenous variables in the system, the modeller is not required to provide forecasts of exogenous explanatory variables; the explanatory variables in a SEM are typically no less difficult to forecast than the dependent variables. In addition, the time series models are less costly to construct and to estimate. This does not imply, however, that VAR models necessarily offer a parsimonious representation for a multivariate process (see, e.g., Fuller, 1976). Despite this lack of parsimony, and the additional uncertainty imposed by the use of a finite-order VAR model as an approximation to the infinite-order VAR representation, VAR models are of interest for practical forecasting applications because of the relative simplicity of their model identification and parameter estimation procedures, superior performance, compared with those associated with structural and VARMA models . For example, Brodie and De Kluyver (1987) have reported empirical results in which simple "naïve" market share models (linear extrapolations of past market share values) have produced forecasts as accurate as those derived from SEM of market shares. Furthermore, the same paper shows that using lagged market share often gives better results than a SEM which incorporates marketing mix variables. Indeed, Danaher and Brodie (1992) gave a criterion which determines whether it is advantageous to use marketing mix information for forecasting market share.

The number of parameters to be estimated may be very large in VAR models, particularly in relation to the amount of data that is typically available for marketing applications. This lack of parsimony may present serious problems when the model is to be used in a forecasting application. Thus, the use of VAR models often involves the choice of some method for imposing restrictions on the model parameters: the restrictions help to reduce the number of parameters and (or) to improve their estimability. One such method, proposed by Litterman (1980), utilises the imposition of stochastic constraints, representing prior information, on the coefficients of the vector autoregression. The resulting models are known as Bayesian Vector Autoregressive (BVAR) models.

The aim of this paper is to develop five classes of time series models for the six major competitors of the Portuguese automobile market for the period January 1988 to December 1992. Then out-of-sample one-through twelve-months-ahead forecasts are computed for the six market shares for the period January 1993 to Jun 1994 and their accuracies are evaluated using four accuracy measures.

The paper is organized as follows. The next section briefly describes the modelling methodologies. The third section describes the data base used and the rational behind the choice of the variables. The fourth presents the models in competition, the forecasting strategy and the main empirical results. The conclusions are given in the final section.

Ramos (1995) describes VAR and BVAR modelling in its general context and presents this methodology comprehensively . In this section, I review the key features of the VAR and BVAR methodology, borrowing extensively from Ramos' presentation.

Structural and vector autoregressive (VAR) models

As a starting point, I assume that the structure under investigation can be described by the following equations:

where Y1t and Y2t are k1 and k2 ( k1+k2=k) vectors; e1t and e2t are error terms, with ; is a set of matrix polynomials of order m.

Note that in this specification, contemporaneous relationships within and between Y1t and Y2t can be capture by the Aij,0 matrices. As usual, the e1t are part of the structure and correspond to economically meaningful concepts. A1 and A2 are vectors of constants. In a more general form, I can write:

where and . Structural models such as equation (2) are estimated under various maintained assumptions such as exogeneity of a block of variables (e.g., Y2t) or exclusion restrictions implied by theory. There are two problems with structural models. First, for proper identification of individual equations in the system, the correct number of variables have to be excluded from an equation in the model. As argued by Cooley and LeRoy (1985), such exclusion is often carried out with little theoretical justification. Second, structural models are poorly suited for forecasting . Projected future values of the exogenous variables are required for this purpose.

It is now well accepted that the basic structural phenomena can be recovered through the estimation of a vector autoregressive model (VAR), where each variable of the system is linked to its own past values as well as the lagged values of all the other variables in the system. More formally, VAR models correspond to the reduced form equation (2):

where ; ut is constructed as a linear combination of structural errors et and cannot be interpreted from an economic viewpoint. However, with minimal identifying assumptions, estimates of et can be obtained from ut. As usual, ut is centred on zero with a variance-covariance matrix given by . I can write a representative equation using the first variable of the system as an example, as:

where d10 is the usual constant.

In this formulation, VAR models impose few constraints and require the estimation of a large number of parameters, thus rapidly exhausting the available degrees of freedom. For models as large as the one considered in this paper, some restrictions are required, for example, to exclude potentially important variables or restrict the number of lags. As well, large unrestricted regressions often suffer from overparameterization and lead to poor forecasts: they tend to pick up sample-specific and temporary relationships which are a source of poor performance outside the sample period. This overparameterization results in multicollinearity and loss of degrees of freedom that can lead to inefficient estimates and large out-of-sample erros. While estimation of such a highly parameterized system will provide a high degree of fit to the data, the out-of-sample forecasts can be very poor in terms of mean square error.

BVAR models

To overcome problems associated with VAR models, Litterman (1980), Doan, Litterman and Sims (1984), and Sims (1989) suggested the incorporation, in a Bayesian fashion, of relevant a priori information in the estimation. A posteriori estimates obtained after combining priors and sample information are the cornerstone of Bayesian models.

Following Litterman, these priors are not based on economic theory but rest on some empirical regularities. For example, according to Litterman (1980) and Nelson and Plosser (1982), most macroeconomic variables can be approximated by the simple discrete random-walk with drift. For a representative equation i of system (3), this takes the form

More specifically, the prior mean for the ith equation will be centred on the elements equal to 1 for , and 0 otherwise.

At first glance, this information appears to be rather restrictive and may not be appropriate for all the series investigated. The degree of constraint will depend on the dispersion associated with each coefficient. Ideally, this a priori uncertainty should reflect some basic requirements. In addition, the procedure chosen should be practical, since it would be impossible to determine (separetely) the values of the coefficients. To meet these requirements, Sims (1989) suggests the following general formulation for the prior variance-covariance matrix associated with :

where is the standard deviation of the prior distribution for the coefficient on lag l of variable j in equation i ; g is the overall tightness parameter. It is the standard deviation of the coefficient of variable i in equation i lagged once, e.g., Yt-1 in equation (5). The parameter w allows the imposition of a tighter standard error on the jth variable in equation i; it operates only when . is a distributed lag function (harmonic specification) with parameter d that tightens the standard errors as the lag length increases. adjusts the prior information to the relative scale of the variables, where is the estimated standard error of a univariate AR model for variable i and is the same statistic computed for variable j.

In fact, Sims (1989) uses this form of prior for all the variables in his BVAR system and reports interesting improvements in forecast performance; I will follow the same strategy.

The BVAR model is estimated using Theil's (1971) mixed-estimation technique that involves supplementing data with prior information on the distributions of the coeffcients. For each restriction on the parameter estimates, the number of observations and degrees of freedom are increased by one in an artificial way. The loss of degrees of freedom due to overparameterization associated with a VAR model is therefore not a problem with the BVAR model.

To apply Litterman's procedure one must search over the parameters g, d, and w until some predetermined objective function is optimized. The objective function can be the out-of-sample mean-squared forecast error, or some other measure of forecast accuracy. Doan, Litterman and Sims (1984) suggest minimizing the log-determinant of the sample covariance matrix of the one-step-ahead forecast errors for all the equations of the BVAR. In a forecasting comparison such as mine, a portion of the sample must be withheld to determine the parameters g, d, and w; while the remainder of the sample is used with the selected model to generate out-of-sample forecasting statistics for comparison purposes.

The data base used for this study is a monthly time series sample of market shares, and marketing decision variables, for the period January 1988 through June 1994 (78 observations on each variable), for the six most important competitors presents in the Portuguese car market. The marketing decision variables include such variables as relative price, major media advertising expenditures (TV, radio, and newspaper), and relative Age. The Portuguese car market consists of twenty five imported brands, but the top six, presented in all segments, account on average for 82.3% of the total market, with a standard deviation of 4.75%.

The time series variables used in this study are defined as follows:

= market share of the ith competitor,

= relative age of the ith competitor,

= relative price of the ith competitor,

= TV advertising share of the ith competitor,

= Radio advertising share of the ith competitor, and

= Press advertisng share of the ith competitor

where i = 1,...,6 represents respectively Renault, Peugeot-Citroen, Ford, Opel, Alfa-Fiat-Lancia, Audi-Seat-VW.

The data on are calculated from the monthly new automobile registrations published by the Portuguese General Directorate of Transports. Figure 1 plots monthly market shares for the sample period.

The relative age represents the differents models (versions) offered by each cpmpetitor in all market segments. It measures the time in market (in months) of the most representative models of each segment. This variable represents the models life cycle of each competitor, and can be seen as a strategic marketing variable . It was obtained as follows:

- for each firm, I measure the time in market after the launch of the most representative model of each segment , i.e., the model with the highest share of the segment;

- for the competitors, I calculate the simple average age of the most representative model of each segment;

- to obtain the firm's average age and (or) of their competitors, I calculate the weighted average age for the models chosen on each segment. The weights are given by the relative importance of each segment on total demand (S1+S2+S3+S4);

- the relative age, named , is then calculated as the ratio between the weighted average age of firm i and the weighted average age of their competitors ().

The pricing decision by each firm is measured by the relative price defined as a price index which is calculated by dividing each firm's average price by an average price of their competitors. The variable is obtained following the steps just described for. The weights are the same, and the price of each model is the consumer's price (all taxes included) of the most representative model of each segment. The price of each model is published on a monthly basis and comes from the "Guia do Automóvel" (The oldest and most read Portuguese car magazine).

The data on are expressed as shares of total advertising expenditures by media (us/industry) and were obtained from "Sabatina". This Portuguese firm records monthly the advertising expenditures by media and by brand. Advertising expenditures represent only "official or contractual prices", and I know in the industry that prices are frequently lower.

All the data (36 time series) were transformed into logarithms to handle nonstationary in variance, i.e., heteroscedasticity.

This dataset is available from the author on request.

Models in competition

Five classes of models are included in my empirical comparisons , each class being represented by one or more specific models. The classes are NAIVE (a first order autoregressive model for each ), ARIMA (a univariate Box-Jenkins model for each ), SEM (a structural econometric market share model for each firm), n-VAR (a near-VAR system), and BVAR. BVAR and n-VAR forecast all market share variables in a system of 36 time series, while the other techniques forecast only one at a time.

The NAIVE model, defined as is a simple model that uses no marketing mix data, only lagged market share. Brodie and De Kluyver (1987), Alsem, Leeflang and Reuyl (1989), and Danaher and Brodie (1992) have reported empirical results in which the predictive accuracy of econometric market share models is not consistently better than that of a "naive" model (, in BdK's paper). This conclusion was reached after reviewing several empirical studies and also analysing data for 15 brands in three markets.

The ARIMA models have been built following the Box-Jenkins' approach. The usual criteria, e.g., stationarity, autocorrelation, and partial autocorrelation functions, significance of coeffcients, and FPE are used to select the best models . As Montgomery and Weatherby (1980, p.306) note: "The Box-Jenkins approach uses inefficient estimates of impulse response wieghts which are matched against a set of anticipated patterns, implying certain choices of the parameters ... the analyst's skill and experience often play a major role in the success of the model building effort".

The purpose of the SEM class is to specify models for predicting the future values of market shares, taking in account not only the firm's marketing instruments, but also the actions developed by competitors. According to Alsem, Leeflang and Reuyl (1989), all estimated market share models are specific forms of the following multiplicative specification:

where is a random disturbance term, and . In equation (7) all variables are expressed competitively (see my definition of variables).

In practice, it is not possible to avoid imposing some restrictions on a VAR system. There is always some limit on the number of variables which can be included in a VAR model as well as on the maximum number of lags. In my case, with a system of 36 variables and six lags on each variable, the total number of regressors in each equation would be 216. This would make the entire modelling process impossible (the in-sample period has only 60 observations). It may happen, then, that some variables have to be excluded prior to modelling. Especially important here is to eliminate those coefficients for which the hypothesis that they are jointly equal to zero cannot be rejected, e.g., using the likelihood ratio (LR) test statistic suggested by Sims (1980) or the more sophisticated procedure of Hsiao (1979).

My strategy was to divide the 36-VAR system in two subsystems as represented below. In the first, defined by the six market-share equations, I allow for different lag lengths for each variable in each equation, using the LR test statistic of Sims. In the second, I construct five 6-equation VARs (one for each marketing instrument) with each equation containing six lags on all variables. Longer lags (up to nine) were also tried but the main results were unchanged.

The near-VAR system can be represented algebrically as follows:

The system is then estimated efficiently using seemingly unrelated regressions (SUR).

In the class of BVAR models, the variables are specified in levels because as pointed out by Sims et al. (1990, p. 360) " ... the Bayesian approach is entirely based on the likelihood function, which has the same Gaussian shape regardless of the presence of nonstationarity, [hence] Bayesian inference need take no special account of nonstationarity" . The model is estimated with six lags on each variable. Longer lags (up to nine) were also tried but the substantive results were unchanged.

To find the hiperparameter values (g, w, and d) that minimize, for the period 1992:1 to 1992:12, the average of the RMSEs statistics for one-to six-months-ahead forecasts, I proceeded by grid-search evaluations, a method especially well-suited to the problem at hand. This way, the two holdout samples (one for estimating the BVAR parameters and the other for checking forecasting accuracy) are obviously differents. Since I have 36 explanatory variables with six lags for each variable, and a constant, each equation has a total of 217 right-hand side variables. I set the admissible values for the coefficients to be estimated as , , and (the step). For example, w = 0.5 means that in the ith equation, all right-hand-side variables, except the own lagged variable, enter with a weight of 50%. For the parameter g, I have assumed a loose value of 0.4 and a tight value of 0.1. The harmonic lag decay parameter was set arround one as recommended by Doan, Litterman and Sims (1984). The best values according to the criterion function were obtained for g = 0.25, w = 0.20, and d = 0.95. Note that it is possible to specify a general form of equation (6). This, however, involves specifying a 36x36 matrix with diagonal elements equal to 1 and the off-diagonal elements representing the weights of other variables . Fine-tuning the prior to this extent is not generally recommended since the problem of overparameterization (i.e. estimating too many coefficients) is then replaced with one of estimating too many hyperparameters.

The final estimation results for all these models are not presented here due to space limitations. However, these can be obtained from the author on request.

Forecasting strategy

Following Dua and Smyth (1995) and other researchers, I adopted the strategy of sequential estimation for generating out-of-sample forecasts for one to twelve horizons. The parameters of the models are based on the most recent information available at the time the forecast is made. More precisely, the models were initially estimated using data from 1988:1 to 1992:12, i.e., the first 77% was used for parameter estimation and the last 23% was used for the out-of-sample forecasting analysis below. The choice as to were to begin forecasting was predicted on the desire to produce short-term forecasts and have sufficient number of observations for each forecast step. This has enable me to produce twelve-steps-ahead (months) forecasts. Forecasts are then generated, using the Kalman filter algorithm in RATS at horizons of one to twelve months. Next, data for 1993:1 are added to the sample and the parameters of each model are re-estimated. New forecasts are then generated for one-to twelve-months-ahead. This process continues through to the last forecast period, 1994:5

Evaluation of accuracy

The accuracy of the out-of-sample forecasts for 1993:1 to 1994:6 is measured by the RMSE and the Theil U statistics for one-to twelve-months-ahead forecasts. If denotes the actual value of a variable, and the forecast made in period t , then the RMSE and the Theil statistic are defined as follows:

where denotes the forecast step and N is the total number of forecasts in the prediction period.

The U statistic is the ratio of the RMSE for the estimated model to the RMSE of the simple random walk model which predicts that the forecast simply equals the most recent information. Hence if U < 1, the model performs better than the random walk model without drift; if U > 1, the random walk outperforms the model. The U statistic is therefore a relative measure of accuracy and is unit-free (Bliemel, 1973). The forecasted values used in the computation of the RMSE and U statistics are the level (in logarithms) of the market shares, so these statistics can be compared across the different models.

The RMSEs and the Theil's U statistics for for the five classes of models discussed above are reported in Table 1. "Theil sum" is the sum of the Theil statistics for each of the twelve forecasting horizons and "Theil <1.0" indicates the number of cases where the tested model outperforms the random walk model. The table also reports tests of significant differences among the RMSEs measures following the procedure given in Brandt and Bessler (1983). The conclusions from Table 1 are as follows:

(1) RMSEs versus Theil U statistics: The RMSEs and the Theil U satistics do not follow a consistent pattern with an increase in the forecasting horizon.

(2) Univariate (simple) versus multivariate (complex) models: Multivariate models are always superior for all forecasting periods and competitors with one exception, MS3. Its RMSEs are not significantly differents when compared across models and time periods. If I rely on the "Theil sum" criterion, the univariate ARIMA model beats all other models in forecasting MS3.

(3) Random-walk versus all the models: Multivariate models (n-VAR and BVAR) out-forecast the random-walk model, winning in anywhere from 9 to 12 attempts. The only exception is the forecast of MS2, where the random-walk outperforms the n-VAR seven times. The random-walk is clearly superior to univariate models, except in forecasting MS6 and MS3.

(4) BVAR versus n-VAR: Without regard to significance, the BVAR outforecasts the random-walk model in four of six competitors, while the n-VAR does so in only to cases (MS1 and MS4). If the criterion is the "Theil sum" neither of the two models outperforms the other. When the two models are compared using the test of significance differences in RMSEs I could say that the BVAR model is the best for longer forecasts (12-steps ahead), while the n-VAR is superior over the shorter horizon of one to six steps. This suggests that the gap between the two models becomes wider over longer forecasting horizons.

In summary, the results in general show that there are gains from using a multivariate (VAR / BVAR) approach to forecasting.

In this paper I have investigated the out-of-sample forecasting performance of a wide class of univariate, structural, n-VAR and BVAR models for market shares of six the major competitors in the Portuguese car market. There are a number of empirical findings worth mentioning. I confirm that the n-VAR and the BVAR models are superior forecasting tools compared to the univariate models. The overall ranking of these models varies over different forecasting horizons. The near-VAR shows substantial improvement in short-medium-term forecasting accuracy of market shares, whereas the BVAR model is more accurate in the long term (normally up to twelve months). An implication of these results is that optimizing short- and long-term forecasting in a system like mine will often require separate efforts. This suggests that market-share forecasters in this market should conduct comparisons or horse races' prior to selecting their preferred model and then periodically check its effectiveness.

Notes: The RMSEs and the Theil's U statistics are reported for the log MSi , where i = 1,...6 represents respectively Renault, Peugeot-Citroen, Ford, Opel, Alfa-Fiat-Lancia, and Audi-Seat-VW. * Sum of Theil for forecasts for all 12 periods. ** Number of Theil values less than 1.0.

A test of significant differences in RMSEs is carried out following the procedure given in Brandt and Bessler (1983). The 'a' signifies that the RMSE of model j (j=2,...5) for each MS is significantly lower (at the 5% significance level) than that of model 1; 'b' signifies the RMSE of model j (j=1,3,4,5) is significantly lower than that of model 2; c, d, and e are defined in an analogous fashion.

References

Alsem, K. J., Leeflang, P. S. and Reuyl, J. C. (1989), "The forecasting accuracy of market share models using predicted values of competitive marketing behavior", International Journal of Research in Marketing, 6, 183-198.

Bliemel, F. (1973), "Theil's forecast accuracy coefficient: A clarification", Journal of Marketing Research, 10 (November), 444-446.

Brandt, J. and Bessler, D. (1983), "Price forecasting and evaluation: An application in Agriculture", Journal of Forecasting, 2, 237-248.

Brodie, R. J. and De Kluyver, C. A. (1987), "A comparison of the short term forecasting accuracy of econometric and naive extrapolation models of market share", International Journal of Forecasting, 3, 423-437.

Cooley, T. F. and LeRoy, S. F. (1985), "Atheoretical macroeconometrics: a critique", Journal of Monetary Economics, 16, 283-308.

Danaher, P.J. and Brodie, R. J. (1992), "Predictive accuracy of simple versus complex econometric market share models: Theoretical and empirical results" International Journal of Forecasting, 8, 613-626.

Doan, T., Litterman, R. and Sims, C. A. (1984), "Forecasting and conditional projection using realistic prior distributions", Econometric Reviews, 3, 1-100.

Dua, P. and Smyth, D. (1995), "Forecasting US home sales using BVAR models and survey data on households' buying attitudes for homes", Journal of Forecasting, 14, 217-227.

Fuller, W. (1976), Introduction to Statistical Time Series. New York: Wiley.

Hanssens, D., Parsons, L. and Schultz, R. (1990), Market Response Models: Econometric and Time Series Analysis. MA: Kluwer Academic Publishers.

Hsiao, C. (1979), "Causality tests in econometrics", Journal of Economic Dynamics and Control, 1, 321-346.

Lambin, J. J. and Dor, E. (1989), "Part de marché et pression marketing: vers une stratégie de modélisation", Recherche et Applications en Marketing, IV, 4, 3-24.

Litterman, R. B. (1980), "A Bayesian procedure for forecasting with vector autoregressions", Working paper, Massachusetts Institute of Technology, Departement of Economics.

------- (1986), "Forecasting with Bayesian vector autoregressions-five years of experience", Journal of Business and Economic Statistics, 4, 25-38.

Montgomery, D. C. and Weatherby, G. (1980), "Modeling and forecasting time series using transfer function and intervention methods", AIIE Transactions, 12, 289-307.

Nelson, C. R. and Plosser, C. I. (1982), "Trends and randomwalks in macroeconomic time series", Journal of Monetary Economics, 10, 139-162.

Ramos, F. F. R. (1995), "Forecasting market shares using VAR and BVAR models: A comparison of their forecasting performance", Working paper no. 57, Faculty of Economics, University of Porto, Portugal.

Sims, C. A. (1980), "Macreconomics and reality", Econometrica, 48, 1-48.

------- (1988), "Bayesian skepticism on unit root econometrics", Journal of Economics and Control, 12, 463-474.

------- (1989), "A nine variable probabilistic macroeconomic forecasting model", Discussion paper no. 14, Institute for Empirical Macroeconomic, Federal Reserve Bank of Minneapolis, Minneapolis.

-------, Stock, J. and Watson, M. (1990), "Inference in linear time series models with some unit roots", Econometrica, 58, 113-144.