Hierarchical Bayes (HB) estimation balances two competing goals: fitting each respondent's observed choices while avoiding overfitting caused by sparse or noisy individual-level data. This article provides an intuitive explanation of Bayesian smoothing in HB-MNL analysis of CBC and MaxDiff studies. It shows how respondents with limited or inconsistent choice data are naturally pulled toward the population distribution, while respondents who answer more choice tasks consistently rely increasingly on their own data. Using plain language and a small amount of mathematics, the article explains why Bayesian smoothing improves the stability and predictive accuracy of individual-level utility estimates.

This article by Bryan Orme discusses how to determine whether a Hierarchical Bayes (HB) Multinomial Logit (MNL) model has converged for choice-based conjoint (CBC) or MaxDiff studies. Bryan discusses the importance of both burn-in and used iterations, with convergence assessed via the R-hat statistic. Most studies Sawtooth customers field in practice are converging fine with the default number of iterations set by the software. For complex datasets with many parameters, longer chains of used draws (e.g., 30K+) improve convergence, especially when long oscillation patterns exist. Orme recommends different iteration settings based on parameter count. While convergence is critical for statistical testing and inference, not converged models usually still predict well for use in market simulators even if formal convergence isn’t met.

Orme summarizes a research effort and article by Kevin Van Horn, of Bayesium Analytics, where Van Horn compared various methods for implementing utility constraints for HB-MNL estimation in CBC experiments. Van Horn compared Sawtooth’s Simultaneous Tying (ST) algorithm for utility constraints to six different methods implemented in Stan involving sign constraints for thermometer (unary) coded ordered attributes. Van Horn finds that for a 5-level ordered attribute (price) in a CBC study, there is no difference in predictive validity for Sawtooth’s ST algorithm versus the various implementations of sign constraints performed in Stan. However, formal convergence diagnostics (Rhat and ESS) are much less favorable for ST.

Recently, Kevin van Horn (of Bayesium Analytics) did some R&D work for Sawtooth Software to compare Hamiltonian Monte Carlo (HMC) as implemented in Stan to the old standby, Sawtooth Software’s CBC/HB, that uses Metropolis Hastings (MH). This white paper summarizes his findings. The main finding is that for small to modest-sized CBC problems (Kevin examined six data sets with 15, 21, and 27 parameters to estimate, with varying degrees of sparseness), HMC is about the same speed or sometimes slower than CBC/HB using MH to achieve equally good predictions of holdouts. HMC iterations are considerably slower than CBC/HB; though each iteration accomplishes more. Formal measures of convergence favor HMC, but it seems for all practical purposes that CBC/HB leads to suitable convergence and quality of the posteriors to meet the needs of consulting projects in practice (in the small to modest sized model range of 15 to 27 parameters to estimate).

Finding useful covariates (variables outside the conjoint task) that can boost the predictive validity of conjoint analysis via HB estimation is a topic that Peter and Stefan have investigated and reported on multiple times at past Sawtooth Software conferences. In this presentation, Peter and Stefan found better success than past investigations by using a simple set of nine questions (binary semantic differentials) that can be added to the survey just prior to the CBC questions. The nine questions are based on principles of behavioral economics and include such pairs as: “I think brands differ a lot” vs. “I think brands are more of less the same” (respondents pick the statement they most agree with). Peter and Stefan proposed that these simple pairs statements help respondents remember their prior shopping situations and prime them to do a more realistic job in answering CBC questions. The showed that hit-rates and out-of-sample predictions could be significantly improved using this framework. They presented results for nine different CBC studies, demonstrating good improvement in both in-sample and out-of-sample hit rates when leveraging the nine covariates. What was perhaps even more intriguing is that just the mere presence of the covariates seemed to improve respondents’ performance on the CBC tasks. Even without using the covariates in HB estimation, the act of completing the nine semantic differential pairs improved the predictability of the respondent’s utilities for out-of-sample holdouts. In addition to the usefulness of covariates for improving predictive accuracy of the conjoint results, Peter and Steven recommended their value in developing useful consumer segmentations.

Sawtooth Software's Bryan Orme and Walter Williams report results of a meta analysis of about 50 commercial CBC and MaxDiff data sets. Specifically, they looked into how the priors settings in CBC/HB (prior variance and degrees of freedom) affect the ability of the estimated part-worth utilities to predict holdout choice tasks. Because most CBC data sets don't have enough holdout choice tasks for this purpose, they developed a special extension to the CBC/HB software that uses a jack-knifing routine to systematically hold out just a few of the choice tasks (and repeat) for hit rate validation while estimating the utilities from the remaining choice tasks.Bryan and Walter found that the default prior setting of variance=2 leads to lower hit rates than lower prior variance settings (the average optimal prior variance setting across the data sets was 0.78). Not surprisingly, they found that the optimal prior variance setting depends on the data set (their optimal found prior variance ranged from 0.1 to 1.65). They also found that the default prior variance setting for MaxDiff data (prior variance = 1) is fairly close to optimal for the MaxDiff data sets they examined, but actually may lead to a little bit of underfitting.

This paper was originally created for the 2009 Sawtooth Software Conference. This paper focuses on what happens during the estimation of CBC/HB utilities. It takes a naïve approach assuming no knowledge of statistics or math beyond basic algebra. It begins by providing a brief overview about the issues surrounding individual level estimation and why CBC/HB receives the attention it does. The paper then goes into more detail about what the algorithm is intuitively doing and what each step is trying to accomplish.The paper is non-technical in nature and is written for the absolute beginner. It is intended to be at a level that can be given to a beginning junior analyst or a non-mathematically inclined client.

The basic (generic) hierarchical Bayes estimation that the first versions of Sawtooth Software's CBC/HB program supported assumed that respondents were drawn from a single, multivariate-normal distribution. All respondents were "shrunk" to some degree or another toward the population means. This article describes relaxing the assumption of a single normal population via covariates included in the upper-level model of the hierarchy. Covariates are segmentation variables that are predictive of respondent's choices. When used in HB, they allow shrinkage to respondent-specific locations in the distribution, depending on the characteristics of the covariates. The results can lead to modest improvement in predictive accuracy, but substantial improvement in terms of differences between segments of respondents on the means. This is of substantial benefit to segmentation research, where the generic model could obscure the true differences between respondents.

Greg Allenby and Peter Rossi describe the history of HB methods as they relate to marketing research methods. They describe the theory behind HB, the challenges in implementing HB methods for marketing research in the 1990s, and what they see as the future applications of HB within marketing research. They predict that over the next 10 years, HB will enable researchers to develop more rich models of consumer behavior. We will extend the standard preference models to incorporate more complex behavioral components, including screening rules in conjoint analysis (conjunctive, disjunctive, compensatory), satiation, scale usage, and inter-dependent preferences among consumers. New models will approach preference from the multitude of basic concerns and interests that give rise to needs. Common to all these problems is a dramatic increase in the number of explanatory variables.

Hierarchical Bayes estimation for choice data represents one of the most successful new developments in our field. HB has proven robust for ratings-based conjoint, ACA, and full-profile CBC projects. Tests comparing HB to other methods of part worth estimation have generally favored HB. However, two anomalies specific to HB estimation have caused us some puzzlement and concern.

The "omitted" level in effects coding for conjoint analysis results in overstatement of the variance, and in extreme cases (very sparse data and very many levels within an attribute) biased point estimates.
‍

HB was demonstrated to have problems in individual-level estimation for some partial-profile CBC data sets.
‍

This paper shows that the problems above can be controlled or even solved by setting proper "priors" in HB. The anomalies therefore do not point to a weakness in HB methods, but simply illustrate that we were not defining the models properly in certain circumstances. HB researchers should be aware of the kinds of data sets that can challenge "generic" HB estimation under Sawtooth Software's default settings, and learn to manage these through more proper specification of the priors.