In HB estimation, covariates allow respondents to be smoothed toward the preferences of similar respondents rather than simply toward the population average. In principle, this should improve CBC predictions and reveal greater differences across segments. Yet reported improvements in hit rates have often been lackluster.
Out-of-sample (OOS) predictive validity provides a better way to assess whether covariates help and don't overfit. Sawtooth Software has developed a new OOS predictive validity tool for CBC studies using k-fold cross-validation. With 10-fold validation, for example, the model is estimated on 90% of respondents and tested on the remaining 10%, repeating the process until every respondent has served as a holdout. The method works even with hundreds of CBC design versions (blocks).
Come see the new OOS validator in action and learn what we’ve discovered by analyzing 20 CBC datasets. Covariates may achieve greater separation across segments while improving OOS predictive validity—or they may not. It depends on the dataset! And, what about "double-dipping the heterogeneity" with LCMNL segments as covariates?
