Cheaters are overtaking online panel data collection in staggering numbers. Both B2C and B2B audiences are impacted, some quite severely. In addition to the tools and techniques panel companies are bringing to fight this common enemy, researchers are having to up our game as well. In this presentation, we quantify the cheating problem and its trend over time. We then discuss best practices to identify cheaters and achieve the cleanest data set possible.

In the not too distant past, surveys were administered via pen and paper then manually analyzed and reported. Today, most survey data is collected electronically, but is still cleaned, analyzed, and reported manually. This takes time and allows for errors. The continuing demand for data means more efficient methods are needed. This presentation demonstrates the integration of Sawtooth Software with multiple systems to create a (nearly) automated study from first respondent to final word.

The Kano method assesses whether a proposed product or feature is attractive or delightful, as opposed to being merely necessary or unexciting. It gives a compelling answer but one that may not be correct. Its implementation uses questionable theory with low-quality survey items. We discuss the theory and method and present an empirical study of Kano reliability. Alternatives -- such as MaxDiff paired with traditional scales -- obtain similar benefits while using higher quality, more reliable methods.

How do you capture complex preferences while also keeping respondents engaged? Adaptive Choice-Based Conjoint (ACBC) provides multiple alternatives for accomplishing both goals. But procuring precious survey content for choice customization can be challenging, particularly when the benefits may not be immediately clear at the design phase of the project. This presentation will use a series of case studies to evaluate different inputs to choice experiments and help the analyst quantify the trade-off between design efficiency and response quality for each.

We propose an extension to Menu Based Conjoint (MBC) in the form of an add-on activity to gauge “Long Term” behavior and risks of customer attrition in the presence of price increases. Unlike previous methods, this activity does not require a separate conjoint activity, and only takes 30 seconds of survey time. Our estimates will be compared to existing approaches, and we will discuss successful applications to current research.

Traditional brand trackers focus on a brand’s ability to generate volume through measures of consideration and preference but often overlook the ability of a brand to charge a higher price. We show how conjoint analysis not only more accurately measures brand volumes but can also measure price premium. Furthermore, we show what drives brand premiums is distinctively different from what drives brand choice. Hence, we provide a new way of thinking on how brands should approach building brand equity.

Conjoint analysis is a method that is commonly used to optimise the configuration and pricing of products/services in a competitive environment. We typically want to understand the optimum “product” price, but often it is important to understand how much consumers are willing to pay for individual features of that product. This paper will compare three Willingness to Pay methods using empirical data sets to get a better understanding of the pros and cons of each method.

Have you finished a conjoint and ended with results that didn’t seem realistic? We demonstrate a case study of a way to integrate in distributional and informational assumptions into a subscription product where the raw conjoint results predicted unrealistic product launch expectations. We incorporate a variety of techniques including Otter’s method, putting in a distributional gate for prospective clients, and increasing the none parameter to increase the value of the external good or ‘none’ option.

Recent studies suggest that we can improve the precision of HB utility estimates by relating features of the choice task to the response time using Gaussian Process Regression. We show how this methodology performs under a variety of conditions of interest to practitioners. We also compare Gaussian Process models with linear models. We conclude with a discussion on the benefits and drawbacks of modeling response time alongside HB utility estimation and provide recommendations for practitioners.

Predicting the volume of a new product to be launched is a task that many researchers have done at least once. There are a variety of methodologies to accomplish the task, which are grouped in three families: real-life tests, benchmarking, and replication of market environments. In this paper, we show the pros and cons of each of them and explain an approach that oversteps some of the limitations of these methods by using them in combination.

Vector Autoregression (VAR) is often used for modeling sales of P items over time. VAR forecasts sales at time tnew using previous sales at tlag, coupled with attributes explaining those changes like price, distribution, and trend. We also model correlated sourcing among P items using a simulated population ~ Multivariate normal(α_lag, ∑). We show how to use conjoint experiments to inform ∑ and how that significantly improves predictions versus modeling ∑ from sales data alone.

Product line design is challenged by the diversity of demand in the market and the wide variety of product features available for sale. Some consumers have broad experience in the activities associated with a product category and others engage narrowly and rely on products in more limited ways. The number of product features and their levels is often large and difficult to characterize in a low-dimensional space. Evaluating marketing opportunities when there exist many usage contexts and product features requires the integration of information on what and when features are demanded, and by whom. We propose an archetypal analysis that combines data on the context of consumption, alternative product usage and feature preferences useful for product line design and management.

This talk discusses some general principles of discrete choice experiment design and introduces the conjointTools R package, which provides tools for assessing experiment designs and sample size requirements under a variety of conditions prior to fielding an experiment. The package contains functions for generating designs, simulating choice data according to assumed models, and estimating models using simulated data to inform sample size requirements, including using designs exported from Sawtooth Software.

Researchers often want to test how a set of items or attributes rank on multiple outcome metrics. One way to do this is by utilizing multiple MaxDiffs in a survey. The present study explores three approaches for this method, with special consideration given to the Tandem MaxDiff approach: presenting both outcome metrics on each screen of a single MaxDiff exercise.

Co-clustering is the simultaneous clustering of rows and columns of data. For example, when used for rating questions, or MaxDiff scores, it provides excellent insight into the underlying heterogeneity of this data: which respondents are similar and which items are similar. Adding covariates in the process – both for respondents and for the variables! – adds another layer of insights. This paper will show different ways of visualising co-clustered data and explain the heuristics on how to do co-clustering with covariates.

Market segmentation usually includes a typing tool that predicts who falls in which segment: vital for a successful implementation. Using commercial real-life datasets, we compare support vector machines and neural nets against a common approach in commercial practice: linear discriminant analysis. We show that ML approaches yield superior performance in terms of segment-level prediction, interpretability and expected profitability.

We present results to validate the findings from the Kurz/Binner 2021 Sawtooth Software Conference presentation that won best paper. Kurz/Binner demonstrated how a simple grid of 9 binary “Behavioral Calibration Questions” that probed how respondents regarded brand, innovation, and price could significantly improve the consistency of respondents’ CBC data and also their holdout predictive validity. We extend the binary questions to include 3 questions related to the importance of features.

In 2021 we presented how 9 simple behavioral questions can enhance choice models. The recall of past shopping experiences has a relevant impact on the results of the following choice exercise. Building on these findings we go the next step by using the behavioral framework to set up a Bayesian model for simultaneous attribute and parameter selection. This more sophisticated approach is used to understand the diversity of consumer preferences in even more detail.

Properly choosing the variables to include in cluster analysis allows analysts to address a set of serious problems that can impair segmentation studies. We'll use artificial data and empirical data sets to test three methods for variable selection: an automatic variable selection algorithm and two manual processes (stepwise discriminant analysis and a stepwise analysis of variance procedure. We'll find out whether any of these methods perform well enough to reduce barriers to successful segmentation.

We programmed a choice experiment that mimics the user experience of an online product comparison. Respondents saw dozens of concepts per screen and were given the option to sort and filter these for easier navigation through the product space. Aside from the respondent choices the experiment captured the usage of filters and additional self-stated information (outside of the experiment) about the respondents’ decision criteria, preferences, and willingness to pay. We compared the results of various ways of incorporating different types of these “secondary signals” into the estimation.

Complicated pricing studies can end up with a lot of part-worth levels of price. From recent conferences, a piecewise function that uses from 2 to 6 breakpoints (aside from the endpoints) is recommended and 12-20 breakpoints have been seen as potentially useful. We would want to investigate whether a dozen or more breakpoints is an overfit and we are better off with a more parsimonious approach. This investigation will also test if it would be best to have multiple simpler price effects. RLH, Holdout Hit Rate, and % of effects that don't need to be constrained will be used as testing criteria.

Often, we run into the challenge that we can only test a certain number of levels per attribute to make sure the estimation remains robust. However, there are scenarios in CBC studies where we want to test many more levels and we just want to find out what the best levels and combinations of levels are. By employing Thompson Sampling, we select preferred products to oversample for each new respondent.

In this session, we’ll discuss how research – primarily through conjoint – informs how Microsoft creates business models that align to customer value and support growth for the company. Research supports decisions over the unit of monetization, how we ‘package’, and how we price. We’ll discuss some of the ongoing challenges in understanding customer tradeoffs for new offerings that require consideration of pricing, packaging, and new business models simultaneously, and will review a couple of case studies involving business problems which required innovative research approaches.

Much of the innovation in MaxDiff has focused on design and analysis which is critical, but less innovation on teaching the MaxDiff methodology. Working with different R&D clients covering 50+ MaxDiff studies from different suppliers, I have found clients often fall into traps as they don’t understand the power of individual level utilities (eg. just looking at overall means). How can we devote innovation to client needs (such as education as well as presentation ready output)?

Tech and tech-adjacent companies increasingly staff "quantitative user experience research" -- a confusing role that overlaps design research, data science, marketing, and survey research. What is it? Do you need it? Can you do it? What is unique? As one of the first Quant UXers at Google, Chris will describe the role, who is a good (and bad) fit ... and give you time to "ask me anything." Bring questions and learn about a unique role at the intersection of human behavior and analytics.

PayPal and SKIM recently developed a pragmatic approach to isolate behavioral drivers of consumer shopping journey at the digital checkout stage among online merchants. We innovated on survey design experience by simulating checkout experience and combining conjoint and non-conjoint analytics. Further, we translated answers into actionable business recommendations that were accessible to non-technical audience as well. This innovation elevated Checkout workstream as a strategic priority at PayPal and adoption of these recommendations to the product, marketing and sales roadmap.

Understanding which product features drive customer value is a key component of the development cycle. This talk shows how we used menu-based conjoint (MBC) to identify a core trio of “must-have” wireless earbud features critical to customer satisfaction, as well as understand which additional “nice-to-have” features would most excite potential customers and add the most value to our next-generation products.

Troomi Wireless is a mobile communication company offering safe mobile phones for children. Their mission is to help kids discover their talents and safeguard them along the way. Troomi Wireless used choice-based conjoint to better understand their target audience. By analyzing the choice patterns, Troomi was able to fine tune its targeting and uncover unexpected insights. The choice patterns (i.e.: utility scores) were used to create respondent-level "affinity" scores. Results from the CBC contributed to a complete change in strategic direction.

Moderna is developing an mRNA vaccine for cytomegalovirus (CMV), the #1 infectious cause of birth defects and a leading cause of pediatric hearing loss. Despite CMV's prevalence, the vast majority of women in the United States have never heard of CMV. Seven different "reasons to care" statements were tested to explore which would resonate the most. We compare Likert scale to MaxDiff responses, finding much clearer ranking of messages from MaxDiff which provide more actionable guidance.

In June 2021, Google announced a new product offering - Google Workspace Individual. This talk walks through the research involved in taking Workspace Individual from concept to launch. The team leveraged Max Diff, Kano, and k-means clustering to keep the team in lock-step and inform feature development, segmentation, and monetization strategies.

A common strategic use for conjoint analysis is for estimating how much consumers are willing to pay (WTP) for new/improved features or a premium brand. There are numerous pitfalls involved to avoid getting unrealistic WTP estimates. Bryan discusses best practices for questionnaire design, data cleaning, and estimating WTP in the context of competitive offerings.

Andrea Mincione from MSC Cruises, and Andrea together with Emilio from Maiora Solutions present a consumer research conducted for Explora Journeys, the new ultra-luxury brand of MSC Cruises. The goal was to identify the best product settings and pricing strategy, to successfully enter the market and provide a consistent value proposition. CBC was key for the definition of the ultra-luxury cruise product, fare types and price levels, allowing Explora to prepare commercial initiatives and complete pricing plans ahead of September 2021 sales opening.