LLMs are impacting nearly every area of market research. There are two key areas where they are impacting the quantitative domain specifically: analysis of dark data (e.g. open ended responses) at scale, and “synthetic responses”. We’ll review different approaches to the application of LLMs to open-ended coding using hands-on examples, and evaluate when each approach is best used. We will then briefly review the existing literature and state of the art on synthetic respondents.

Data fraud is penetrating every aspect of our industry, including panels, customer databases, and phone research. With no safe place to turn, researchers are left to vet the quality of respondents on our own. The methods cheaters use evolve over time, making it a challenge to keep up with them. This paper summarizes current approaches you can apply to detect data fraud, both real-time and during data analysis.

When it comes to online surveys, we need to trust that respondents understand and interpret the tasks the way we expect them to. To investigate this, we conduct a series of qualitative interviews aimed to observe how respondents answer, understand and perceive conjoint exercises. Our findings, among others, show that instructions are crucial but often too lengthy, choices depend on anchoring and choices are driven by variety seeking or opportunity to trial new things.

This research paper explores the potential of utilizing eye-tracking technology to advance conjoint modeling. It explores how eye-tracking data can enhance data quality, validate models, and improve UX in conjoint questionnaires. Key inquiries include participant attention to text and images, the emergence of simplification heuristics, attribute sequencing effects, and the impact of pre-exercise learning sections. The study offers valuable insights into the integration of eye tracking in conjoint research and its implications on model performance.

Product needs vary not only between individuals but can also vary within the same individual based on the situation in which the product is used. We introduce a new method which enables researchers to address this concern: Situational MaxDiff. Situational MaxDiff investigates participant needs based in pre-defined situations. We present a method that leverages pooled data to keep questionnaire time manageable and still produce precise findings which can improve segmentation findings.

We will explore a variety of improvements that can be employed to improve the quality of a mobile questionnaire until it can be described as “mobile first” instead of just “mobile friendly”. We will examine KPIs like dropout rate and data quality both in old studies and in a custom research specifically designed for this purpose. Using this we will quantify the benefits for and identify specific approaches to improve mobile surveys.

The purpose of the presentation will be to present the use and capacity of Artificial Intelligence (AI) on open-ended responses in survey data, particularly as it relates to the analysis stage of a survey data project. Moreover, the presentation will introduce the audience to best practices in the use of two open-ended analysis techniques: topic analysis and sentiment analysis.

This paper explores the impact of GEN AI in research projects, highlighting real-world applications and insights. It covers questionnaire development, dynamic surveys, data quality management, and text analytics. Key findings include the effectiveness of AI-generated questionnaires, the advantages of dynamic surveys in capturing respondent sentiment, and the potential of fine-tuned models for data quality management. Additionally, machine-driven text analytics shows promise in sentiment analysis and theme identification, achieving near human-level performance on larger datasets.

The post-COVID surge in online shopping underscored the need for effective E-Commerce websites, opening research avenues beyond traditional methods. To address this, we developed an immersive Conjoint driven E-commerce replica for surveys. This approach offers valuable behavioural insights and extends a range of research possibilities, from pricing and user engagement metrics. The paper delves into its methodology, initial findings, and scalability, with the potential to transform market research and enhance e-commerce customer insights.

Share of Search (SOS) is currently generating significant buzz in the marketing industry. Marketing effectiveness expert, Les Binet, has suggested that Excess Share of Search (ESOS) can predict future changes in market share. We test Binet's theory using hundreds of technology brands and show how we can enhance the predictive accuracy further by improving the ESOS methodology. Our approach offers a highly valuable quick and cost-effective alternative to costly and time-consuming bespoke forecasting models.

This research introduces a practical method for assessing the collective impact of feature sets on adoption rates, extending beyond traditional MaxDiff and TURF analysis. While TURF assumes that one feature can reach a customer, our approach recognizes the necessity of multiple must-have features for convincing consumers. Particularly valuable when evaluating extensive feature lists with numerous "must-haves" and when a Conjoint exercise isn't feasible, our method surpasses conventional techniques such as simple select questions or TURF.

This presentation shows how to improve market segmentation studies with the judicious use of machine learning methods. We start by outlining the four steps of a typical market segmentation process: • Selecting basis variables • Creating segments • Profiling segments • Predicting segment membership Traditionally, the industry used taxonomic and statistical methods for these steps. In a few specific instances, those methods are the correct ones to use. But they don’t generalize well to all or even many cases of market segmentation. Frequently over-used in lazy (segment creation via cluster analysis) or simply incorrect (variable selection via factor analysis) ways, they almost always be improved upon. Machine learning methods can improve the segmentation research process at various of its four steps: • Sometimes as additional tools to enhance what we were able to do before with traditional statistics (e.g., decision trees for segment creation) • Sometimes as improvements over what traditional statistics can do (e.g., random forests for segment prediction) • Sometimes to do things properly that misunderstood statistical methods are inappropriately tasked to do (e.g., factor analysis is frequently, and incorrectly, thought to be a data pre-treatment method for segmentation analysis, but machine learning techniques are the vastly better way to select basis variables).

Drivers analysis is a key part of any analyst’s toolbox, but many of the techniques in use today can still be demonstrably improved upon. In order to assist with this, Key Drivers Analysis (incorporating Johnson’s e) is now available within Lighthouse Studio v9.16. Join Walter Williams and Dean Tindall as they introduce you to this newly released module and the reasons why we chose to implement this particular method.

Choice modelling is often used to optimise the configuration of products/services or to create a hierarchy ranking of items. This paper will investigate the use of large language models (LLMs) to replicate the results of previous Conjoint and MaxDiff projects. We will test whether LLMs can be used to generate realistic responses to tasks, whether responses are consistent across LLMs and parameter settings, and whether you can train LLM’s to produce better quality data.

We examine how AI large language models compare with direct stated pricing models such as Van Westendorp and indirect derived pricing models such as a price brand trade-off exercise when determining optimal pricing recommendations for a product line across a variety of distribution channels.

Designs where many attributes in a task overlap completely and where overlapped levels are shaded reportedly have several benefits: the less complex choice task results in reduced response error, dropouts and attribute non-attendance. We seek to confirm these findings and extend them to see whether these designs will be immune to the ill-effects of dominating attributes and whether they will have fewer respondents giving random responses.

Previous research on many-item Best Worst tasks has shown that Sparse BW designs have generally outperformed Express BW designs, especially regarding out-of-sample predictions. Recently, a suggestion was made to improve Express BW designs by including a small, fixed number of items [3-5] across all respondents. We undertake this research to ascertain whether item seeding results in better out-of-sample predictions than traditional Express BW designs have without this seeding.

Building on our paper presented at the A&I Conference 2023, this research extends the application of proportional pricing in conjoint analysis to manufacturer brands. It examines the effects of omitting explicit monetary values on consumer choice behaviour and, consequently, price elasticity. It also assesses how well proportional pricing method predicts real-world market performance, validating it as a useful alternative especially for complex product offerings where traditional monetary pricing is elusive.

Advances in automation technology and artificial intelligence have made it easier to create and deploy bots for various purposes, including survey participation. As AI technology becomes more sophisticated, survey bots can become more intelligent and difficult to detect. This can exacerbate the challenges associated with identifying and preventing bots from participating in surveys, which is a concern for the market research industry. Consequently, market researchers need to continuously adapt and develop more robust methods to distinguish between genuine survey participants and automated bots. However, AI bots also have weaknesses (so far) and we at Numerious developed a new approach that is designed to exploit their weaknesses.

Fairness in clustering refers to the requirement that certain respondent types like minority grouping have adequate representation across clusters to avoid bias. It has emerged as an actively researched area in last few years. Fairness can be a practical concern in market segmentation. I will briefly introduce fair clustering, then focus on comparing selected promising and practical algorithms on both benchmark and real data sets. I will discuss method choices, fairness specification, computation, and implementation.

Social media platforms have emerged as influential spaces where individuals and brands interact, share information, and influence opinions. The huge amount of data generated on these platforms contains rich information but presents challenges in harnessing it. We introduced a quantitative framework to navigate this complexity, highlighting the process of data collection, analysis, and insight generation, culminating in the development of a brand benchmarking framework. The framework can be applied to any industry with relevant customizations.

To capture consumer’s response to psychological price points, ideally, every Brand-SKU X price point should be sufficiently exposed and estimated as part-worth (non-linear) for every respondent. Many a times, this is not feasible. So, we explored creating ASD price-groups such that brand-SKUs with similar tested price-levels and similar response to these price changes(after studying count analysis), fall in same price-group. We then estimate utilities for price-groups. This method restructures the same information that is already captured, without increasing sample size and performs well at estimating psychological price points.

Join us for an exciting session as we reveal the latest enhancements and updates to Sawtooth Software's new Discover tool. Discover is evolving into a powerhouse of research capabilities, and we're eager to showcase its newest features. During this session we'll cover all the amazing updates that have recently been added to Discover, and we will also demonstrate some insider tips and tricks to help you maximize its potential. We will conclude by sharing our future plans and exciting developments on the horizon. Don't miss this opportunity to elevate your research game and unleash your inner superhero with Discover!

In complex choice situations it is unlikely that consumers will break down each attribute and each level, assign values to the levels, integrate these values and then choose the highest overall value. Instead, respondents may, at least partially, make choices based on a holistic evaluation of the features. We found that the holistic variable was highly statistically significant, and we found several tipping points in the utility function of the holistic variable.

In this presentation we illustrate how latent class (LC) clustering should be performed on HB utilities to avoid strange results, and describe the situations where Latent GOLD’s scale-adjusted (SALC) model would be expected to result in more meaningful segments. Our MaxDiff example yields 88% agreement with the gold standard, vs. 60% when clustering is done incorrectly. Attendees will take away new insights into the segmentation process when latent class models are applied to HB utilities.

As conference attendees have seen over the past few years, these are exciting times for practitioners that look to improve our toolkit for marketing segmentation. But are we including the right variables in our segmentation studies? This paper will look at combining automated search algorithms and variations on latent class analysis to see whether we can demonstratively present better resolution in our findings compared to previous segmentation work with commercially accessible software.

Respondents are generally assumed to become tired by taking repeating CBC exercises. However, prior research points to the absence of Respondent Fatigue in the CBC part of surveys. This research confronts this by stress-testing respondents in CBC through 3 studies with 32 tasks each and different levels of complexity. We tested for Stated Fatigue, Implied Fatigue (trap questions), and Derived Fatigue (model fit). Results show Respondent Fatigue exists, but the implications are limited for practitioners.

Six decades ago, Luce and Tukey's seminal paper marked the inception of conjoint analysis. Now, it's time to delve into the historical evolution of conjoint analysis over the past 60 years. In particular, we'll explore a concise history of advancements in various conjoint models, the estimation of part-worth utilities, the creation of experimental designs, and how these innovations have shaped research outcomes. We'll also examine which of the challenges that emerged during this period have been successfully addressed and which questions still remain open.

Artificial intelligence and human wisdom will transform how and what do in the future, but we must learn our way towards that future. We aspire to ensure that synthesis becomes faster and richer; to develop a ‘no waste research’ posture; to unleash KM to broaden reach, deepen insight generation, and leverage methodological and analytical techniques that leverage AI in quantitative and qualitative approaches.

The combination of CBC and market simulation is a powerful tool for understanding consumers’ choices when presented with a variety of audio products. But what if some of the products are configurable, with the option to unlock more features for an additional fee after purchase? Tune in as we compare several options for handling configurable products in a choice exercise, along with all the New Wave song references you’ve come to expect from a Bose presentation.

As General Manager of Commercial Intelligence at Coca-Cola, Renata Soares, along with Whitelaw & Co., developed a market simulation model based on Sawtooth Software’s CBC with HB analysis of utilities. The model guided Coke’s new package introductions in the US, market share expansion in Brazil, response to sugar tax increases in Mexico, and many other initiatives. Renata will describe how they developed the simulator and implemented it in a variety of applications within Coke.

Join Supernal as we delve into revolutionizing urban travel with electric vertical takeoff and landing (eVTOL) aircraft. Our presentation will uncover key findings from our recent U.S.-based conjoint analysis, detailing consumer attitudes toward pricing, shared vs. private rides, and market potential. Learn how we’re breaking down barriers to make aerial mobility a reality for everyone and discover the strategic shifts this research is driving for a more efficient, sustainable urban future.

Tribal Credit is a Fintech startup providing credit to Small-Medium Enterprise clients in emerging markets in the form of credit cards, revolving-capital accounts, and local/international payments. As a Silicon Valley startup, there's a strong culture of making data-driven decisions but also developing their products at break-neck speed. These two values often clash. We'll dive into the complexities and challenges faced when trying to deliver high quality research in a company that is changing rapidly.

Come, take a tour on a conjoint safari at P&G. We’ll discuss how this discrete choice method is just the ticket for product / concept design in the CPG industry. Cast all your “prohibitions” aside! Along the journey, we’ll try to avoid picking up bad panelists thumbing for a ride, while making sure that the good ones (and their research guides) get a great view of the product stimuli from beginning to end. We’ll observe how these “interactive” stimuli tend to travel together out there in the wild. We’ll also search deep within the design space until we finally spot the perfect specimen from multiple objectives. Who wouldn’t be “willing to pay” to see that?

As you gain more experience in choice modeling, you come to realize that CBC (Choice-Based Conjoint) is a specific and limited case within a broader family of choice experiments. Sometimes clients push examples upon you that bring you to this realization. Sometimes you see examples at conferences and in white papers that open your eyes and spur your interest. There is, of course, the classic example resembling a fast-food restaurant where respondents can pick one of multiple items to add to their cart for purchase. There are other examples that don’t immediately strike you as being an MBC, but that exceed CBC’s capabilities and can be handled with Sawtooth Software’s MBC software. Examples include asking respondents if they will buy the product, rent the product, or do neither. Or, situational choice in which the choice of multiple fixed alternatives is a function of respondent characteristics, buying situation characteristics, or patient characteristics. Join Bryan Orme in this 45-minute clinic to learn about the possibilities with MBC!

AI has the potential to radically transform the way insights are generated – what research we do and how we do it – but our industry is just starting out on this journey. In this session, we’ll discuss some of the initial explorations that we are doing at Microsoft. We’ll cover some case studies, such as comparing synthetic responses via GPT to human responses (including conjoint to simulate customer decisions involving feature/price trade-offs), and other AI use cases such as text/video analytics.

Some questions can seem unanalyzable until you’re forced to find the right solution. In a series of failed experiments, we worked through multicollinearity, low sample sizes, and confounding seasonal effects to solve a core problem: what drives guest satisfaction at a theme park? Learn about our journey to a viable analysis, including the methodologies we combined and how we overcame these data challenges.

Although many of us prefer choice-based conjoint methods for conducting survey-based pricing research, there are opportunities where van Westendorp PSM and monadic pricing experiments come in handy. In this presentation, Bryan Orme shows ways to make these widely used pricing approaches (PSM and monadic) work better than traditional applications/analysis we often see.

Our research, utilizing Sawtooth Software's ACBC, identified key customer segments that have now reached a crucial implementation phase and are being evaluated through A/B testing. Our methodology underscores the importance of achieving critical mass for the broad acceptance of data-driven marketing within sizable corporations. By equipping business units with advanced analysis and Simulation Tools, and ensuring comprehensive training and widespread buy-in, the application of our conjoint results has taken on momentum, evolving, and growing organically. Here we want to explain the process to achieve this broad buy-in.

MaxDiff surveys and importance scores have become an instrumental and frequent tool within our company for an abundance of use cases. Although the core of MaxDiff results was very effective in informing business strategy and decisions, we have now tested and utilized multiple additions and methods to a MaxDiff approach that have increased the interpretability for business stakeholders and also the potential actionability and overall impact of the research.