LLMs are impacting nearly every area of market research. There are two key areas where they are impacting the quantitative domain specifically: analysis of dark data (e.g. open ended responses) at scale, and “synthetic responses”. We’ll review different approaches to the application of LLMs to open-ended coding using hands-on examples, and evaluate when each approach is best used. We will then briefly review the existing literature and state of the art on synthetic respondents.
LLMs are impacting nearly every area of market research. There are two key areas where they are impacting the quantitative domain specifically: analysis of dark data (e.g. open ended responses) at scale, and “synthetic responses”. We’ll review different approaches to the application of LLMs to open-ended coding using hands-on examples, and evaluate when each approach is best used. We will then briefly review the existing literature and state of the art on synthetic respondents.
Data fraud is penetrating every aspect of our industry, including panels, customer databases, and phone research. With no safe place to turn, researchers are left to vet the quality of respondents on our own. The methods cheaters use evolve over time, making it a challenge to keep up with them. This paper summarizes current approaches you can apply to detect data fraud, both real-time and during data analysis.
Data fraud is penetrating every aspect of our industry, including panels, customer databases, and phone research. With no safe place to turn, researchers are left to vet the quality of respondents on our own. The methods cheaters use evolve over time, making it a challenge to keep up with them. This paper summarizes current approaches you can apply to detect data fraud, both real-time and during data analysis.
When it comes to online surveys, we need to trust that respondents understand and interpret the tasks the way we expect them to. To investigate this, we conduct a series of qualitative interviews aimed to observe how respondents answer, understand and perceive conjoint exercises. Our findings, among others, show that instructions are crucial but often too lengthy, choices depend on anchoring and choices are driven by variety seeking or opportunity to trial new things.
When it comes to online surveys, we need to trust that respondents understand and interpret the tasks the way we expect them to. To investigate this, we conduct a series of qualitative interviews aimed to observe how respondents answer, understand and perceive conjoint exercises. Our findings, among others, show that instructions are crucial but often too lengthy, choices depend on anchoring and choices are driven by variety seeking or opportunity to trial new things.
This research paper explores the potential of utilizing eye-tracking technology to advance conjoint modeling. It explores how eye-tracking data can enhance data quality, validate models, and improve UX in conjoint questionnaires. Key inquiries include participant attention to text and images, the emergence of simplification heuristics, attribute sequencing effects, and the impact of pre-exercise learning sections. The study offers valuable insights into the integration of eye tracking in conjoint research and its implications on model performance.
This research paper explores the potential of utilizing eye-tracking technology to advance conjoint modeling. It explores how eye-tracking data can enhance data quality, validate models, and improve UX in conjoint questionnaires. Key inquiries include participant attention to text and images, the emergence of simplification heuristics, attribute sequencing effects, and the impact of pre-exercise learning sections. The study offers valuable insights into the integration of eye tracking in conjoint research and its implications on model performance.
Product needs vary not only between individuals but can also vary within the same individual based on the situation in which the product is used. We introduce a new method which enables researchers to address this concern: Situational MaxDiff. Situational MaxDiff investigates participant needs based in pre-defined situations. We present a method that leverages pooled data to keep questionnaire time manageable and still produce precise findings which can improve segmentation findings.
Product needs vary not only between individuals but can also vary within the same individual based on the situation in which the product is used. We introduce a new method which enables researchers to address this concern: Situational MaxDiff. Situational MaxDiff investigates participant needs based in pre-defined situations. We present a method that leverages pooled data to keep questionnaire time manageable and still produce precise findings which can improve segmentation findings.
We will explore a variety of improvements that can be employed to improve the quality of a mobile questionnaire until it can be described as “mobile first” instead of just “mobile friendly”. We will examine KPIs like dropout rate and data quality both in old studies and in a custom research specifically designed for this purpose. Using this we will quantify the benefits for and identify specific approaches to improve mobile surveys.
We will explore a variety of improvements that can be employed to improve the quality of a mobile questionnaire until it can be described as “mobile first” instead of just “mobile friendly”. We will examine KPIs like dropout rate and data quality both in old studies and in a custom research specifically designed for this purpose. Using this we will quantify the benefits for and identify specific approaches to improve mobile surveys.
The purpose of the presentation will be to present the use and capacity of Artificial Intelligence (AI) on open-ended responses in survey data, particularly as it relates to the analysis stage of a survey data project. Moreover, the presentation will introduce the audience to best practices in the use of two open-ended analysis techniques: topic analysis and sentiment analysis.
The purpose of the presentation will be to present the use and capacity of Artificial Intelligence (AI) on open-ended responses in survey data, particularly as it relates to the analysis stage of a survey data project. Moreover, the presentation will introduce the audience to best practices in the use of two open-ended analysis techniques: topic analysis and sentiment analysis.
This paper explores the impact of GEN AI in research projects, highlighting real-world applications and insights. It covers questionnaire development, dynamic surveys, data quality management, and text analytics. Key findings include the effectiveness of AI-generated questionnaires, the advantages of dynamic surveys in capturing respondent sentiment, and the potential of fine-tuned models for data quality management. Additionally, machine-driven text analytics shows promise in sentiment analysis and theme identification, achieving near human-level performance on larger datasets.
This paper explores the impact of GEN AI in research projects, highlighting real-world applications and insights. It covers questionnaire development, dynamic surveys, data quality management, and text analytics. Key findings include the effectiveness of AI-generated questionnaires, the advantages of dynamic surveys in capturing respondent sentiment, and the potential of fine-tuned models for data quality management. Additionally, machine-driven text analytics shows promise in sentiment analysis and theme identification, achieving near human-level performance on larger datasets.
The post-COVID surge in online shopping underscored the need for effective E-Commerce websites, opening research avenues beyond traditional methods. To address this, we developed an immersive Conjoint driven E-commerce replica for surveys. This approach offers valuable behavioural insights and extends a range of research possibilities, from pricing and user engagement metrics. The paper delves into its methodology, initial findings, and scalability, with the potential to transform market research and enhance e-commerce customer insights.
The post-COVID surge in online shopping underscored the need for effective E-Commerce websites, opening research avenues beyond traditional methods. To address this, we developed an immersive Conjoint driven E-commerce replica for surveys. This approach offers valuable behavioural insights and extends a range of research possibilities, from pricing and user engagement metrics. The paper delves into its methodology, initial findings, and scalability, with the potential to transform market research and enhance e-commerce customer insights.
Share of Search (SOS) is currently generating significant buzz in the marketing industry. Marketing effectiveness expert, Les Binet, has suggested that Excess Share of Search (ESOS) can predict future changes in market share. We test Binet's theory using hundreds of technology brands and show how we can enhance the predictive accuracy further by improving the ESOS methodology. Our approach offers a highly valuable quick and cost-effective alternative to costly and time-consuming bespoke forecasting models.
Share of Search (SOS) is currently generating significant buzz in the marketing industry. Marketing effectiveness expert, Les Binet, has suggested that Excess Share of Search (ESOS) can predict future changes in market share. We test Binet's theory using hundreds of technology brands and show how we can enhance the predictive accuracy further by improving the ESOS methodology. Our approach offers a highly valuable quick and cost-effective alternative to costly and time-consuming bespoke forecasting models.
This research introduces a practical method for assessing the collective impact of feature sets on adoption rates, extending beyond traditional MaxDiff and TURF analysis. While TURF assumes that one feature can reach a customer, our approach recognizes the necessity of multiple must-have features for convincing consumers. Particularly valuable when evaluating extensive feature lists with numerous "must-haves" and when a Conjoint exercise isn't feasible, our method surpasses conventional techniques such as simple select questions or TURF.
This research introduces a practical method for assessing the collective impact of feature sets on adoption rates, extending beyond traditional MaxDiff and TURF analysis. While TURF assumes that one feature can reach a customer, our approach recognizes the necessity of multiple must-have features for convincing consumers. Particularly valuable when evaluating extensive feature lists with numerous "must-haves" and when a Conjoint exercise isn't feasible, our method surpasses conventional techniques such as simple select questions or TURF.
This presentation shows how to improve market segmentation studies with the judicious use of machine learning methods. We start by outlining the four steps of a typical market segmentation process: • Selecting basis variables • Creating segments • Profiling segments • Predicting segment membership Traditionally, the industry used taxonomic and statistical methods for these steps. In a few specific instances, those methods are the correct ones to use. But they don’t generalize well to all or even many cases of market segmentation. Frequently over-used in lazy (segment creation via cluster analysis) or simply incorrect (variable selection via factor analysis) ways, they almost always be improved upon. Machine learning methods can improve the segmentation research process at various of its four steps: • Sometimes as additional tools to enhance what we were able to do before with traditional statistics (e.g., decision trees for segment creation) • Sometimes as improvements over what traditional statistics can do (e.g., random forests for segment prediction) • Sometimes to do things properly that misunderstood statistical methods are inappropriately tasked to do (e.g., factor analysis is frequently, and incorrectly, thought to be a data pre-treatment method for segmentation analysis, but machine learning techniques are the vastly better way to select basis variables).
This presentation shows how to improve market segmentation studies with the judicious use of machine learning methods. We start by outlining the four steps of a typical market segmentation process: • Selecting basis variables • Creating segments • Profiling segments • Predicting segment membership Traditionally, the industry used taxonomic and statistical methods for these steps. In a few specific instances, those methods are the correct ones to use. But they don’t generalize well to all or even many cases of market segmentation. Frequently over-used in lazy (segment creation via cluster analysis) or simply incorrect (variable selection via factor analysis) ways, they almost always be improved upon. Machine learning methods can improve the segmentation research process at various of its four steps: • Sometimes as additional tools to enhance what we were able to do before with traditional statistics (e.g., decision trees for segment creation) • Sometimes as improvements over what traditional statistics can do (e.g., random forests for segment prediction) • Sometimes to do things properly that misunderstood statistical methods are inappropriately tasked to do (e.g., factor analysis is frequently, and incorrectly, thought to be a data pre-treatment method for segmentation analysis, but machine learning techniques are the vastly better way to select basis variables).
Drivers analysis is a key part of any analyst’s toolbox, but many of the techniques in use today can still be demonstrably improved upon. In order to assist with this, Key Drivers Analysis (incorporating Johnson’s e) is now available within Lighthouse Studio v9.16. Join Walter Williams and Dean Tindall as they introduce you to this newly released module and the reasons why we chose to implement this particular method.
Drivers analysis is a key part of any analyst’s toolbox, but many of the techniques in use today can still be demonstrably improved upon. In order to assist with this, Key Drivers Analysis (incorporating Johnson’s e) is now available within Lighthouse Studio v9.16. Join Walter Williams and Dean Tindall as they introduce you to this newly released module and the reasons why we chose to implement this particular method.
Choice modelling is often used to optimise the configuration of products/services or to create a hierarchy ranking of items. This paper will investigate the use of large language models (LLMs) to replicate the results of previous Conjoint and MaxDiff projects. We will test whether LLMs can be used to generate realistic responses to tasks, whether responses are consistent across LLMs and parameter settings, and whether you can train LLM’s to produce better quality data.
Choice modelling is often used to optimise the configuration of products/services or to create a hierarchy ranking of items. This paper will investigate the use of large language models (LLMs) to replicate the results of previous Conjoint and MaxDiff projects. We will test whether LLMs can be used to generate realistic responses to tasks, whether responses are consistent across LLMs and parameter settings, and whether you can train LLM’s to produce better quality data.
We examine how AI large language models compare with direct stated pricing models such as Van Westendorp and indirect derived pricing models such as a price brand trade-off exercise when determining optimal pricing recommendations for a product line across a variety of distribution channels.
We examine how AI large language models compare with direct stated pricing models such as Van Westendorp and indirect derived pricing models such as a price brand trade-off exercise when determining optimal pricing recommendations for a product line across a variety of distribution channels.
Designs where many attributes in a task overlap completely and where overlapped levels are shaded reportedly have several benefits: the less complex choice task results in reduced response error, dropouts and attribute non-attendance. We seek to confirm these findings and extend them to see whether these designs will be immune to the ill-effects of dominating attributes and whether they will have fewer respondents giving random responses.
Designs where many attributes in a task overlap completely and where overlapped levels are shaded reportedly have several benefits: the less complex choice task results in reduced response error, dropouts and attribute non-attendance. We seek to confirm these findings and extend them to see whether these designs will be immune to the ill-effects of dominating attributes and whether they will have fewer respondents giving random responses.
Previous research on many-item Best Worst tasks has shown that Sparse BW designs have generally outperformed Express BW designs, especially regarding out-of-sample predictions. Recently, a suggestion was made to improve Express BW designs by including a small, fixed number of items [3-5] across all respondents. We undertake this research to ascertain whether item seeding results in better out-of-sample predictions than traditional Express BW designs have without this seeding.
Previous research on many-item Best Worst tasks has shown that Sparse BW designs have generally outperformed Express BW designs, especially regarding out-of-sample predictions. Recently, a suggestion was made to improve Express BW designs by including a small, fixed number of items [3-5] across all respondents. We undertake this research to ascertain whether item seeding results in better out-of-sample predictions than traditional Express BW designs have without this seeding.
Building on our paper presented at the A&I Conference 2023, this research extends the application of proportional pricing in conjoint analysis to manufacturer brands. It examines the effects of omitting explicit monetary values on consumer choice behaviour and, consequently, price elasticity. It also assesses how well proportional pricing method predicts real-world market performance, validating it as a useful alternative especially for complex product offerings where traditional monetary pricing is elusive.
Building on our paper presented at the A&I Conference 2023, this research extends the application of proportional pricing in conjoint analysis to manufacturer brands. It examines the effects of omitting explicit monetary values on consumer choice behaviour and, consequently, price elasticity. It also assesses how well proportional pricing method predicts real-world market performance, validating it as a useful alternative especially for complex product offerings where traditional monetary pricing is elusive.
Advances in automation technology and artificial intelligence have made it easier to create and deploy bots for various purposes, including survey participation. As AI technology becomes more sophisticated, survey bots can become more intelligent and difficult to detect. This can exacerbate the challenges associated with identifying and preventing bots from participating in surveys, which is a concern for the market research industry. Consequently, market researchers need to continuously adapt and develop more robust methods to distinguish between genuine survey participants and automated bots. However, AI bots also have weaknesses (so far) and we at Numerious developed a new approach that is designed to exploit their weaknesses.
Advances in automation technology and artificial intelligence have made it easier to create and deploy bots for various purposes, including survey participation. As AI technology becomes more sophisticated, survey bots can become more intelligent and difficult to detect. This can exacerbate the challenges associated with identifying and preventing bots from participating in surveys, which is a concern for the market research industry. Consequently, market researchers need to continuously adapt and develop more robust methods to distinguish between genuine survey participants and automated bots. However, AI bots also have weaknesses (so far) and we at Numerious developed a new approach that is designed to exploit their weaknesses.
Fairness in clustering refers to the requirement that certain respondent types like minority grouping have adequate representation across clusters to avoid bias. It has emerged as an actively researched area in last few years. Fairness can be a practical concern in market segmentation. I will briefly introduce fair clustering, then focus on comparing selected promising and practical algorithms on both benchmark and real data sets. I will discuss method choices, fairness specification, computation, and implementation.
Fairness in clustering refers to the requirement that certain respondent types like minority grouping have adequate representation across clusters to avoid bias. It has emerged as an actively researched area in last few years. Fairness can be a practical concern in market segmentation. I will briefly introduce fair clustering, then focus on comparing selected promising and practical algorithms on both benchmark and real data sets. I will discuss method choices, fairness specification, computation, and implementation.
Social media platforms have emerged as influential spaces where individuals and brands interact, share information, and influence opinions. The huge amount of data generated on these platforms contains rich information but presents challenges in harnessing it. We introduced a quantitative framework to navigate this complexity, highlighting the process of data collection, analysis, and insight generation, culminating in the development of a brand benchmarking framework. The framework can be applied to any industry with relevant customizations.
Social media platforms have emerged as influential spaces where individuals and brands interact, share information, and influence opinions. The huge amount of data generated on these platforms contains rich information but presents challenges in harnessing it. We introduced a quantitative framework to navigate this complexity, highlighting the process of data collection, analysis, and insight generation, culminating in the development of a brand benchmarking framework. The framework can be applied to any industry with relevant customizations.
To capture consumer’s response to psychological price points, ideally, every Brand-SKU X price point should be sufficiently exposed and estimated as part-worth (non-linear) for every respondent. Many a times, this is not feasible. So, we explored creating ASD price-groups such that brand-SKUs with similar tested price-levels and similar response to these price changes(after studying count analysis), fall in same price-group. We then estimate utilities for price-groups. This method restructures the same information that is already captured, without increasing sample size and performs well at estimating psychological price points.
To capture consumer’s response to psychological price points, ideally, every Brand-SKU X price point should be sufficiently exposed and estimated as part-worth (non-linear) for every respondent. Many a times, this is not feasible. So, we explored creating ASD price-groups such that brand-SKUs with similar tested price-levels and similar response to these price changes(after studying count analysis), fall in same price-group. We then estimate utilities for price-groups. This method restructures the same information that is already captured, without increasing sample size and performs well at estimating psychological price points.
Join us for an exciting session as we reveal the latest enhancements and updates to Sawtooth Software's new Discover tool. Discover is evolving into a powerhouse of research capabilities, and we're eager to showcase its newest features. During this session we'll cover all the amazing updates that have recently been added to Discover, and we will also demonstrate some insider tips and tricks to help you maximize its potential. We will conclude by sharing our future plans and exciting developments on the horizon. Don't miss this opportunity to elevate your research game and unleash your inner superhero with Discover!
Join us for an exciting session as we reveal the latest enhancements and updates to Sawtooth Software's new Discover tool. Discover is evolving into a powerhouse of research capabilities, and we're eager to showcase its newest features. During this session we'll cover all the amazing updates that have recently been added to Discover, and we will also demonstrate some insider tips and tricks to help you maximize its potential. We will conclude by sharing our future plans and exciting developments on the horizon. Don't miss this opportunity to elevate your research game and unleash your inner superhero with Discover!
In complex choice situations it is unlikely that consumers will break down each attribute and each level, assign values to the levels, integrate these values and then choose the highest overall value. Instead, respondents may, at least partially, make choices based on a holistic evaluation of the features. We found that the holistic variable was highly statistically significant, and we found several tipping points in the utility function of the holistic variable.
In complex choice situations it is unlikely that consumers will break down each attribute and each level, assign values to the levels, integrate these values and then choose the highest overall value. Instead, respondents may, at least partially, make choices based on a holistic evaluation of the features. We found that the holistic variable was highly statistically significant, and we found several tipping points in the utility function of the holistic variable.
In this presentation we illustrate how latent class (LC) clustering should be performed on HB utilities to avoid strange results, and describe the situations where Latent GOLD’s scale-adjusted (SALC) model would be expected to result in more meaningful segments. Our MaxDiff example yields 88% agreement with the gold standard, vs. 60% when clustering is done incorrectly. Attendees will take away new insights into the segmentation process when latent class models are applied to HB utilities.
In this presentation we illustrate how latent class (LC) clustering should be performed on HB utilities to avoid strange results, and describe the situations where Latent GOLD’s scale-adjusted (SALC) model would be expected to result in more meaningful segments. Our MaxDiff example yields 88% agreement with the gold standard, vs. 60% when clustering is done incorrectly. Attendees will take away new insights into the segmentation process when latent class models are applied to HB utilities.
As conference attendees have seen over the past few years, these are exciting times for practitioners that look to improve our toolkit for marketing segmentation. But are we including the right variables in our segmentation studies? This paper will look at combining automated search algorithms and variations on latent class analysis to see whether we can demonstratively present better resolution in our findings compared to previous segmentation work with commercially accessible software.
As conference attendees have seen over the past few years, these are exciting times for practitioners that look to improve our toolkit for marketing segmentation. But are we including the right variables in our segmentation studies? This paper will look at combining automated search algorithms and variations on latent class analysis to see whether we can demonstratively present better resolution in our findings compared to previous segmentation work with commercially accessible software.
Respondents are generally assumed to become tired by taking repeating CBC exercises. However, prior research points to the absence of Respondent Fatigue in the CBC part of surveys. This research confronts this by stress-testing respondents in CBC through 3 studies with 32 tasks each and different levels of complexity. We tested for Stated Fatigue, Implied Fatigue (trap questions), and Derived Fatigue (model fit). Results show Respondent Fatigue exists, but the implications are limited for practitioners.
Respondents are generally assumed to become tired by taking repeating CBC exercises. However, prior research points to the absence of Respondent Fatigue in the CBC part of surveys. This research confronts this by stress-testing respondents in CBC through 3 studies with 32 tasks each and different levels of complexity. We tested for Stated Fatigue, Implied Fatigue (trap questions), and Derived Fatigue (model fit). Results show Respondent Fatigue exists, but the implications are limited for practitioners.
Six decades ago, Luce and Tukey's seminal paper marked the inception of conjoint analysis. Now, it's time to delve into the historical evolution of conjoint analysis over the past 60 years. In particular, we'll explore a concise history of advancements in various conjoint models, the estimation of part-worth utilities, the creation of experimental designs, and how these innovations have shaped research outcomes. We'll also examine which of the challenges that emerged during this period have been successfully addressed and which questions still remain open.
Six decades ago, Luce and Tukey's seminal paper marked the inception of conjoint analysis. Now, it's time to delve into the historical evolution of conjoint analysis over the past 60 years. In particular, we'll explore a concise history of advancements in various conjoint models, the estimation of part-worth utilities, the creation of experimental designs, and how these innovations have shaped research outcomes. We'll also examine which of the challenges that emerged during this period have been successfully addressed and which questions still remain open.
