Fit for purpose? Assessing the robustness of discrete choice experiment designs in the context of evolving health technologies.

During product development and early real-world use new scientific evidence emerges and knowledge about the benefits and risks of medical products evolves. In rapidly changing settings, it is important to understand if stated preferences are robust to available knowledge and changes in attribute specification. During COVID-19 vaccine development, new information emerged about vaccine safety and duration of protection. We examine this issue using two discrete choice experiment (DCE) specifications fielded in France. In December 2020, a representative sample of 6007 adults aged 18-64 years was randomly allocated to either a RESTRICTED specification (n = 1201), which replicated an earlier June 2020 DCE study design, or a more complete REFERENCE specification (n = 4806) that included a booster-related attribute and expanded the serious side-effect risk attribute levels. We test whether the RESTRICTED specification affects the relative importance of common attributes and the consistency of respondents' choices. Relative attribute importance was assessed using partial log-likelihood measures, and choice consistency was examined using heteroskedastic mixed logit and latent class models. We find the RESTRICTED specification does not significantly change the relative importance ranking of the common attributes, but it is associated with higher choice consistency. Our results suggest that DCE results may be robust to attribute omission and level simplification in terms of attribute importance ranking, but that richer and more realistic specifications can increase decision noise. The results suggest the need to balance informational completeness against respondent burden when designing DCEs for evolving health technologies.