Optimizing Proactive Patient Engagement Through SMART Trial Designs Of LLM-initiated Conversations
Population health initiatives often rely on cold outreach to close Healthcare Effectiveness Data and Information Set (HEDIS) gaps. Tailoring engagement messages to diverse patient populations remains a challenge, as traditional A/B testing requires large sample sizes to test only two alternative messaging strategies. With the increasing availability of large language models (LLMs) for outreach, population health programs are utilizing tiered strategies that deploy LLM text robots, then LLM voice robots, then human agents as a last resort to maximize cost-effectiveness. However, identifying which patients need each level of engagement is difficult. This study evaluates an alternative approach using Sequential Multiple Assignment Randomized Trials (SMART) to develop personalized communication strategies involving tiered automated and human agent outreach. We developed a microsimulation model to compare the performance of traditional A/B testing versus SMART designs in detecting heterogeneous treatment effects (HTEs) and optimizing patient engagement. Our findings demonstrate that while SMART designs consistently achieve better cost-effectiveness, their statistical power to detect HTEs varies depending on the specific HTE and its associated intervention stage. Notably, SMART designs exhibit a higher false positive rate compared to A/B testing, raising concerns about the potential for spurious findings. We conclude that while SMART designs offer promise for optimizing patient engagement, their increased risk of false positives necessitates further research to develop mitigation strategies before widespread implementation.