General Medicine · Medical Literature
Effects of a Safety User Interface Bundle on Verification Intentions in Generative AI Chat Use Among Older Chinese Adults: Randomized Vignette Survey.
Abstract
Generative AI chat systems are increasingly used for everyday information seeking, but plausible errors and omissions can mislead users when outputs are accepted without scrutiny. Interface-level safety cues may help users calibrate trust and engage in verification; yet, evidence in older Chinese adults remains limited. This study aimed to test whether adding a safety user interface (UI) bundle to a generative AI chat interface increases verification intention among older Chinese adults and to examine selected secondary outcomes, including reliance intention, trust calibration, perceived trustworthiness, comprehension, usability/readability, cognitive load, and a behavioral proxy of verification. We conducted a cross-sectional survey with an embedded randomized UI vignette experiment between May 22, 2025, and September 3, 2025. Chinese adults aged ≥60 years were recruited through community sites, outpatient clinic waiting areas, and WeChat (Tencent Holdings Ltd) groups, and randomized 1:1 to view screenshots of a baseline chat UI or a safety UI bundle containing generic source-label cues, and an uncertainty and verification nudge. Each participant completed 2 scenarios (service/travel decision and general well-being related to sleep/fatigue), followed by measures of verification intention (primary), reliance intention, trust calibration index, comprehension (0-8), perceived trustworthiness, usability/readability, cognitive load (0-10), manipulation checks, and a behavioral proxy (expanding optional "source information"). Analyses used intention-to-treat regression models with covariate adjustment. Of 214 consenting respondents who started the survey, 200 were included in the analysis (100 per arm). The safety UI bundle increased verification intention (mean 4.72, SD 0.63 vs 4.41, SD 0.59 on a 7-point scale; adjusted β=0.293, 95% CI 0.128-0.457; P<.001). Reliance intention did not increase (mean 4.97, SD 0.54 vs 5.03, SD 0.58; adjusted β=-0.105, 95% CI -0.239 to 0.029; P=.13). Trust calibration improved (trust calibration index: mean -0.29, SD 1.43 vs 0.29, SD 1.43; adjusted β=-0.567, 95% CI -1.005 to -0.129; P=.01). Expansion of optional source information was numerically higher, although the adjusted CI included the null (42% vs 27%; adjusted odds ratio [OR]=1.76, 95% CI 0.95-3.27; P=.07). Comprehension remained high and similar across arms (mean 6.33, SD 1.14 vs 6.32, SD 1.08; adjusted β=-0.132, 95% CI -0.428 to 0.163; P=.38). Perceived trustworthiness was modestly lower in the Safety UI arm (mean 5.20, SD 0.61 vs 5.39, SD 0.66; adjusted β=-0.199, 95% CI -0.382 to -0.016; P=.03). Usability/readability was unchanged, and cognitive load did not increase. Manipulation checks indicated higher cue recognition in the Safety UI arm. In a randomized static-vignette survey of older Chinese adults, a brief safety UI bundle was associated with higher verification intention and a trust calibration index consistent with lower overreliance risk, without detectable reductions in comprehension or usability/readability. Because the intervention was tested as a bundle using screenshots and generic source labels, findings should be interpreted as evidence for a practical interface-level strategy rather than proof that any single cue caused the observed effects.
Clinical relevance
Among Chinese adults aged ≥60 years, does adding a safety user interface (UI) bundle — comprising generic source-label cues, uncertainty signaling, and a verification nudge — to a generative AI chat interface increase the intention to verify AI-generated information compared with a baseline chat UI? __SNAPSHOT__ {"studyDesign":"Cross-sectional survey with an embedded parallel-group randomized vignette experiment (1:1 allocation); two within-participant scenarios per participant","setting":"Community sites, outpatient clinic waiting areas, and WeChat (Tencent Holdings Ltd) groups; country not explicitly stated but context is China (Chinese-speaking adults recruited via WeChat and community channels)","sampleSize":"Not reported in available abstract (enrollment period May 22 – September 3, 2025; 1:1 randomization stated)","population":"Chinese adults aged ≥60 years recruited through community sites, outpatient clinic waiting areas, and WeChat groups; additional inclusion/exclusion criteria not reported in the available abstract","intervention":"Safety UI bundle added to a generative AI chat interface, comprising: (1) generic source-label cues, (2) uncertainty signaling, and (3) a verification nudge displayed within the chat interface","comparator":"Baseline generative AI chat interface without safety UI components (standard chat UI screenshots)","primaryOutcome":"Verification intention (operationalization, scale, and scoring not further detailed in the available abstract)","followUp":"Single cross-sectional encounter (vignette survey); no longitudinal follow-up","fundingSource":"Not reported"}
Key findings
Quantitative results — including point estimates, effect sizes, confidence intervals, p-values, and NNT/NNH — are not reported in the available abstract. The abstract describes the primary outcome as verification intention and lists secondary outcomes as reliance intention, trust calibration, perceived trustworthiness, comprehension, usability/readability, cognitive load, and a behavioral proxy of verification. The study was designed to test whether a composite safety UI bundle increased verification intention relative to a baseline interface; however, whether this hypothesis was confirmed, the magnitude of any difference, and the direction of effect on secondary outcomes (including whether reliance intention decreased, consistent with better trust calibration) cannot be determined from the truncated abstract alone. No adverse events are applicable to this non-interventional survey study. All quantitative findings must be obtained from the full publication.
Practice implications
IMPORTANT LIMITATION: This summary is based solely on a truncated abstract; the full results, statistical analyses, and conclusions are not available for review. What the study was designed to establish: whether a low-cost, interface-level safety nudge (source labeling plus uncertainty and verification cues) can shift older Chinese adults toward more cautious, verification-oriented engagement with generative AI outputs — a clinically meaningful objective given documented AI hallucination risks and the documented vulnerability of older adults to health misinformation. What the study did NOT establish (from the available abstract): whether the primary outcome was met, the magnitude of any effect, durability beyond the experimental session, generalizability to actual AI use behavior, or whether the behavioral proxy of verification correlated with the self-reported intention measure. The population studied (Chinese adults ≥60 years recruited partly from outpatient waiting areas) may partially represent patients who seek health information from AI; however, digital access, AI literacy, and cultural context may limit generalizability to other older adult populations or to non-Chinese AI chat interfaces. The vignette methodology measures stated intention, not observed behavior — a well-recognized gap in health behavior research. Findings apply specifically to interface-level design interventions and do not directly guide clinical counseling on AI use. Practice should not change on the basis of an abstract alone; the full manuscript is required. __LIMITATIONS__ 1. Abstract-only access: primary and secondary outcome results, effect estimates, confidence intervals, and p-values are unavailable, precluding evidence appraisal. 2. Vignette-based design: participants viewed screenshots rather than interacting with a live AI system; ecological validity is limited and intention-behavior gaps are well documented. 3. Cross-sectional, single-encounter measurement: no assessment of whether observed intention changes persist in real-world AI use over time. 4. Behavioral proxy of verification (nature unspecified in abstract): unclear whether this proxy validly reflects actual verification behavior. 5. Recruitment through community sites and WeChat groups may select for more digitally literate, socially engaged older adults, potentially overestimating safety cue effectiveness in less digitally literate populations. 6. Outpatient clinic waiting area recruitment may introduce a clinical health-seeking sample that differs from the general older population in AI use patterns and health information motivation. 7. Potential demand characteristics and social desirability bias inherent to self-reported intention outcomes. 8. The UI bundle tested is a composite (source labels + uncertainty cues + verification nudge); individual component contributions to the effect (if any) cannot be disaggregated. 9. Generalizability is limited to Chinese-language generative AI interfaces and may not extend to AI systems used in other linguistic or cultural contexts. 10. Funding source not reported; potential conflicts of interest cannot be assessed. __BOTTOM_LINE_CATEGORY__ promising_but_preliminary __BOTTOM_LINE__ This randomized vignette experiment evaluated whether a safety UI bundle (source labels, uncertainty cues, and a verification nudge) increases verification intention among Chinese older adults using generative AI chat; the study addresses a clinically important safety question in an understudied, high-risk population. However, quantitative results are unavailable from the truncated abstract, and the vignette methodology measures stated intention rather than real-world verification behavior. Clinicians and health systems interested in AI safety design for older patients should await the full publication before drawing practice-informing conclusions.