Pediatrics · Medical Literature
Usability and Utility of Virtual and In-Person Direct Observation of Resident-Patient Encounters.
Abstract
Direct observation (DO) of clinical encounters is fundamental in medical education, but barriers to in-person observation of encounters (IOE) have been well described. Virtual observation of encounters (VOE) could be an alternative, but related literature is sparse. This study compared VOE with IOE of pediatric residents in terms of usability, utility, and acceptability. We conducted a randomized cross-over interventional study of first-year residents in the hospital medicine setting, with each individual undergoing 1 VOE and 1 IOE. Likert-style surveys based on the Technology Acceptance Model, System Usability Scale, and Mini Clinical Evaluation Exercise were presented to the resident, observer, and family after each observation. The resident and observer were surveyed again after completing both observations. Responses were compared using Wilcoxon signed-rank test. After 40 encounters, no significant differences were found for time, effort, comfort, or perceived performance effects between VOE and IOE. IOE had higher median ratings for adequate observation of interview, examination, rapport, efficiency, and competence. On a 1-to-7 ("strongly prefer in-person" to "strongly prefer virtual") scale, median (IQR) response was 2 (2, 5) for observers and residents (P = .86). Technology issues occurred in 60% of VOEs. Observers without technology issues preferred VOE (median [IQR] 5 [4, 6]); observers with technology issues preferred IOE (median [IQR] 2 [1.5, 2]; P < .01). Families indicated comfort regardless of format. This study identified high usability and utility for IOE and VOE from observer, resident, and family perspectives. IOE scored higher for adequacy of observation and overall preference. VOE may be an acceptable alternative when IOE is less feasible, especially with reliable technology.
Clinical relevance
Does virtual observation of resident-patient encounters (VOE) provide equivalent usability, utility, and acceptability compared to in-person observation (IOE) for assessing first-year pediatric residents in the hospital medicine setting? __SNAPSHOT__ {"studyDesign":"Randomized crossover interventional study","setting":"Pediatric hospital medicine inpatient setting; Norton Children's and University of Louisville School of Medicine, Louisville, KY (primary site), with co-investigators from Johns Hopkins All Children's Hospital, St. Petersburg, FL","sampleSize":"40 encounters (each of the first-year residents completed 1 VOE and 1 IOE; total dyadic N not explicitly stated in the abstract)","population":"First-year pediatric residents (PGY-1) in the hospital medicine setting; patient families also surveyed","intervention":"Virtual observation of clinical encounters (VOE) — remote video-based observation of resident-patient encounters","comparator":"In-person observation of encounters (IOE) — standard direct bedside observation","primaryOutcome":"Usability, utility, and acceptability of VOE vs. IOE, assessed via Likert-style surveys based on the Technology Acceptance Model (TAM), System Usability Scale (SUS), and Mini Clinical Evaluation Exercise (Mini-CEX), completed by residents, observers, and families","followUp":"Single paired crossover (each participant completed both observation types); longitudinal follow-up duration not reported","fundingSource":"Not reported"}
Key findings
Across 40 observed encounters, no statistically significant differences were detected between VOE and IOE for time required, effort expended, comfort of the resident, or perceived effects on resident performance. However, IOE demonstrated higher median Likert ratings than VOE in several clinically meaningful domains: adequacy of observation of the clinical interview, physical examination, rapport with the patient/family, efficiency of the encounter, and overall competence assessment. A 1-to-7 global preference scale was administered after completion of both observation types; the abstract was truncated before the preference results were fully reported. Statistical comparisons were performed using the Wilcoxon signed-rank test, an appropriate nonparametric method for paired ordinal data. Specific median values, effect sizes, confidence intervals, p-values for individual comparisons, and NNT/NNH are not reported in the available abstract.
Practice implications
This study provides preliminary evidence that VOE is broadly feasible and does not significantly increase resident burden (time, effort, comfort) relative to IOE in a pediatric hospital medicine setting. However, the finding that IOE yielded higher observer-rated adequacy across several key clinical domains — including physical examination observation, rapport assessment, and overall competence — suggests that VOE may not be a fully equivalent substitute for direct bedside observation, particularly for competencies that require visual proximity and tactile assessment context. The study population was limited to PGY-1 residents at academic pediatric hospital medicine settings; findings may not generalize to other training levels, specialties, outpatient environments, or community hospitals. Family perspectives were included, which adds an important stakeholder dimension rarely captured in medical education research. Because only the abstract is available, the magnitude of differences on individual Likert items, subgroup analyses, and the global preference findings cannot be fully evaluated. The abstract was truncated before the preference scale results were presented, leaving a critical outcome unreported in the source material provided. Practice implications remain preliminary pending review of the full manuscript. __LIMITATIONS__ Small sample size (40 encounters total), limiting statistical power to detect modest but clinically meaningful differences between modalities.,Crossover design with a single observation per modality per resident introduces risk of order effects, carry-over bias, and insufficient sampling of clinical variability.,Survey-based outcomes (TAM, SUS, Mini-CEX Likert items) are subjective and may be subject to social desirability bias, particularly given that observers and residents knew the study purpose.,Blinding was not possible given the nature of the intervention; observer and resident awareness of modality could influence ratings.,Restriction to PGY-1 residents in one clinical context (hospital medicine) limits generalizability across training levels, specialties, and settings.,The abstract is truncated and does not report the global preference scale results or full statistical outputs, preventing complete appraisal of the primary findings.,Specific p-values, medians, and effect sizes for most outcomes are not reported in the abstract, precluding quantitative synthesis.,Funding source is not disclosed; potential conflicts of interest cannot be assessed.,The study does not assess downstream educational outcomes (e.g., resident skill acquisition, competency milestone attainment) — all endpoints are perceptual/process measures. __BOTTOM_LINE_CATEGORY__ promising_but_preliminary __BOTTOM_LINE__ VOE appears logistically feasible and comparably burdensome to IOE for PGY-1 pediatric residents in a hospital medicine setting, but observers rated IOE higher for observing key clinical competencies including physical examination, rapport, and overall performance. VOE may serve as a practical supplement to IOE when in-person observation is not possible, but should not yet be considered a fully equivalent replacement — particularly for physical examination-dependent competency assessment. Full manuscript review is necessary before incorporating these findings into residency program policy.