Last week, the KeyLIME podcast looked at a tool created to evaluate peer surgical coaching; this week, they look at a tool created to evaluate internal medicine residents’ leadership skills when leading rounds. The skills needed to lead rounds have a great effect on education and care, as they effect both learner experiences and patient outcomes. Despite this, little is known what exactly these skills are, let alone how to teach and assess them. The authors have this paper have created a direct observation instrument for this purpose and put in to the test in this study.
Listen to the episode and learn more here.
KeyLIME Session 265
Reference
Ricotta et. al., The Mindful Manager: Validation of a Rounding Leadership Instrument for Residents J Gen Intern Med. 2020 Apr;35(4):1161-1166
Reviewer
Linda Snell (@LindaSMedEd)
Background
What happens on clinical rounds? What are the skills needed to lead rounds?
Rounds: ‘the process of seeing, assessing, and caring for patients as a team’. They involve both patient care and education, integrally mixed, and . Often led by senior resident (PGY2-4), the skills needed to lead these rounds contribute to education and care; variability in practices affects both learner experiences and patient outcomes . Yet within this ‘leading rounds’ competency, little is known about what skills, how to teach and how to assess these abilities … and there is an absence of validated and acceptable tools to measure this competency. Generic leadership tools cannot address heterogeneity and opportunistic nature of rounds, and best rounding practices have not been ‘codified’.
A digression into Kane: Any assessment tool must be validated.
Purpose
“To develop and collect validity evidence for a direct observation instrument of internal medicine residents’ leadership skills during daily in patient care rounds for future formative assessment.”
Key Points on the Methods
Quaternary teaching hospital, internal medicine teaching units, teams of a PGY2 or 3, two PGY1s, med students and an attending staff, with senior residents (PGY2-3) being the subject.
Steps: Lit review and tool development; consensus, pilot testing & revisions; rater training and norming; supervisors observe and rate residents on a morning rounds using tool; paired supervisor and RA observations / ratings; faculty survey perceptions of tool
Development of observation tool:
1. Lit review -> team leadership skills relevant to managing inpatient teams, categorized into domains and aligned with ACGME competencies
2. Consensus building (3 rounds with experts – not a Delphi) for domains below
- creating a safe learning environment;
- providing consistent and effective feedback;
- role-modeling clinical skills, attitudes, and behaviors;
- running the team efficiently;
- developing situational awareness;
3. described behaviours for each domain and obtained consensus with experts (2 rounds)
4. Piloted and revised
Validation used Kane’s framework
scoring, are observations accurate, reproducible and quantifiable: done through lit review, consensus building for instrument development and norming during faculty development
generalization, measures reliability and how the sample items reflect overall performance: done through consensus building choice of behaviors, and IRR of paired faculty supervisors’ and RAs’ observations
extrapolation, correlating testing and real-world performance: This was already done in real world; so compared PGY2s and 3s scores thinking 3s would be better; faculty survey question did behaviors reflect actual skills needed
implication, interpretation & action about results
Key Outcomes
scoring, Total 223 observations – 140 for 63 PGY2s, 83 for 29 PGY3s (sample size of available rounds/rounders)
generalization, IRR of paired observations
Instrument item Inter-rater reliability (kappa)
*Q1: Comes to rounds already with a sense of what clinical plan should be for each patient 0.44
Creates a safe learning environment
*Q2: Sets expectations—at the start of rotation or on a daily basis 0.39
Q3: Asks questions in a non-threatening way 1.0
Q4: Provides support and encouragement 0.84
Q5: Incorporates all members of the healthcare team 1.0
Q6: Identifies learners needs and knowledge gaps 0.77
Maintains situational awareness
Q7: Sets agenda—verbalizes a rounding order 0.91
Q8: Sets agenda–identifies strategies for rounding 0.87
Q9: Sets agenda—risk stratifies patients for rounding (dire, dilemma, discharge, delineated) 0.94
Q10: Acknowledges needs of the team 0.94
Q11: Limits interruptions 0.94
Role models
*Q12: Explicitly verbalizes own specific behaviors or actions 1.0
Q13: Demonstrates effective bedside manner 0.47
Q14: Avoids pejorative comments in front of learners 0.77
Efficiency
Q15: Starts rounds on time 1.0
Q16: Ends rounds at a specified time point 0.89
Q17: Redirects learners to avoid tangential or circumferential discussion 0.77
Q18: Summarizes plan at the end of clinical encounters 0.94
Q19: Explicitly assigns tasks, delegates responsibilities 1.0
Teaching
Q20: Uses open ended “why” and “how” questions as opposed to “what” questions 0.89
Q21: Uses body language effectively to promote learner engagement 0.72
Q22: Teaching is incorporated into clinical care 1.0
Feedback
Q23: Provides immediate feedback during or immediately after rounds 0.83
Q24: Feedback is specific and actionable 0.92
Q25: Debriefs after clinical encounters 0.94
*Q26: Checks in with team daily 0.18
extrapolation, no difference in PGY2&3 performance (p=0.22)
Faculty survey positive: instrument elements represented best practices of managing an inpatient medical team, reflected residents’ global leadership skills, applied their interpretations of the residents’ behaviors to coaching.
implication, downstream consequences, used for formative assessment only
Overall results: “we identified the relevant constructs of clinical leadership for daily rounds, based on a review of the literature, synthesized through a consensus-driven process, and refined through iterations of pilot testing. We trained hospitalists and research assistants as standardized raters and demonstrated strong inter-rater reliability. We found evidence in all four stages of Kane’s validity inference and for use of the instrument in clinical practice. Faculty raters felt the instrument reflected the overall leadership skills of the individual, was relevant to clinical practice, was easy to use, and took minimal time to complete. Moreover, nearly all faculty used the instrument for feedback and coaching following observations”
Items with poor IRR have likely explanations (RAs not preset to observe, RAs not health prof)
Key Conclusions
The authors conclude… “Rounding leadership is foundational to medical training, is challenging to measure, and can directly influence the clinical learning environment.”
They codified best practices, used tool for feedback and suggested other uses (faculty development, resident -as-leader curricula, use on other services)
Future: add a global rating scale, correlate with patient outcomes.
Spare Keys – other take home points for clinician educators
Content: Good to identify relevant elements of abilities for daily clinical rounds … are they all ‘leadership’?
Validation of a new tool is essential, and there are frameworks to do this.
When is a Delphi useful?
Access KeyLIME podcast archives here
The views and opinions expressed in this post and podcast episode are those of the host(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our ‘About’ page