#KeyLIMEPodcast 355: Can we see into the brain of a trainee by looking at their notes?

Reference

Schaye et. al., Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes:a Shared Mental Model for Feedback J Gen Intern Med. 2022 Feb;37(3):507-512.

Reviewer

Linda Snell (@LindaSMedEd)_

Background

I am an internal medicine specialist – clinical reasoning is my bread and butter: doing it, teaching it, and yes, assessing it. I  usually do this in discussion with learners or by looking at the consult note, but do not have a useful concept map or model to help me. Written notes are a way of assessing a trainee’s clinical reasoning, and this has made its way into competency frameworks e.g.  an ACGME Internal Medicine milestone “appropriate utilization and completion of medical records including effective communication of clinical reasoning.” The authors say that feedback on notes is infrequent, with one barrier bein a lack of a shared mental model for high-quality clinical reasoning, and variability of assessment tools that exist. The validated instruments for rating notes have varying degrees of evaluation of reasoning.

The IDEA tool does have “a robust assessment of clinical reasoning documentation” but lacks descriptive anchors for each domain which might threaten reliability. The IDEA domains are: (I)  Interpretive summary,  (D) Differential diagnosis with commitment to the most likely diagnosis, (E) Explanation of reasoning in choosing the most likely diagnosis, (A) Alternative diagnosis with explanation of reasoning.

Purpose

\”…to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool… the process of developing and validating the Revised-IDEA assessment tool with standard setting for item rating in order to … increase reliability and create a shared mental model for feedback on clinical reasoning documentation.”

Key Points on the Methods

Methods well described and mostly understandable:
Used 4 of items in Messick’s validity framework to derive evidence: content validation (do items reflect the construct they are intended to measure), response process (how well the answer reflects the observed performance), internal structure (associations between assessment results and other measures or learner), and consequences (the impact, beneficial or harmful, of the assessment itself and the decisions and actions that result.)

content validation – panel of experts
response process – piloted, feedback, revisions for tool
internal structure – inter-rater reliability on 282 notes
consequences – cut-off for what makes good documentation

There are many methods for determining pass/fail. “Methods for standard setting are categorized as relative methods or absolute methods. Relative methods compare the students’ performance to each other, while absolute methods refer to an external reference point. Absolute methods are preferable.” Hofstee is a compromise method, between absolute and relative, that prevents failing too many by polling an expert panel on the maximum and minimum acceptable failure rates, which is incorporated into the determination of the cut score.

In summary they took the old IDEA, validated content to have a newer IDEA, piloted and revised this to have the newest IDEA for which they developed reliability and cut off for what constituted high- vs. low- quality documentation of clinical reasoning.

Key Outcomes

content validation – The newest ‘Revised IDEA’ – same four core domains of the original with  more detailed descriptions in the prompts and descriptive anchors for the Likert rating scale in each of the four domains – giving total maximum score of 10.