8. Learner Assessment
8aAssess the effectiveness and efficiency of instructional strategies and student learning achievements through the evaluation of program and/or curricula outcomes.
8bCreate engaging learning solutions to improve health professions education.
Artifact 8a-A
Psychometric Analysis
Reflection
The purpose of this assignment in HPTT 805 – Evaluation and Assessment of Teaching and Learning in Health Professions Education was to conduct an item analysis of multiple-choice examination data, together with written answers on test reliability and item security. I selected this artifact to represent competency 8a because it is the only place in the portfolio where I work with assessment data rather than describe an approach to assessment.
Each of the three items analyzed required a different remedy. On the first, two distractors that nobody selected were why the difficulty index sat below goal, so revising them raises difficulty on what I interpreted as a competency item. On the second, difficulty was too high for a discrimination item and I proposed changes to the question stem to adapt. The third had acceptable difficulty but a discrimination index below 0.20 and a Q-index under 0.65, which again point towards the stem as a potential issue. In interpreting a falling Kuder-Richardson 20 across repeated administrations, I kept three explanations open: a change in teaching quality, a change in the cohort’s foundational preparation, or a compromised examination.
One of the challenges encountered was that the same index means different things depending on what an item was written to do. A difficulty near 0.9 sits just below goal on a competency item and far too high on a discrimination item; however, nothing in the report identifies the question’s intent. Going forward, I intend to run this analysis on assessments I am responsible for, and to adapt the questions that I write for my medical school lectures based on the provided psychometric data.
Download the psychometric analysis (PDF) (opens in a new tab)
Artifact 8a-B
Test Blueprint
Reflection
The purpose of this assignment in HPTT 805 – Evaluation and Assessment of Teaching and Learning in Health Professions Education was to develop a blueprint for a summative examination given to medical students at the end of a Hepatology lecture series, specifying content areas, learning objectives, taxonomic levels, weighting and item types. I selected this artifact to represent competency 8a because building the blueprint required me to evaluate what the lecture series covered, and at what level, before assessing any student against it.
The blueprint comprises three content areas, each with its own objectives: approaching patients with chronic liver disease, liver anatomy and physiology, and elevated liver enzymes. These carry five, seven and eight of the twenty multiple-choice items respectively. Across them, three levels of Bloom’s taxonomy are weighted at 35% knowledge, 30% comprehension and 35% application. Liver anatomy and physiology draws the largest share of knowledge items, while elevated liver enzymes is entirely comprehension and application, which matches what a student would be asked to do with such a patient in clinic.
One of the challenges encountered was deciding how to weight the three content areas against one another. Liver anatomy and physiology takes some of the most lecture time and carries the largest share of knowledge items, as these are the foundational topics that the clinical content builds on. That being said, hepatology clinical rotations are not mandatory in our curriculum, so I felt it was more important to secure those foundations than to weight the examination towards the clinical areas. In an ideal world hepatology would be a core rotation for medical students, and if that changed I would adapt the blueprint accordingly.
Artifact 8b-A
SBAR Course Project: Recommendation
Reflection
The SBAR Course Project in HPTT 805 – Evaluation and Assessment of Teaching and Learning in Health Professions Education was submitted as four separate assignments, each carrying the earlier sections forward, so the Recommendation document contains all four. It addresses how endoscopic competency is judged within our gastroenterology fellowship and proposes a new means of assessment. I selected the Recommendation section to represent competency 8b because it is the part that designs a solution rather than describing a problem, and one I am able to implement within my own program.
The Situation and Background sections set out the status quo. American Board of Internal Medicine milestones are vague on endoscopic competency, the 2007 national core curriculum recommended 130 upper endoscopies and 140 colonoscopies before competence was assessed, and the 2014 update removed those thresholds without replacing them. The Assessment section compared three validated instruments against criteria I set in advance: generalizability, time to complete, examiner training, cost and repeatability. I recommended the Skill Assessment in Fellow Endoscopy Training (SAFE-T) tool because it covers both upper endoscopy and colonoscopy, is five questions long, and needs minimal examiner training.
I proposed that five endoscopies be reviewed every six months by at least two supervising attendings, on paper rather than online, and timed to the existing clinical competency committee meeting so that no new meeting is created. One of the challenges encountered was that none of the three tools was a complete fit. The barriers are also not financial, as the tool is free to use but assessor time and attending buy-in are not. An assessment is only useful if something follows from it, so a fellow scoring below expectation would have their schedule adjusted to allow more endoscopy time. I have already had preliminary conversations with fellowship leadership about implementation.
Artifact 8b-B
Clinical Reasoning Analysis
Reflection
The purpose of this assignment in HPTT 840 – Foundations of Clinical Teaching was to identify where clinical reasoning already sits in the simulation activity I had proposed earlier in the course, and to redesign one facet where it could be improved. I selected this artifact to represent competency 8b because the second half of the assignment required me to refine my own design to strengthen clinical reasoning, and to give the rationale behind the change.
Reasoning is already present at the third step of Peyton’s four-step approach in the originally devised paracentesis simulation. Here the third-year medical student does not perform the procedure but verbally directs the faculty instructor through it, covering safe landmarking, sterile technique and the Z-track approach. Directing the faculty member’s movements forces retrieval, anticipation of the consequences of each instruction, and metacognitive checking against the procedural checklist. The weakness is in the third learning objective, planned as three written cases for calculating and interpreting the serum-ascites albumin gradient. Providing students with the data removes both hypothesis generation and the patient narrative, leaving superficial pattern matching rather than reasoning.
The redesign moves the activity into a sandboxed electronic medical record, similar to one already used for second-year medical student small group sessions. Students would read an admission note, build their own differential diagnosis, order the serum and ascitic studies they want, and present an assessment and plan. One of the challenges encountered was recognizing that my original activity looked like clinical reasoning without actually requiring it. Going forward, I intend to design my other teaching activities the same way, by evaluating whether students have to gather the information themselves or are simply provided it.
Download the clinical reasoning analysis (PDF) (opens in a new tab)