Research and Efficacy

Visually diagnosed medical tests (e.g. radiographs, electrocardiograms) are the most commonly ordered tests in front-line medicine. As such, front-line health care professionals are faced with the task of learning the skill of interpreting these images to an expert performance level by the time they provide opinions that guide patient management decisions. However, discordant interpretations of these images between front-line physicians and expert counterparts (radiologists, cardiologists) is a common cause of medical error. In paediatrics, this problem is even greater due to the changing physiology with age leading to increased risk of interpretation errors.

Currently, most approaches to learning the interpretation of medical images include case-by-case exposure in clinical settings and tutorials that are either didactic or the passive presentation of cases on-line. However, these strategies have not demonstrated optimal effectiveness in clinical studies that examined the accuracy of front-line physicians in interpreting these images. Furthermore, many of the continuing medical education activities emphasize clinical knowledge, do not provide opportunities for feedback, and require little more than documentation of attendance, which limits the potential for improvement in the practicing physician.

ImageSim provides a comprehensive and evidence-based on-line education system that teaches health care professionals the interpretation of visually diagnosed medical tests using the concepts of deliberate practice and simulation. That is, our learning model includes sustained active practice of hundreds of cases where the learner is forced to commit to diagnosis for every case and then receives immediate specific feedback on their interpretation so that the participant instantly learns from each case. Importantly, we have presented these images as we encounter them in practice, and included a normal to abnormal radiograph ratio (with a spectrum of pathology) reflective of our day-to-day practice.

Our research to date shows that this learning method works – and all physicians at varying levels of expertise had significantly increased their accuracy in image interpretation along a “learning curve” as graphed below:

Fig1 - Prototypical Learning Curve

We started this system for paediatric musculoskeletal images, and our participants enjoyed it and asked us for more. We just launched our pediatric chest radiograph course, and we are close to finalizing similar learning systems for point-of-care ultrasound, the pre-pubertal vaginal examination, and pediatric electrocardiograms.

 

Selected Publications

  1. Pecaric M, Boutis K, Beckstead J, Pusic M. A Big Data Learning Analytics Approach to Process-Level Feedback in Cognitive Simulations. Acad Med. 2016.  May 24
  2. Pusic M, Boutis K, Pecaric M, Savenkov O, Beckstead JW, Jabour M.  A primer on the statistical modelling of learning curves in health professions education. Adv Health Sci Educ Theory Pract. 2016 Oct 3.
  3. Beckstead J, Boutis K, Pecaric M, Pusic M. Sequential Dependencies in Categorical Judgments of Radiographic Images. AHSE. Jun 8. 2016.
  4. Boutis K, Cano S, Pecaric M, et al. Interpretation Difficulty of Normal versus Abnormal Radiographs Using a Paediatric Example. CMEJ. 2016 Mar 31;7(1):e68-77.
  5. Pusic M, Chiaramonte R, Pecaric M, Boutis K. A Simple Method for Improving Self-Assessment While Learning Radiograph Interpretation. Med Educ. 2015;49(8):838-46.
  6. Pusic M, Boutis K, Hatala R, Cook D. Learning curves: A Primer for Health Professions Education. Acad Med. 2014;90(8):1034-1042.
  7. Beckstead J, Boutis K, Pecaric M, Pusic M. The Influences of Sequential Effects in Randomized Online Learning Tasks. Applied Cognition. 2014; 27(5):625-632
  8. Boutis K, Pecaric M, Ridley J, Andrews J, Gladding S, Pusic M. Hinting Strategies for Improving the Efficiency of Medical Student Learning of Deliberately Practiced Web-based Radiographs. Med Educ. 2013;47(9):877-887.
  9. Pusic MV, Andrews JS, Kessler DO, Boutis K. Determining the optimal case mix of abnormals to normals for learning radiograph interpretation: a randomized control trial. Medical Education. 2012;46(3):289-298.
  10. Pusic MV, Kessler D, Szyld D, Kalet A, Pecaric M, Boutis K. Experience Curves as an Organizing Framework for Deliberate Practice in Emergency Medicine Learning. Acad Emerg Med. 2012;19(12):1476-80.
  11. Pusic M, Pecaric M, Boutis K. How Much Practice is Enough? Using Learning Curves to Assess the Deliberate Practice of Radiograph Interpretation. Acad Med. 2011;86(6):731-736.
  12. Boutis K, Pecaric M, Pusic M. Using Signal Detection Theory to Model Changes In Serial Learning of Radiological Image Interpretation. Adv Health Sci Educ Theory Pract. 2010;15:647-658.
  13. Boutis K, Pecaric M, Pusic M. Selecting Radiographs For Teaching The Interpretation of X-Rays Based On Ratings by the Target Population. Med Educ. 2009;43(5):434-441.

 

Selected Grants

  1. Performance-Based Competency in the Interpretation of Pediatric Musculoskeletal Images. Royal College of Physicians and Surgeons. 2016
  2. Bridging the G.A.P.: A Deliberate Practice Method for Learning Genital Abnormalities in Prepubescent Girls. CAME Wooster Family Education Grant. 2016
  3. ImageSim – Innovating Medical Education of Medical Images. Innovation Fund, Hospital for Sick Children. 2016
  4. Establishing Learning Curves for Distinguishing Abusive from Non-abusive Burns and Bruises in Children. Colorado Department of Public Health and Environment, EMS and Trauma Program, Systems Improvement Category. 2015
  5. Learning Retention and the Timing of Refresher Education After Deliberate Practice of Radiograph Interpretation.  Royal College of Physicians and Surgeons. 2014.
  6. The Deliberate Practice of Paediatric ECGs. Department of Paediatrics, Hospital For Sick Children. 2014.
  7. Learning the Interpretation of Point of Care Emergency Ultrasound Images. Academic Paediatric Association. 2014.
  8. Hinting Strategies for Improving the Efficiency of Medical Student Learning of Deliberately Practiced Web-based Radiographs. Educational Dean’s Fund, University of Toronto. 2011.
  9. Case Mix Study. Royal College of Physicians and Surgeons, Medical Education Research Grant. 2009.
  10. Climbing the Learning Curve: a new approach to teaching non-radiologists x-ray interpretation of ankle x-rays. Royal College of Physicians and Surgeons Medical Education Research Grant. 2005.

Selected Presentations

  1. Learning How To Interpret Visually Diagnosed Tests Using Simulation and Deliberate Practice. Great Ormond Street. Pediatric Grand Rounds: London, UK, 2017.
  2. Climbing the Learning Curve to Competency of Visually Diagnosed Tests Using Simulation and Deliberate Practice. Alder Hey Children’s Hospital. Pediatric Grand Rounds: Liverpool, UK, 2017.
  3. The Deliberate Practice of Medical Images – Getting You To Competency. Cruces University Hospital. Pediatric Grand Rounds: Northwick Park, Harrow, UK, 2017.
  4. Climbing the Learning Curve to Competency of Visually Diagnosed Tests Using Simulation and Deliberate Practice. Addenbrook’s Hospital. Pediatric Grand Rounds: Cambridge, UK, 2017.
  5. Pediatric Emergency Medicine Medical Education Rounds. Boston Children’s Hospital, Boston, USA, 2017.
  6. Medical Education Grand Rounds. Children’s Hospital of Philadelphia. Philadelphia, USA, 2017.
  7. The Deliberate Practice of Medical Images – Getting You To Competency. Cruces University Hospital. Pediatric Grand Rounds: Bilbao, Spain, 2016.
  8.  Medical student self-assessment deteriorates over and above skill decay on a retention test of radiograph interpretation skills. Council on Medical Student Education in Pediatrics – Ottawa, ON, 2014
  9. Using Process Data from a Radiology Interpretation Simulator To Augment Feedback to Medical Learners International Conference for Residency Education – Calgary, AB; 2013; People for Education – Prague, Czech Republic; 2013; Pediatric Academic Societies – Washington DC; 2013; International Pediatric Simulation Society – New York, NY 2013
  10. Learning Analytics in Screen Based Simulation of Radiograph Interpretation International Conference on Residency Education: Simulation – Calgary AB, 2013
  11. Physicians’ Judgments of Radiological Images on a Multi-trial Discrimination Task: Evidence for the Use of Cognitive Heuristics. Society for Medical Decision Making – Pheonix AZ, 2012.
  12. Determining the Optimal Mix of Abnormals to Normals for Learning Radiograph Interpretation: A Randomized Controlled Trial of Residents. The Ottawa Conference – Miami FL, 2010
  13. Using Signal Detection Theory to Model Changes in Longitudinal Learning of Pediatric Ankle X-ray Interpretation. Pediatric Academic Societies – Baltimore MS, 2009