Research and Efficacy

Visually diagnosed medical tests (e.g. radiographs) are the most commonly ordered tests in front-line medicine. As such, front-line health care professionals are faced with the task of learning the skill of interpreting these images to an expert performance level by the time they provide opinions that guide patient management decisions. However, discordant interpretations of these images between front-line physicians and expert counterparts (e.g. radiologists) is a common cause of medical error. In paediatrics, this problem is even greater due to the changing physiology with age leading to increased risk of interpretation errors.

ImageSim provides a comprehensive and evidence-based on-line education system that teaches health care professionals the interpretation of visually diagnosed medical tests using the concepts of deliberate practice and simulation. That is, our learning model includes sustained active practice of hundreds of cases where the learner is forced to commit to diagnosis for every case and then receives immediate specific feedback on their interpretation so that the participant instantly learns from each case. Importantly, we have presented these images as we encounter them in practice, and included a normal to abnormal radiograph ratio (with a spectrum of pathology) reflective of our day-to-day practice.

Our research to date shows that this learning method works – and all physicians at varying levels of expertise had significantly increased their accuracy in image interpretation along a “learning curve” as graphed below:

Fig1 - Prototypical Learning Curve

Selected Publications

  1. Lee MS, Pusic M, Pecaric M, Camp M, Stimec J, Carrier B, Dixon A, Herman J, Boutis K. A Target Population Derived Method for Developing a Competency Standard in Radiograph Interpretation. Teaching and Learning in Medicine. 2021 May 17;1-11.
  2. Thau E, Perez M, Pusic M, Pecaric M, Rizutti D, Boutis K. Image Interpretation: Learning Analytics Informed Education Opportunities. AEM Educ and Training. 2021 Apr 1;5(2):e10592.
  3. Kwan C, Weerdenburg K, Pusic M, Chen A, Rempell R, Hermann J, Boutis K. Learning Pediatric Point-of-Care Ultrasound: How Many Cases Does Mastery of Image Interpretation Take? Pediatric Emergency Care. In press.
  4. Campos S, Pusic MV, Smith T, Legano L, Brown J, Pecaric M, Boutis K. Prepubertal Female Genital Examination: Evidence Informed Learning Opportunities. Journal of Pediatric and Adolescent Gynecology. 2021 Apr;34(2):117-123.
  5. Adelgais K, Pusic MV, Abdoo D, Caffrey S, Snyder K, Boutis K.  Child Abuse Recognition Training using Deliberate Practice for Prehospital Providers. Prehospital Care. Epub. November 11, 2020.
  6. Boutis K, Pecaric M, Stimec J, Carrier B, Chan J, Pusic M. Radiograph Interpretation and Skill Retention: A Multicentre Randomized Control Trial. Med Teach. 2019 May 2:1-9.
  7.  Lee M, Pusic M, Carriere B, Dixon A, Pecaric M, Stimec J, Boutis K. Building Competency in Pediatric Musculoskeletal Radiograph Interpretation: A Multicentre Prospective Cohort Study. Acad Med – Education and Training. 2019 Mar 12;3(3):269-279.
  8. Kwan C, Pusic M, Pecaric M, Weerdenburg K, Tesssaro M, Boutis K. The Variable Journey in Learning to Interpret Pediatric Point-of-care Ultrasound Images: A Multicenter Prospective Cohort Study. AEM Educ Train. 2019 Jul 30;4(2):111-122.
  9. Yoon P, Boutis K, Pecaric M, Fefferman NR,  Ericsson A, Pusic M. A Think-Aloud Study to Inform the Design of Radiograph Interpretation Practice. AHSE. 2020. 25(4): 877-903.
  10. Pusic M, Boutis K, McGaghie W. Role of Scientific Theory in Simulation Education Research. Simulation in Healthcare. 2018 Jan 24 Epub ahead of print.
  11. Pusic M, Boutis K, Pecaric M, Savenkov O, Beckstead JW, Jabour M. A primer on the statistical modelling of learning curves in health professions education. Adv Health Sci Educ Theory Pract. 2017 Aug;22(3):741-759
  12. Pecaric M, Boutis K, Beckstead J, Pusic M. A Big Data Learning Analytics Approach to Process-Level Feedback in Cognitive Simulations. Acad Med. 2017 Feb;92(2):175-184.
  13. Beckstead J, Boutis K, Pecaric M, Pusic M. Sequential Dependencies in Categorical Judgments of Radiographic Images. Adv Health Sci Educ Theory Pract. 2017 Mar;22(1):197-207
  14. Pusic M, Chiaramonte R, Pecaric M, Boutis K. Accuracy of self-monitoring during learning of radiograph interpretation. Med Educ. 2015 Aug;49(8):838-46.
  15. Pusic M, Boutis K, Hatala R, Cook D. Learning curves: A Primer for Health Professions Education. Acad Med. 2014. Aug;90(8):1034-42
  16. Beckstead J, Boutis K, Pecaric M, Pusic M. Sequential Dependencies in Categorical Judgments of Radiographic Images. AHSE. Jun 8. 2016.
  17. Boutis K, Cano S, Pecaric M, et al. Interpretation Difficulty of Normal versus Abnormal Radiographs Using a Paediatric Example. CMEJ. 2016 Mar 31;7(1):e68-77.
  18. Pusic M, Chiaramonte R, Pecaric M, Boutis K. A Simple Method for Improving Self-Assessment While Learning Radiograph Interpretation. Med Educ. 2015;49(8):838-46.
  19. Pusic M, Boutis K, Hatala R, Cook D. Learning curves: A Primer for Health Professions Education. Acad Med. 2014;90(8):1034-1042.
  20. Beckstead J, Boutis K, Pecaric M, Pusic M. The Influences of Sequential Effects in Randomized Online Learning Tasks. Applied Cognition. 2014; 27(5):625-632
  21. Boutis K, Pecaric M, Ridley J, Andrews J, Gladding S, Pusic M. Hinting Strategies for Improving the Efficiency of Medical Student Learning of Deliberately Practiced Web-based Radiographs. Med Educ. 2013;47(9):877-887.
  22. Pusic MV, Andrews JS, Kessler DO, Boutis K. Determining the optimal case mix of abnormals to normals for learning radiograph interpretation: a randomized control trial. Medical Education. 2012;46(3):289-298.
  23. Pusic MV, Kessler D, Szyld D, Kalet A, Pecaric M, Boutis K. Experience Curves as an Organizing Framework for Deliberate Practice in Emergency Medicine Learning. Acad Emerg Med. 2012;19(12):1476-80.
  24. Pusic M, Pecaric M, Boutis K. How Much Practice is Enough? Using Learning Curves to Assess the Deliberate Practice of Radiograph Interpretation. Acad Med. 2011;86(6):731-736.
  25. Boutis K, Pecaric M, Pusic M. Using Signal Detection Theory to Model Changes In Serial Learning of Radiological Image Interpretation. Adv Health Sci Educ Theory Pract. 2010;15:647-658.
  26. Boutis K, Pecaric M, Pusic M. Selecting Radiographs For Teaching The Interpretation of X-Rays Based On Ratings by the Target Population. Med Educ. 2009;43(5):434-441.

Selected Presentations

  1. The Deliberate Practice Journey to Visual Diagnosis Expertise.  Hôpital de La Tour. August 2021.
  2. Building Competency in Medical Image Interpretation. Astrid Lindgren’s Children’s Hospital. May 2021.
  3. Online Learning of Medical Image Interpretation to a Performance Standard. International Association for Medical Education. December 2020.
  4. Using ImageSim to Teach Field Physicians in Resources Limited Settings. Global Health Forum. November 2020.
  5. ImageSim – An Online Image Interpretation Learning System. Canadian Association of Emergency Medicine, May 2019.
  6. Teaching the Pediatric Emergency Physician How to Interpret Pediatric Point of Care Ultrasound Images. Pediatric Academic Societies and Canadian Association of Emergency Medicine, May 2018.
  7. Competency Based Interpretation of Pediatric Musculoskeletal Images. Pediatric Academic Societies and Canadian Association of Emergency Medicine, May 2018.
  8. Radiograph Interpretation and Skill Retention: A Multicentre Randomized Control Trial. Pediatric Academic Societies (May 2018), Canadian Association of Emergency Physicians (May 2018), Sim ONE (December 2018), International Conference of Residency Education (October 2018).
  9. ImageSim – on-line education innovation. SIM One. December 2017.
  10. Learning How To Interpret Visually Diagnosed Tests Using Simulation and Deliberate Practice. Great Ormond Street. Pediatric Grand Rounds: London, UK, 2017.
  11. Climbing the Learning Curve to Competency of Visually Diagnosed Tests Using Simulation and Deliberate Practice. Alder Hey Children’s Hospital. Pediatric Grand Rounds: Liverpool, UK, 2017.
  12. The Deliberate Practice of Medical Images – Getting You To Competency. Cruces University Hospital. Pediatric Grand Rounds: Northwick Park, Harrow, UK, 2017.
  13. Climbing the Learning Curve to Competency of Visually Diagnosed Tests Using Simulation and Deliberate Practice. Addenbrook’s Hospital. Pediatric Grand Rounds: Cambridge, UK, 2017.
  14. Pediatric Emergency Medicine Medical Education Rounds. Boston Children’s Hospital, Boston, USA, 2017.
  15. Medical Education Grand Rounds. Children’s Hospital of Philadelphia. Philadelphia, USA, 2017.
  16. The Deliberate Practice of Medical Images – Getting You To Competency. Cruces University Hospital. Pediatric Grand Rounds: Bilbao, Spain, 2016.
  17.  Medical student self-assessment deteriorates over and above skill decay on a retention test of radiograph interpretation skills. Council on Medical Student Education in Pediatrics – Ottawa, ON, 2014
  18. Using Process Data from a Radiology Interpretation Simulator To Augment Feedback to Medical Learners International Conference for Residency Education – Calgary, AB; 2013; People for Education – Prague, Czech Republic; 2013; Pediatric Academic Societies – Washington DC; 2013; International Pediatric Simulation Society – New York, NY 2013
  19. Learning Analytics in Screen Based Simulation of Radiograph Interpretation International Conference on Residency Education: Simulation – Calgary AB, 2013
  20. Physicians’ Judgments of Radiological Images on a Multi-trial Discrimination Task: Evidence for the Use of Cognitive Heuristics. Society for Medical Decision Making – Pheonix AZ, 2012.
  21. Determining the Optimal Mix of Abnormals to Normals for Learning Radiograph Interpretation: A Randomized Controlled Trial of Residents. The Ottawa Conference – Miami FL, 2010
  22. Using Signal Detection Theory to Model Changes in Longitudinal Learning of Pediatric Ankle X-ray Interpretation. Pediatric Academic Societies – Baltimore MS, 2009

Selected Grants

  1. Deliberate Practice as an Educational Method for Differentiating Burn and Bruise Injuries in Children. Physician Services Incorporated. 2020.
  2. Climbing the Learning Curve for Pediatric Ocular Point of Care Ultrasound. SAEM. 2020.
  3. Climbing the Learning Curve for First Trimester Pregnancy Bedside Transvaginal Ultrasound Image Interpretation. EMF. 2020.
  4. Climbing the Learning Curve to Pediatric Dermatology Image Interpretation. Canadian Dermatology Foundation. 2020.
  5. Cognitive Simulation and Deliberate Practice to Improve Optical Diagnosis and Management Decision-making Skills for Colorectal Polyps. American Society of Gastrointestinal Endoscopy. 2020.
  6. Chest Radiograph: When, Where, and Why We Make Mistakes. Boutis K, Thau E, Pusic M, Perez M. Pediatric Consultants, Hospital for Sick Children, 2018.
  7. Performance-Based Competency in the Interpretation of Pediatric Musculoskeletal Images. Royal College of Physicians and Surgeons, 2016.
  8. Bridging the G.A.P.: A Deliberate Practice Method for Learning Genital Abnormalities in Prepubescent Girls. CAME Wooster Family Education Grant, 2016.
  9. ImageSim – Innovating Medical Education of Medical Images. Innovation Fund, Hospital for Sick Children, 2016.
  10. Establishing Learning Curves for Distinguishing Abusive from Non-abusive Burns and Bruises in Children. Colorado Department of Public Health and Environment, EMS and Trauma Program, Systems Improvement Category, 2015.
  11. Learning Retention and the Timing of Refresher Education After Deliberate Practice of Radiograph Interpretation.  Royal College of Physicians and Surgeons, 2014.
  12. The Deliberate Practice of Paediatric ECGs. Department of Paediatrics, Hospital For Sick Children, 2014.
  13. Learning the Interpretation of Point of Care Emergency Ultrasound Images. Academic Paediatric Association, 2014.
  14. Hinting Strategies for Improving the Efficiency of Medical Student Learning of Deliberately Practiced Web-based Radiographs. Educational Dean’s Fund, University of Toronto, 2011.
  15. Case Mix Study. Royal College of Physicians and Surgeons, Medical Education Research Grant, 2009.
  16. Climbing the Learning Curve: a new approach to teaching non-radiologists x-ray interpretation of ankle x-rays. Royal College of Physicians and Surgeons Medical Education Research Grant, 2005.