Research and Efficacy

Visually diagnosed medical tests (e.g. radiographs) are the most commonly ordered tests in front-line medicine. As such, front-line health care professionals are faced with the task of learning the skill of interpreting these images to an expert performance level by the time they provide opinions that guide patient management decisions. However, discordant interpretations of these images between front-line physicians and expert counterparts (e.g. radiologists) is a common cause of medical error. In paediatrics, this problem is even greater due to the changing physiology with age leading to increased risk of interpretation errors.

ImageSim provides a comprehensive and evidence-based on-line education system that teaches health care professionals the interpretation of visually diagnosed medical tests using the concepts of deliberate practice and simulation. That is, our learning model includes sustained active practice of hundreds of cases where the learner is forced to commit to diagnosis for every case and then receives immediate specific feedback on their interpretation so that the participant instantly learns from each case. Importantly, we have presented these images as we encounter them in practice, and included a normal to abnormal radiograph ratio (with a spectrum of pathology) reflective of our day-to-day practice.

Our research to date shows that this learning method works – and all physicians at varying levels of expertise had significantly increased their accuracy in image interpretation along a “learning curve” as graphed below:

Fig1 - Prototypical Learning Curve

We started this system for paediatric musculoskeletal images, and our participants enjoyed it and asked us for more.  We now have three live courses and are actively developing several more.

Selected Publications

  1. Boutis K, Pecaric M, Stimec J, Carrier B, Chan J, Pusic M. Radiograph Interpretation and Skill Retention: A Multicentre Randomized Control Trial. Med Teach. In press.
  2.  Lee M, Pusic M, Carriere B, Dixon A, Pecaric M, Stimec J, Boutis K. Building Competency in Pediatric Musculoskeletal Radiograph Interpretation: A Multicentre Prospective Cohort Study. Acad Med – Education and Training. In press.
  3. Pusic M, Boutis K, McGaghie W. Role of Scientific Theory in Simulation Education Research. Simulation in Healthcare. 2018 Jan 24 Epub ahead of print.
  4. Pusic M, Boutis K, Pecaric M, Savenkov O, Beckstead JW, Jabour M. A primer on the statistical modelling of learning curves in health professions education. Adv Health Sci Educ Theory Pract. 2017 Aug;22(3):741-759
  5. Pecaric M, Boutis K, Beckstead J, Pusic M. A Big Data Learning Analytics Approach to Process-Level Feedback in Cognitive Simulations. Acad Med. 2017 Feb;92(2):175-184.
  6. Beckstead J, Boutis K, Pecaric M, Pusic M. Sequential Dependencies in Categorical Judgments of Radiographic Images. Adv Health Sci Educ Theory Pract. 2017 Mar;22(1):197-207
  7. Pusic M, Chiaramonte R, Pecaric M, Boutis K. Accuracy of self-monitoring during learning of radiograph interpretation. Med Educ. 2015 Aug;49(8):838-46.
  8. Pusic M, Boutis K, Hatala R, Cook D. Learning curves: A Primer for Health Professions Education. Acad Med. 2014. Aug;90(8):1034-42
  9. Beckstead J, Boutis K, Pecaric M, Pusic M. Sequential Dependencies in Categorical Judgments of Radiographic Images. AHSE. Jun 8. 2016.
  10. Boutis K, Cano S, Pecaric M, et al. Interpretation Difficulty of Normal versus Abnormal Radiographs Using a Paediatric Example. CMEJ. 2016 Mar 31;7(1):e68-77.
  11. Pusic M, Chiaramonte R, Pecaric M, Boutis K. A Simple Method for Improving Self-Assessment While Learning Radiograph Interpretation. Med Educ. 2015;49(8):838-46.
  12. Pusic M, Boutis K, Hatala R, Cook D. Learning curves: A Primer for Health Professions Education. Acad Med. 2014;90(8):1034-1042.
  13. Beckstead J, Boutis K, Pecaric M, Pusic M. The Influences of Sequential Effects in Randomized Online Learning Tasks. Applied Cognition. 2014; 27(5):625-632
  14. Boutis K, Pecaric M, Ridley J, Andrews J, Gladding S, Pusic M. Hinting Strategies for Improving the Efficiency of Medical Student Learning of Deliberately Practiced Web-based Radiographs. Med Educ. 2013;47(9):877-887.
  15. Pusic MV, Andrews JS, Kessler DO, Boutis K. Determining the optimal case mix of abnormals to normals for learning radiograph interpretation: a randomized control trial. Medical Education. 2012;46(3):289-298.
  16. Pusic MV, Kessler D, Szyld D, Kalet A, Pecaric M, Boutis K. Experience Curves as an Organizing Framework for Deliberate Practice in Emergency Medicine Learning. Acad Emerg Med. 2012;19(12):1476-80.
  17. Pusic M, Pecaric M, Boutis K. How Much Practice is Enough? Using Learning Curves to Assess the Deliberate Practice of Radiograph Interpretation. Acad Med. 2011;86(6):731-736.
  18. Boutis K, Pecaric M, Pusic M. Using Signal Detection Theory to Model Changes In Serial Learning of Radiological Image Interpretation. Adv Health Sci Educ Theory Pract. 2010;15:647-658.
  19. Boutis K, Pecaric M, Pusic M. Selecting Radiographs For Teaching The Interpretation of X-Rays Based On Ratings by the Target Population. Med Educ. 2009;43(5):434-441.

Selected Presentations

  1.  ImageSim – An Online Image Interpretation Learning System. Canadian Association of Emergency Medicine, May 2019.
  2. Teaching the Pediatric Emergency Physician How to Interpret Pediatric Point of Care Ultrasound Images. Pediatric Academic Societies and Canadian Association of Emergency Medicine, May 2018.
  3. Competency Based Interpretation of Pediatric Musculoskeletal Images. Pediatric Academic Societies and Canadian Association of Emergency Medicine, May 2018.
  4. Radiograph Interpretation and Skill Retention: A Multicentre Randomized Control Trial. Pediatric Academic Societies (May 2018), Canadian Association of Emergency Physicians (May 2018), Sim ONE (December 2018), International Conference of Residency Education (October 2018).
  5. ImageSim – on-line education innovation. SIM One. December 2017.
  6. Learning How To Interpret Visually Diagnosed Tests Using Simulation and Deliberate Practice. Great Ormond Street. Pediatric Grand Rounds: London, UK, 2017.
  7. Climbing the Learning Curve to Competency of Visually Diagnosed Tests Using Simulation and Deliberate Practice. Alder Hey Children’s Hospital. Pediatric Grand Rounds: Liverpool, UK, 2017.
  8. The Deliberate Practice of Medical Images – Getting You To Competency. Cruces University Hospital. Pediatric Grand Rounds: Northwick Park, Harrow, UK, 2017.
  9. Climbing the Learning Curve to Competency of Visually Diagnosed Tests Using Simulation and Deliberate Practice. Addenbrook’s Hospital. Pediatric Grand Rounds: Cambridge, UK, 2017.
  10. Pediatric Emergency Medicine Medical Education Rounds. Boston Children’s Hospital, Boston, USA, 2017.
  11. Medical Education Grand Rounds. Children’s Hospital of Philadelphia. Philadelphia, USA, 2017.
  12. The Deliberate Practice of Medical Images – Getting You To Competency. Cruces University Hospital. Pediatric Grand Rounds: Bilbao, Spain, 2016.
  13.  Medical student self-assessment deteriorates over and above skill decay on a retention test of radiograph interpretation skills. Council on Medical Student Education in Pediatrics – Ottawa, ON, 2014
  14. Using Process Data from a Radiology Interpretation Simulator To Augment Feedback to Medical Learners International Conference for Residency Education – Calgary, AB; 2013; People for Education – Prague, Czech Republic; 2013; Pediatric Academic Societies – Washington DC; 2013; International Pediatric Simulation Society – New York, NY 2013
  15. Learning Analytics in Screen Based Simulation of Radiograph Interpretation International Conference on Residency Education: Simulation – Calgary AB, 2013
  16. Physicians’ Judgments of Radiological Images on a Multi-trial Discrimination Task: Evidence for the Use of Cognitive Heuristics. Society for Medical Decision Making – Pheonix AZ, 2012.
  17. Determining the Optimal Mix of Abnormals to Normals for Learning Radiograph Interpretation: A Randomized Controlled Trial of Residents. The Ottawa Conference – Miami FL, 2010
  18. Using Signal Detection Theory to Model Changes in Longitudinal Learning of Pediatric Ankle X-ray Interpretation. Pediatric Academic Societies – Baltimore MS, 2009

Selected Grants

  1. Chest Radiograph: When, Where, and Why We Make Mistakes. Boutis K, Thau E, Pusic M, Perez M. Pediatric Consultants, Hospital for Sick Children, 2018.
  2. Performance-Based Competency in the Interpretation of Pediatric Musculoskeletal Images. Royal College of Physicians and Surgeons, 2016.
  3. Bridging the G.A.P.: A Deliberate Practice Method for Learning Genital Abnormalities in Prepubescent Girls. CAME Wooster Family Education Grant, 2016.
  4. ImageSim – Innovating Medical Education of Medical Images. Innovation Fund, Hospital for Sick Children, 2016.
  5. Establishing Learning Curves for Distinguishing Abusive from Non-abusive Burns and Bruises in Children. Colorado Department of Public Health and Environment, EMS and Trauma Program, Systems Improvement Category, 2015.
  6. Learning Retention and the Timing of Refresher Education After Deliberate Practice of Radiograph Interpretation.  Royal College of Physicians and Surgeons, 2014.
  7. The Deliberate Practice of Paediatric ECGs. Department of Paediatrics, Hospital For Sick Children, 2014.
  8. Learning the Interpretation of Point of Care Emergency Ultrasound Images. Academic Paediatric Association, 2014.
  9. Hinting Strategies for Improving the Efficiency of Medical Student Learning of Deliberately Practiced Web-based Radiographs. Educational Dean’s Fund, University of Toronto, 2011.
  10. Case Mix Study. Royal College of Physicians and Surgeons, Medical Education Research Grant, 2009.
  11. Climbing the Learning Curve: a new approach to teaching non-radiologists x-ray interpretation of ankle x-rays. Royal College of Physicians and Surgeons Medical Education Research Grant, 2005.