The development and validation of a heuristic checklist for clinical decision support mobile applications

Abstract

Mobile phone use is ubiquitous in the healthcare setting and as a result, mobile health applications targeted for clinician use have proliferated. However, there is currently a lack of guidance to evaluate these types of applications (Dwivedi et al., 2017). This study used an eight-step method to develop and validate a heuristic checklist to evaluate clinical-decision-support mobile applications (mCDSapps) for their usability and safety for point of care use. The new mCDSapp checklist was evaluated against a control checklist specific to mobile interfaces through heuristic evaluation of two mCDSapps (MDCalc Medical Calculator and Evidencio). Evaluators using the mCDSapp-specific checklist identified more usability issues, including mCDSapp domain-specific issues, and issues that were considered critical to the evaluation of mCDSapps, than the evaluators using the control checklist. Feedback received during validation resulted in 83 adaptations, six eliminations, and five new questions. The new checklist was also evaluated through expert judgment by a Human Factors Practitioner from the healthcare domain and two attending physician hospitalists with human-factors knowledge. Based on feedback at this stage in the validation process, eight adaptations were made, and one question was eliminated. User testing was conducted with representative users on MDCalc and Evidencio to compare usability issues found with those identified through heuristic evaluation. Usability issues identified in the user testing mostly overlapped with the issues detected during the heuristic evaluation, however, unique findings contributed to the addition of three new checklist questions. The final mCDSapp heuristic checklist includes 282 questions grouped into 13 heuristic categories.