Physicians as usability evaluators – first aid for poor EHR usability?

According to earlier studies, Finnish physicians suffer from stress caused by poor usability of their digital tools and feel excluded from healthcare IT development. Active participation in usability evaluation could help involve doctors into developmental work, but more effective and participatory methods are needed. The objective of this study was to research whether physicians with no training in usability evaluation are able to discover usability problems using heuristic walkthrough technique. In addition, we wanted to find out if the level of EHR user experience affects the nature of the usability problems found. Heuristic walkthrough using a modified set of Nielsen’s heuristics and clinical tasks was performed to evaluate usability of a widely used Electronic Health Record (EHR) in Finland. Medical students and physicians with no previous experience in usability evaluation and 1) with or 2) without prior experience in using the EHR in question were recruited as evaluators. A control group of usability experts was used. Physicians were able to identify usability problems, ranging from cosmetic flaws to problems concerning patient safety and significant hindrance for work. Analysis of the found usability problems revealed that experienced EHR users discovered usability problems in nearly all major functions within the given scenario and time span. Physicians as evaluators in heuristic walkthrough has the potential to produce relevant data about EHR usability. The arrangement benefits participating physicians by introducing them to usability. Further research needs to find out, whether the suggested method could support better communication and collaboration between end-users and software developers.


Introduction
The use of an Electronic Health Record (EHR) constitutes approximately one third of a primary health care physicians working day [1].Nationwide surveys conducted in 2010, 2014 and 2017 show, however, that Finnish physicians repeatedly give poor grades to the usability of their EHR systems [2][3][4]].Physicians answering the questionnaires reported that EHRs support their daily work poorly, even if the user has significant experience in using the system in question.A significant portion (about 30%) of the respondents also experienced that improper functioning of an EHR had posed a serious threat to patient safety [2][3][4].
Poorly functioning, time-consuming and inadequate EHRs also cause stress to their users.A longitudinal study among Finnish physicians found out that stress 21. 5.2018 FinJeHeW 2018;10(2-3) 298 derived from healthcare IT systems is increasing [1].
Poor EHR usability has a negative effect on physicians' working ergonomics.Using data from nationwide surveys, researchers have shown that problems experienced when using an EHR are related to higher time pressure and lower job control [5].
Studies have also indicated that many physicians in Finland are interested in participating in the development of healthcare IT systems [6].In the nationwide survey conducted in 2014, more than half of the respondents were willing to discuss their ideas with a colleague who had a connection to software developers, whereas nearly one-third of the respondents expressed interest in giving their feedback directly.However, when asked about current experiences in participating in developmental work, many physicians felt that developers were not interested in their ideas and needs.Physicians also commented that developers were hard to reach, and this created the feeling of IT systems being developed solely by engineers and administrative staff.
It seems that new approaches on development work of healthcare IT systems and innovative methods for enduser participation are needed [7].Active end-user participation in usability evaluation is hoped to lead to both better usability [e.g.6,7] and increased feeling of job control [5] in the future.Being able to utilise physicians as EHR end-users in usability evaluation could provide new possibilities to include contextual focus and clinical viewpoint on evaluation of healthcare IT systems and thereby help develop more usable systems in the future.
Earlier studies in e.g.Sweden indicated that physicians working in clinics show interest in both learning about usability and participating in heuristic usability evaluation [8].Based on their study on involving both usability experts and physicians as evaluators in multidisciplinary teams, Scandurra et al concluded that it would be feasible to train and use healthcare staff in rapid usability inspections (e.g.heuristic evaluation) to locate especially domain-specific (i.e., clinical) usability defects in healthcare IT systems [8].
The overall aim of this study is to find out if a heuristic usability evaluation method can produce plausible results when carried out by a group of primary care physicians.Heuristic methods are of specific interest when it comes to usability evaluation by end-users, because they were originally developed to be used only by trained usability experts [9].This study was designed to find out whether physicians with no training in usability evaluation are able to conduct a heuristic walkthrough procedure and discover usability problems.In addition, we compared the usability problems found by evaluators with different amounts of EHR user experience to see if previous use affects the nature of the usability problems found.
If clinically working physicians are able to use heuristic techniques to evaluate usability of their own digital tools, important usability information could be collected more rapidly and effectively in the future.Furthermore, new methods for communication between software developers and end-user physicians could be achieved.

Background
A user interface with good usability is quick to learn, pleasant and efficient to use and easy to return to [9].Usability can be assessed by using various techniques ranging from simple end-user testing to complex and expensive evaluation procedures that require skilled professionals.Different usability assessment techniques produce different results.Hence, they are commonly used in combinations, depending on the nature and budget of the project in question.

Heuristic usability evaluation methods
Classical heuristic evaluation technique was developed by Jakob Nielsen and Rolf Molich [10], and includes ten heuristics or rules of good user interface design.Commonly referred to as 'Nielsen's heuristics', the list of rules in its current form was composed by Nielsen in 1994 after a profound evaluation of many known usability guidelines [11] non-specific, and may therefore be applied to a large variety of devices and software, including healthcare IT systems.
In a typical heuristic evaluation procedure, 3-5 evaluators freely examine a set of user interfaces and make notes of the usability problems they encounter.Usability problems found by the evaluators are then combined and categorized using a list of usability heuristics and classified by their severity or significance.Nielsen has proposed that including both highly trained usability specialists and domain experts with 'field experience' in a group of evaluators often produces best results.[9] This has later been verified in many usability evaluation studies, also in the field of healthcare informatics [12][13][14].
Cognitive walkthrough is another expert usability evaluation method based on theories of cognitive psychology.The method was first described by Wharton, Rieman, Lewis and Polson in 1994 [15].In a cognitive walkthrough scenario, the evaluators are presented with pre-formatted scenarios or tasks and questions that help taking the role of an end-user ("Does the user know what to do next?"etc.).The goal of cognitive walkthrough is to identify parts of the user interface in which real users are prone to experience usability problems.
Heuristic approach has been shown to produce a larger amount of usability problems than cognitive walkthrough, but with lower mean significance [16].
Cognitive walkthrough techniques, on the other hand, have been criticized for their complexity, slowness and narrow perspective, as the evaluation only focuses on the given user tasks [16].Thereby, combination techniques have been developed with an aim of getting the best out of both techniques.
Heuristic walkthrough is a combination of heuristic evaluation and cognitive walkthrough.It is a taskoriented inspection-based evaluation technique that also includes a free-form pass similar to a heuristic evaluation.It has been shown that heuristic walkthrough produces as many or more serious usability problems as either heuristic evaluation or cognitive walkthrough alone.However, the method produces less non-significant problems than heuristic evaluation [16].
Especially when using evaluators that are not experts in the field of user interface design or evaluation, a simple enough evaluation method is beneficial [16].This is why heuristic walkthrough, as a task-oriented and fairly straightforward technique, was selected as the basis of developing an evaluation tool to be used by physicians with no training in the field of usability evaluation.

Materials and Methods
The objective of the study was to find out whether physicians with no training in usability evaluation were able to conduct a heuristic walkthrough evaluation and discover meaningful usability problems.To accomplish this, a modified heuristic walkthrough method was used to evaluate the usability of a commonly used EHR in Finland.A total of six medical students or physicians (50% female) with or without previous user experience of the EHR in question were recruited as evaluators.
Evaluators were divided into two equally large end-user groups based on whether they had used the EHR in question before.A control group of 3 usability experts (usability engineering master students or usability researchers with theoretical knowledge and practical experience in heuristic usability evaluation) was used.
The study took place in Vallila Healthcare center, Helsinki, Finland.A training version of a commonly used EHR was used as the evaluated system.The training version corresponds well to the production version, with only minor differences in e.g.accessing external databases, such as the national electronic prescription service.
The study design was approved by Research ethics committee in the Faculty of Medicine, University of Helsinki.Permission to conduct the study in a public healthcare center was granted by the City of Helsinki (HEL 2017-012541).

Individual phase
First part of the usability evaluation was an individual evaluation phase that consisted of a short oral introduction to usability and usability problems, two clinically oriented tasks that required using most of the basic features in the EHR under evaluation and a free-form evaluation pass.
A modified version of heuristics was created for this study (Figure 1).It was given to the evaluators in the beginning of the individual evaluation phase.The list of heuristics used is based on a Finnish translation of Nielsen's heuristics [17].The list was reviewed by a team of medical students and doctors prior to distribution, and based on the given commentary some slight modifications were made for better understanding and clarity.Heuristics were also divided in two groups (related to 'flow of use' and 'stability of use') to make the list faster to use.

Task-oriented evaluation pass
The task-oriented evaluation pass included two fictional patient cases.In the first case the evaluator was asked to register new patient information using the EHR, order a medical imaging examination and write a referral to an emergency department of a tertiary hospital.In the second case the evaluator was asked to make computerized patient order entries to alter medication and order routine laboratory tests.The fictional cases were presented in Finnish.All medical information was included in the task instructions, and no medical expertise was required to complete the tasks.
The evaluators were instructed to make notes of all things they considered to be usability problems, based on the introduction given prior to the evaluation.Notes were made by taking screenshots and pasting them into a text-editing tool, followed by a short description of the problem.The evaluators had 90 minutes to carry out the tasks.Completion of the tasks was not required, and evaluators were free to switch between the tasks and even skip tasks according to their liking.

Free-form evaluation pass
In the free-form evaluation the evaluators were asked to freely explore the EHR.For simplicity, access to some of the features used mainly by nurses and other healthcare professionals was restricted.The evaluators continued to make notes of found usability problems, and were encouraged to use the list of heuristics as a source of inspiration.The evaluators had 30 minutes to carry out the free-form pass.

Concluding and evaluating the found usability problems
After the individual evaluation phase, the evaluators met in groups to conclude their findings.This was done in a different session 1-2 weeks after the individual assessments.The groups were formed as following: • Novice users: Three physicians or medical students with moderate (2-5 years) working experience and no prior experience in using the EHR in question.No experience in usability evaluation.
• Experienced users: Three physicians or medical students with moderate (2-5 years) working experience and a minimum of one-year previous experience in continuous use of the EHR in question.
No experience in usability evaluation.
• Usability experts: Three usability engineering master students or researchers with knowledge and practical experience in usability evaluation techniques but no medical working experience and no experience in using the EHR in question.

21.5.2018
FinJeHeW 2018;10(2-3) 301  Table 1.Working experience and experience in using EHRs according to participants' own estimate.One month of working experience was defined as working full-time or having an extensive amount of on-call duties per month (8 or more).There was a small difference in working experience and EHR usage between the Novice user group and Experienced user group, but this was considered as insignificant.

Experience in clinical work Novice users Experienced users
As medical student (group total) During this session, all usability problems found by the group members were combined to a single list and assessed according to their severity and clinical significance.Usability problems were also categorized by the heuristics they violated.

Severity rating
Severity of the found usability problems was assessed by the group members using a scale from 1-4.The rating was based on two main factors: 1) whether the found usability problem presented a hindrance for the work being carried out, and 2) whether the usability problem could pose a risk for patients.
The scale was presented as following: 1 = cosmetic problem causing no hindrance and no risk for patients, 2 = irritates the user, causes slight hindrance but no risk for patients, 3 = significant hindrance and/or possible risk for patients, 4 = catastrophic hindrance and/or certain risk for patients.
The usability problems were assessed one at a time.After a brief discussion, each member of the group gave their own estimates of the severity, after which an average was calculated.

Role of the instructor
In the beginning of the individual evaluation phase, the researcher acted as an instructor and gave a short verbal introduction about usability and usability problems.After this, the evaluators were only allowed to ask for help if they were stuck or otherwise unable to continue carrying out the tasks.It is important to note, that such assistance was only needed by the groups of novice users and usability experts who had no experience in using the EHR in question.
During the conclusion phase, the instructor was responsible for compiling the group's found usability problems into a single list and documenting the associated heuristics and severity ratings.The instructor also gave suggestions in how to better verbalize the problems, although this was mostly needed in the beginning of the session.All severity ratings were given solely by the group members.

Results
Table 2 indicates that a total of 49 usability problems were discovered by novice users, 54 by experienced users and 56 by usability experts.
While the total number of usability problems found by different groups was similar, severity ratings given by the groups varied.As predicted, no severity ratings of 4 were given, as the identical production version of the FinJeHeW 2018;10(2-3) 304 EHR is already widely in use.Severity rating of 3 was most used by the group of experienced EHR users, while the group of usability experts discovered most usability problems with a severity rating of 1.In average, usability experts gave lower estimates on problem severity than physician groups (1.71 vs. 1.80 by experienced users and 1.84 by novice users).
A typical usability problem was discovered by all groups when attempting to make an order entry for adding multiple new medications simultaneously.The users were unable to cancel only one of the multiple drug additions, and if a mistake was made, they were forced to cancel the whole order entry and start all over again.The reason was that a button for cancelling only a single order could not be found.There was a button however, but it was located away from the active dialog, in a top menu.
The dialog that presented most usability problems was adding and viewing patient chart notes, as Table 3A indicates.Depending on the group, a total of 13 to 22 usability problems were found in this dialog, and most of them were given a severity rating of 2.
During the free-form pass, a number of usability problems was discovered in dialogs not included in the patient tasks or affecting the whole system.The group of novice EHR users discovered only a few of these, while the group of experienced users recorded a total of 21 such usability problems with severities ranging from 1 to 3. Also the group of usability researchers discovered a total of 13 usability problems outside the dialogs presented in the patient tasks.A summary of these findings is presented in Table 3B.
Table 4 shows a summary of usability problems by violated heuristics.'Aesthetics and minimalist design', 'consistency and standards' and 'visibility of system status' were amongst the heuristics most commonly associated with found usability problems.
Table 2. Found usability problems by amount and severity.Severity rating: 1 = cosmetic problem, no hindrance, no risk for patient. 2 = irritates the user, slight hindrance, no risk for patient, 3 = significant hindrance, possible risk for patient, 4 = catastrophical hindrance, risk for patient.3A.Abbreviations: EHR = Electronic Health Record, CPOE = Computerized Patient Order Entry.Chart notes refers to the collection of a single patient's health data stored by the EHR, and the dialog for adding new health information.Medication CPOE refers to the list of patient's current and past medications included in the Electronic Health Record (EHR).Laboratory and Imaging CPOE refer to the dialogs used in referring the patient to laboratory test and imaging, i.e. x-ray.Referrals refers to the electronic chart used in referring the primary care patient to a (secondary or tertiary) hospital.Other refers to dialogs outside patient tasks, please see Table 3B.3B.Further classification of dialogs, that include usability problems.Global refers to usability problems appearing in various dialogs throughout the EHR, e.g.symbols used in menus.Patient info dialog refers to patient's connect information stored in the EHR.End of appointment dialog refers to a dialog appearing when the user terminates an appointment.Mail and messages is a dialog used by the physician or other personnel to contact patients and read incoming messages, laboratory results and imaging reports etc.List of patients is a list appearing at the beginning of Task 1, please see Materials and Methods.Prescription refers to the dialog used for drug prescriptions.According to results of a short feedback questionnaire included in this study, nearly all (5 out of 6) doctors and medical students that participated claimed that they had learned about usability during the study, and estimated that they could describe future EHR problems more precisely.All participants were also interested in taking part in a similar evaluation in the future and would recommend it to a colleague.

Conclusion and discussion
The findings of the study indicate that using a group of end-user physicians as evaluators in heuristic walkthrough has the potential to produce promising results when evaluating the usability of a fully functional EHR.Difference in the nature of found usability problems appeared as expected.This can be utilised further by selecting both novice and experienced users as participants in usability evaluation.
Analysis of the found usability problems revealed that experienced EHR users discovered usability problems in all core functions included in the clinical tasks, but also in a multitude of other dialogs inside the given clinical scenario and time limit.The reported problems were often related to difficulties in continuous use (e.g."When adding a new referral, text cannot be copied or pasted using keyboard shortcuts, which slows the user down").Also problems with customisability were noted (e.g."Customising order entry headlines is not possible").Experienced users were more likely to make notes regarding usability problems that could pose a threat to patient safety (e.g."When opening multiple patient charts simultaneously, the windows are 'stacked' instead of 'tabbed', which could easily lead to making entries for the wrong patient").
Also novice users discovered usability problems in the core functions, but they were usually more related to difficulties faced when learning to use the product.(E.g."Cannot find where to add a new chart entry.")Experienced users noted less such mistakes and gave them lower severity ratings than novice users.Usability experts without domain expertise discovered both types of problems, but with an emphasis on difficulties faced when learning to use the EHR.

First aid for poor usability?
The findings of this study suggest that in some cases physicians are able to conduct heuristic usability evaluation after a short training and can identify usability problems.This could provide benefits to both healthcare IT developers and end-user physicians.
By recruiting clinically working physicians as usability evaluators, heuristic methods could be applied on a much larger scale of projects than before.Typically, end-user physicians are much more readily available than trained usability experts.One 'round' of the heuristic walkthrough procedure described in this study lasted four hours, making the whole evaluation procedure possible to be carried out during one afternoon.This makes the described evaluation procedure attractive for healthcare IT projects of various sizes.
Regardless whether physicians are used instead of or in addition to trained usability specialists, heuristic methods offer a structured way to benefit from the clinical experience that the end-users of EHR systems have.Clinical experience may assist in finding more context relevant usability problems and help estimate severity and clinical significance of the found usability problems.Assessing significance of usability problems can be a difficult task for usability specialists without domain expertise [7,8].
For participating physicians, heuristic evaluation methods act as an introduction to principles of usability and proper user interface design -in essence, they tell the evaluators how an ideal system should or shouldn't work.According to a short feedback query, after participating in this study the participants felt they were able to better describe problems they found in an EHR.In addition, when it comes to administrative physicians making decisions about new healthcare IT purchases, knowing principles of usability could help demand systems of higher quality.[7] 21.5.2018FinJeHeW 2018;10(2-3) 308 Although the results look promising, a disclaimer should be made.The role of trained usability experts is undisputable, and using them in usability evaluation produces important information that end-users without proper training in usability engineering hardly can come up with.In addition, the scale of this study is only minor, and further experience in using physicians as usability evaluators should be collected before applying the method to commercial development projects.But as double experts that have both clinical experience in the field of Finnish healthcare and training in usability evaluation are few, more end-users participating in usability evaluation and collaborating with usability specialists is likely to benefit all sides of the development projects to come.
By actively engaging physicians in usability evaluation, better communication and collaboration between software developers and end-user physicians could hopefully be achieved in the future.However, further research is needed to find out more about developers' attitudes towards collaborating with physicians participating in usability evaluation and whether the suggested method could support better communication between physicians as end-users, software developers and usability specialists.
More research is also needed in the case of end-users evaluating usability of other healthcare IT systems, such as physician communication platforms and applications that collect patient health data.As new digital tools for physicians keep appearing at a high rate, engaging doctors in their development could make it easier for these applications to be accepted for clinical use.

Figure 1 .
Figure 1.The modified list of Nielsen's heuristics in Finnish used in this study.The heuristics have been divided into two categories and small modifications have been made for better understanding and clarity (e.g.'user' replaced with 'doctor' etc).

Figure 2 .
Figure 2. Analysis of the found usability problems revealed that experienced EHR users discovered usability problems in all core functions included in the clinical tasks, but also in a multitude of ther dialogs inside the given EHR scenario.Found problems were often related to difficulties in continuous use.(E.g."When adding a new Referral, text cannot be copied or pasted using keyboard shortcuts, which slows the user down.")Also novice users discovered usability problems included in the core functions, but the problems were usually related to difficulties faced when learning to use the product.(E.g."Cannot find where to add a new Chart Entry.")

Figure 3 .
Figure 3. Results of a short feedback survey conducted for medical students and physicians after the usability evaluation procedure (n=6).

Table 4 .
Usability problems by heuristics violated.