Digi-HTA: Health technology assessment framework for digital healthcare services

Health technology assessment (HTA) refers to the systematic evaluation of the properties, effects, and/or impacts of health technology. The main purpose of the assessment is to inform decisionmakers in order to better support the introduction of new health technologies. New digital healthcare solutions like mHealth, artificial intelligence (AI), and robotics have brought with them a great potential to further develop healthcare services, but their introduction should follow the same criteria as that of other healthcare methods. They must provide evidence-based benefits and be safe to use, and their impacts on patients and organizations need to be clarified. The first objective of this study was to describe the state-of-the-art HTA methods for mHealth, AI, and robotics. The second objective of this study was to evaluate the domains needed in the assessment. The final aim was to develop an HTA framework for digital healthcare services to support the introduction of novel technologies into Finnish healthcare. In this study, the state-of-the-art HTA methods were evaluated using a literature review and interviews. It was noted that some good practices already existed, but the overall picture showed that further development is still needed, especially in the AI and robotics fields. With the cooperation of professionals, key aspects and domains that should be taken into account to make fast but comprehensive assessments were identified. Based on this information, we created a new framework which supports the HTA process for digital healthcare services. The framework was named Digi-HTA.


Introduction
Health technology assessment (HTA) is the systematic evaluation of the properties, effects, and/or impacts of health technology.The main purpose of the assessment is to inform decision-makers in order to better support the introduction of new health technologies.[1,2] A health technology is defined by the WHO as "the application of organized knowledge and skills in the form of medicines, medical devices, vaccines, procedures and systems developed to solve a health problem and improve quality of life" [3].It can also be defined as covering all interventions that may be used to promote health; to prevent, diagnose, or treat acute or chronic disease; or for rehabilitation, including pharmaceuticals, devices, procedures, and organizational systems used in healthcare [4].
HTA is a multidisciplinary process that summarizes information that has been collected in a systematic, transparent, unbiased, and robust manner [5].It covers nine domains: (1) the health problem and current use of technology; (2) description and technical characteristics of the new technology; (3) safety assessment; (4) clinical effectiveness; (5) economic evaluation, typically cost-effectiveness analysis or cost-utility analysis; (6) ethical analysis; (7) organizational aspects; (8) social aspects; and (9) legal aspects [5,6].Conducting the assessment requires explicit analytical frameworks, drawing on clinical, epidemiological, health economic, and other information and methodologies [4,7].The themes of evaluation arise from healthcare and health policy.After the assessment, knowledge must be disseminated and implemented which affects healthcare and health policies [4].Since options in healthcare treatment are growing faster than the available resources, the need for value assessment is continuous [6,8].
The assessments' report forms can vary between a full HTA report, a rapid review, contextualization of assessment reports produced elsewhere, and a mini-HTA report.A full HTA report covers all nine domains, whereas a rapid review covers the first four domains, being therefore transferable from one country to an-other.[5][6][7] The mini-HTA questionnaire was developed by Danish HTA experts for local decisionmakers.It covers four HTA domains-technology, patient, organization, and economy-and is therefore easier and quicker to do than a full HTA.[9] Digitalization of various services in our modern society has become feasible because information like voice, images, and text can be stored and transmitted in digital-i.e., binary-format.In healthcare, this means that digital patient and client information can be shared among all those parties that use that particular information [10,11].According to the European Union (EU) definition, eHealth is the use of information and communication technology in health products, services, and processes, combined with organizational change in healthcare systems and new skills [12].The World Health Organization simply defines eHealth as the use of information and communication technologies for health [13].The EU further emphasizes the ability to improve citizens' health, the efficiency and productivity of healthcare delivery, and the economic and social value of health [12].
Digitalization in healthcare can be divided into three major developmental waves.The first wave of digitalization includes the construction of basic local healthcare information infrastructure.This consists of electronic medical records, picture archiving and communication systems, laboratory systems, as well as other auxiliaries to medical records.Regional communication and repositories as well as referral systems are included in this phase.[14,15] The second wave of digitalization means further utilization of the collected and stored data as well as citizen involvement.A national health information exchange combines the information of various local and regional data sources and allows the citizens to have ubiquitous access to their data.[16] New types of services connected to the local and national infrastructure are emerging, like selfcare services and prehospital and posthospital care paths for citizens [17,18].Typical aspects of these new services are that they empower citizens with the responsibility for their own health and are accessible via mobile phones [18,19].There are

6.11.2019
FinJeHeW 2019;11(4) 328 already plenty of independent mobile phone apps for health and welfare monitoring.In order to make these data usable, Finland is building a national repository for the data of certified applications [20].This mHealth evolution follows the current trends in our society: A smartphone is the most flexible platform for digital services [21][22][23].
The third wave of digitalization improves decision support, guidance, and processes based on existing data.This includes novel solutions involving artificial intelligence (AI) and machine learning in healthcare [24][25][26].
The potential ways to use AI in healthcare are assisted or automated diagnosis, personalized medication and care, and medical imaging [24,27,28].A specific application area is robotics in healthcare, which can provide a large variety of solutions like automated information processing, rehabilitation, helping personnel to lift patients, helping to distribute drugs, and even communicative robots [29][30][31].
The introduction of mHealth solutions in Finnish healthcare has progressed, but the use of robotics and AI is still limited [32,33].For that reason, the Ministry of Social Affairs and Health (STM) has launched the Wellbeing and Health Sector's Artificial Intelligence and Robotics Program (Hyteairo) to support and speed up the utilization of AI and robotics in the healthcare sector.The program's areas of focus are living at home, care and logistics in the hospital environment, pharmacotherapy and pharmaceutical service, and well-being coaching and rehabilitation.[30] Traditional HTA does not cover all factors relevant to digital tools, such as accessibility and data security and protection [34][35][36][37][38].In digital technologies there is great variability, even within the same family of technologies, requiring evaluations at the level of products rather than intervention types.Healthcare professionals want to know which digital tools they can use and what the clear benefits are in their daily work.Companies need information on what is required to get their solutions approved for use.Finally, assessments provide evidence-based information for decision-makers to support their decisions related to new digital healthcare services.All these factors require a specialized evaluation framework for digital services.
The main aims of the study were as follows: 1. Clarify which available HTA frameworks are suitable for assessment of mHealth, AI and robotics 2. Evaluate which HTA domains are needed for assessment of mHealth, AI, and robotics 3. Develop an HTA framework for digital healthcare services to support the introduction of novel technologies into Finnish healthcare.

Material and methods
We identified mHealth, AI and robotics as technological fields that are key drivers of the need for a new HTA framework for digital healthcare services.To evaluate the status of existing HTA frameworks for mHealth, AI, and robotics, we performed an integrative literature review to identify the state-of-the-art frameworks and their HTA domains to assess digital healthcare services.We used Boolean searches to obtain relevant articles from PubMed, Medline (Ovid), Scopus, Web of Science and CINAHL databases.The searches were performed separately for mHealth, AI and robotics.We checked the papers related to health technology assessment or assessment frameworks for all those technology subfields.The search terms for mHealth were ("health technology assessment" OR "assessment framework") AND (mobile app* OR mobile medical app* OR mHealth OR mobile health app* OR electronic health app* OR eHealth app*).The search terms for robotics were ("health technology assessment" OR "assessment framework") AND (robot*) AND (health* OR medic* OR rehabilitation OR logistic* OR hospital OR homecare) and for AI the terms were ("health technology assessment" OR "assessment framework") AND ("artificial intelligence" OR AI OR "machine learning" OR "deep learning") AND (health* OR medicine OR hospital OR care).The final inclusion of the relevant HTA literature for mHealth, AI and robotics was done by selecting the most relevant papers to the HTA domains and our framework.Figure 1 illustrates the complete literature selection process.We identified five relevant articles for final evaluation of mHealth HTA frameworks.A total of ten HTA articles were identified related to robotics.All those articles were HTA reports on robotic surgery not actual HTA frameworks.Anyhow, we included all those articles in the final evaluation to find the key HTA domains to assess robotics.We included three articles related to AI for final evaluation.
Due to the limited number of relevant HTA framework papers for mHealth, AI, and robotics, we performed an additional literature search of well-known organizations' webpages and databases.The aim was to identify methods to assess those new technologies and to find relevant technology guidelines.Additionally, the relevant regulatory and standardization documents were included in the search.With this search we found four web pages for mHealth, two guidelines and one document for AI, and four documents for robotics.
The material was complemented by unstructured interviews with seven technology companies and five healthcare service providers.These technology companies were vendors for mobile applications and robot vendors for the rehabilitation, logistics, and medicine distribution fields.The interviewees' job titles ranged from technology specialist to chief executive officer.The healthcare service providers represented homecare, rehabilitation, and hospital district organiza-tions.The selected organizations used mHealth and robotics solutions in their daily work.The interviewees' job titles ranged from physiotherapist to project coordinator.The aim of the interviews was to deepen the knowledge of what key issues should be taken into account when introducing novel technologies in healthcare.Notes were written during the interviews and gathered into a memo afterwards.Usability and accessibility criteria were checked with experts from the Finnish Federation of the Visually Impaired (FFVI) and the Finnish Association on Intellectual and Developmental Disabilities (FAIDD) via email interviews.In addition, one phone conference was arranged with National Health Service (NHS) Digital to clarify their assessment process for mHealth solutions.
During the development of the framework, four multiprofessional workshops were held.The participants consisted of a senior planning officer from the Finnish Coordinating Center for Health Technology Assessment (FinCCHTA) and an HTA specialist, AI specialist, and medical doctor from the Faculty of Medicine, University of Oulu.The aim of the workshops was, with the cocreation of the professionals, to summarize the collected information and define the overall structure of an HTA framework and criteria in detail.A common web workspace shared among the professionals was used during development.

Selected by abstracts
The removal of duplicates

Literature review and web search
A systematic literature review by Moshi et al. showed that existing mHealth evaluation frameworks are not suitable for use in HTA because none of the evaluated frameworks cover all the core HTA domains [39].All of the frameworks included in the evaluation assessed effectiveness, but nearly a quarter of the frameworks did not assess safety, and only one assessed costs [39].
Another review paper also identified several missing HTA domains in mHealth HTA reports against the EU-netHTA Core Model and the INAHTA checklist [40].A total of twelve HTA agencies were evaluated, and the finding was that none of them had a formal HTA process for mHealth [41].Bradway et al. highlighted that there was a clear need to define a standard HTA framework for mHealth technologies that would identify potential solutions that may provide added value to patients and the healthcare system.The HTA framework should provide information from the following domains: intended use, functionalities and content, level of development, data security and privacy, interoperability standards, and usability.[42] In addition, Zelmer et al.'s articles related to e-mental health apps also highlighted the aspects of clinical applicability, supported platforms, targeted users, developers' transparency, funding transparency, and price [43].
NHS Digital from the United Kingdom (UK) has a wellknown NHS Apps library, which includes more than one hundred evaluated mHealth solutions [44,45].Their criteria and their questionnaire called Digital Assessment Questions (DAQ) are publicly available on NHS webpages.Their assessment includes pre-assessment, effectiveness, clinical safety, data protection, security, usability and accessibility, interoperability, and technical stability [46] There are also two companies called ORCHA and Our Mobile Health in the UK which assess mHealth solutions, but their detailed criteria are not publicly available [47,48].ORCHA also has its own apps library for evaluated mHealth solutions [49].The United States Food and Drug Administration (FDA) has only assessed mHealth solutions that fall under the definition of "medical device" [50].
Evaluated HTA reports related to robotic surgery have mostly covered the cost effectiveness, but some of the reports also included the following HTA domains: clinical effectiveness, safety, organizational issues, and technology [51][52][53][54].In addition, Fosch-Villaronga et al.'s articles related to service robots also highlighted the aspects of usability, data protection, and security [55,56].From a regulatory point of view, the field of robotics is broad so at the moment that there is no specific robot regulation in which clear procedures, boundaries, and requirements are explained [57,58].
The literature review showed that there is a lack of HTA frameworks for AI because it is a new era in HTA, and for that reason, new HTA criteria and processes are needed [59].Assessment of the systems could be very complex, since their input data could be from multiple sources and an adaptive AI-based algorithm learns continuously and becomes more effective over time [59,60].AI brings a new perspective to decision-making in support of healthcare professionals' own decisions, so its decisions must be trustworthy and transparent [61][62][63]

Interviews
The key findings from the interviews were that new technologies typically change the care path.For that reason, the implications for patients and organizations should be understood.The other thing that came out was that there should be clear targets for the introduction of the digital healthcare service, meaning that the desirable effectiveness of the new product is determined and the indicators to measure it are defined.The other aspects which arose during the interviews were safety and usability issues, as well as cooperation between companies and healthcare service providers like customer support and trainings.
Experts from FFVI and FAIDD highlighted that end-users with different constraints, such as vision and hearing impairments, should be taken into account when designing a product's usability.The development of accessibility and usability should also be a continuous process developed on the basis of customer feedback.
The main message from the NHS Digital interview was that they wanted to provide reliable digital tools for healthcare professionals and citizens.For that reason, they have developed a comprehensive assessment process, which they continuously develop and update, e.g., when the standards and regulations change.

Workshops
The HTA process that is based on the mini-HTA questionnaire is easy and fast and therefore suitable for assessing rapidly developing digital healthcare services.However, it lacks important elements that are essential for digital health services.Consequently, the aspects presented in the previous chapters should be included in the digital HTA framework in addition to the aspects of the mini-HTA questionnaire.
At first, during the workshops, we identified which main aspects should be included in the HTA framework based on information from the literature review, web searches, and the interviews.The main aspects selected were intended use of the product, intended user groups of the product, patient and organizational aspects, description and technical characteristics of the product, level of development, cost, effectiveness, safety, technical stability, interoperability, usability and accessibility, and data security and privacy.We decided to combine the first five aspects under the HTA domain of product information.Also, we added the company information domain to clarify what the company's business model was and whether the quality management systems that are required in the healthcare sector were in use.In addition to these general criteria, we discovered that AI and robotics need their own specific criteria as well.
Finally, we concluded that there were eleven main domains that needed to be included in the HTA framework to cover key assessment requirements of all identified technology subfields.These domains were company information, product information, cost, effectiveness, clinical safety, technical stability, usability and accessibility, interoperability, data security and protection, AI, and robotics.We decided to start creating our digital HTA framework using an Excel-based questionnaire, where the main HTA domains were included in different sheets.The NHS uses the same approach in their questionnaire [46].An Excel-based approach also gives us the freedom to easily add or remove features in the rapidly changing technology field.
A project called Kyberterveys, supervised by the National Emergency Supply Agency, has developed its own requirements for healthcare service providers to evaluate data security and protection issues in the procurement phase [65].Requirements, which are mainly based on in ISO27k ja IEC-62443 standards, were developed within eleven years in several projects [66].Their assessment procedure includes two documents, (1) Data Security and Protection Preliminary Task and (2) Information Security and Data Protection Requirements.We evaluated that those requirements are also very suitable for the HTA process, and thus, we decided not to start developing our own detailed criteria for data security and protection assessment but instead to use their checklist as such.This means that discussing those detailed data security and protection criteria was not within the scope of this article.

Health technology assessment framework for digital healthcare services
The framework is called Digi-HTA because it combines the HTA process and digital healthcare services.As a summary, our framework includes all of the traditional HTA aspects except ethical, social, and legal issues [5,6].

Implementation process
The assessment process for digital healthcare services is presented in Figure 2. Three documents are used to collect all needed information on the product under assessment from the company offering the product.The Digi-HTA framework collects all information except data security and protection issues, which are covered in the Data Security and Protection Preliminary Task and Information Security and Data Protection Requirements documents.The company fills out all the documents and sends them to HTA and data security and protections experts for further evaluation.After the assessment HTA experts will publish their HTA recommendation for the product.The most important things in the product recommendation are safety, effectiveness, cost, data security and protection, as well as usability and accessibility.

Discussion and conclusion
The first objective of this study was to learn the stateof-the-art HTA methods for mHealth, AI, and robotics.The available HTA frameworks and their HTA domains were evaluated.A key finding was that formal HTA processes for those novel technologies were missing [41,59].There were some good and proven practices like the NHS Apps Library's assessment process for mHealth, but the overall picture shows that a lot of development work is still needed in this field, especially for AI and robotics [44,45,59].The introduction of these technologies in healthcare is still in an early stage, and they bring with them new aspects for assessments [59].
Although the possibilities of applying robots in healthcare are broad, HTA studies are still focused mainly on surgical robots and their cost effectiveness [52,53].Service robot studies have highlighted that the aspects of usability, data protection, and security should be taken into account when assessing robot applications [55,56].Due to the special nature of AI solutions, HTA and guideline-producing agencies should be adapting their methods and processes for AI solutions.The biggest issues are that AI is a new decisionmaker in addition to healthcare professionals, and the algorithm is learning continuously and becoming more effective over time.[59,60] Those aspects create new challenges when performing comprehensive HTA for AI.
The second objective of this study was to evaluate the HTA domains needed in the assessment of mHealth, AI, and robotics.The evaluated studies highlighted that, in addition to traditional HTA domains (e.g.effectiveness and safety), usability and accessibility issues should be taken into account in all applications, from mHealth to robotics [42,43,55,56].A well-designed user experience improves product acceptance and increases people's confidence in the product.Patients may have limited abilities to use digital services due to age or illness, in which cases accessibility and usability play an especially crucial role.[38] From the healthcare personnel's point of view, poor usability can pose a risk to patient safety due to misuse of the product [35].
Data security and protection play a key role in ensuring people's trust in a new digital service.In the healthcare sector, this is even more important because the handled data could be very sensitive [64].mHealth applications can transfer data through various interfaces and different kinds of wireless technologies, and thus, data security and protection should be guaranteed from end to end [36].Even personal healthcare data from multiple sources could also be used as input for AI solutions, and therefore, the access rights and data protection issues should be in order in every use case, including the retraining of the AI system [64].In robotics, it must be ensured that the system cannot be hacked, which enables avoiding the possible safety risks to end users [56].
The key aspect when implementing robotics in healthcare is safety issues-for example, when the connection is lost, there should be no harm to the patient due unexpected behavior of the robot.As the robot moves, it must not collide with people.The forces used by the robots must be such that the patients are not injured, in any case.[55,67] In the commissioning phase, when the robot may need training, such features as navigation maps should be implemented [31,68].Needed infrastructure changes should also be identified, such as whether there is a need to renew elevator control systems or whether the corridors are wide enough to guarantee the efficient use of robots [31].
Operating and charging times are crucial for batterypowered robots, as they can limit their continuous working time, e.g., in hospital logistics and rehabilitation [31,56,69].
When utilizing AI in healthcare, the key question is the exact definition of which problem the AI will solve [61].
Due to the nature of the healthcare sector, the AI solutions should be reliable in every circumstance because they could affect people's lives [61,63].The design should be so robust that it can handle situations like missing or erroneous data, e.g., incorrect recording in patient information systems [61,63].Access rights to data should be in order in every case, including retraining [64].The AI's operational logic should be transparent to ensure that healthcare personnel can trust the system, e.g., in cases when it suggests an action which conflicts with the personnel's own decisions [61].Personnel must know what kind of recommendations the AI provides and who makes the final decisions on treatment [61].
From an HTA perspective, the change from the Medical Device Directive (MDD) to the new Medical Device Regulation (MDR) will create new aspects to assess [70].Accordingly, MDR manufacturers will need to generate and provide more in-depth clinical data to prove safety and performance claims, including tighter equivalency standards.There are also requirements for quality management system (QMS) and post-market surveillance systems.[71] The regulation transition period will be end of May 2020; after that, all products will have to meet the MDR requirements to be CE marked [70].The medical device classification rules are going to change, and products will be classified on the basis of the risk they generate [71].In practice, this will typically mean a much stricter classification for software as a medical device [72].
Technology is developing fast in this sector, and consequently, fast assessments are needed.Also, productlevel assessments are needed because the technology solutions could vary a lot in certain care needs.Mini-HTA has been developed for fast assessments, but those above thematic areas are not covered in it, so there is a need for a new HTA framework to better support the introduction of digital services.Collected information technical stability, usability and accessibility, interoperability, and data security and protection issues should be included in this HTA framework in addition to the aspects introduced by mini-HTA.Also, aspects from AI and robotics need their own specific criteria.
Thus, the third objective of this study was to develop an HTA framework for digital healthcare services to support the introduction of novel technologies into Finnish healthcare.Based on the theoretical background, this work produces a framework called Digi-HTA, an HTAframework for digital healthcare services based solidly on HTA principles.The framework collects all the needed information on the product under assessment.It covers all the aspects that we identified as being im-portant in order to perform a fast yet comprehensive HTA review.It can be used to assess various digital healthcare solutions with different degrees of maturity.
Our goal is to provide a tool for facilitating the work of Finnish HTA experts in evaluation of new technologies.
To the best of our knowledge, this is the first HTA framework developed for digital healthcare services which combines novel technologies like mHealth, AI, and robotics in the same framework.
The first pilot assessments will start with mHealth and robotics solutions.This development work is also linked to the national Hyteairo strategy because we are going to assess the products in the key priorities of the strategy like medicine-dispensing and rehabilitation robots.
The development of the Digi-HTA framework will be continued by collecting the information from pilot assessments and making changes according to companies' and healthcare service providers' feedback.The Digi-HTA framework will also be updated when new technology features are introduced and when there are updates to specifications or regulations.A possible future development target would be to add an automatic scoring system at least to some part of the framework to speed up assessments.

Limitations
At this phase, the implementation of the Digi-HTA framework has just started with pilot assessments.This means that we don't yet have any feedback from companies and healthcare professionals about further development needs for the implementation of Digi-HTA.The indicators to evaluate the developed HTA criteria and their suitability for assessment of digital healthcare services are missing from this development phase, so further research is needed on this subject.

Declaration of conflicting interests
None.

Figure 1 .
Figure 1.Assessment process for digital healthcare services.