Evaluation of open access COVID-19 related mobile applications in India: An application store-based quantitative analysis

: Purpose: To assess the features/functionalities and quality of the (open access) COVID-19 specific mobile application for India using the Mobile Application Rating Scale (MARS) and the quality of the reported COVID-19 data using the COVID-19 Data Reporting System (CDRS). Methods: We used an analytical, cross-sectional study in which we reviewed all open access (free) mobile phone-based applications across the application stores, namely Google Android Play Store and iTunes, and Google engine. We used MARS and CDRS to assess the mobile applications applicable to India. Results: We found a total of 247 applications through the iTunes store (n=176), android store (n=70) and Google search (n=1). Out of 247, 70 applications matched the inclusion criteria, and only 42 applications were accessible for detailed evaluation using MARS. The overall mean (SD) MARS score was 3.27 (0.59). The mean (SD) score for application mean quality, app subjective quality and app-specific quality domains were 3.43 (0.43), 2.95 (0.71), and 3.44 (0.82), respectively. Of the 20 applications evaluated using CDRS, Aarogya (Agra) Sarvam Setu and Odisha COVID had the highest normalized score (0.9), whereas Madhya Pradesh COVID response app and WHO Academy COVID-19 had the lowest (0.1). Conclusion


Introduction
World Health Organization (WHO) declared the coronavirus disease 2019 (COVID-19) pandemic on March 11, 2020, due to its novelty, high infectivity, and fatality [1]. The global count of COVID-19 cases has crossed 328 million and deaths around 5.5 million as of January 16,2022. Despite the intensive and international efforts, there has been limited development in identifying the drug of choice for curing COVID-19 [2]. Even though various vaccines have been approved against the virus, early detection and isolation and non-pharmacological preventive measures are still regarded as the most effective ways to wade through the pandemic [3].
Governments and various agencies actively used digital technologies to control COVID-19 transmission [4,5]. The high penetration of information and communication technologies in the community and health sectors facilitated this process. For example, a total of 3.5 billion smartphone users are present worldwide as of 2020 [6]. Mobile phone-based application (hereafter mobile applications) is one such technology used in the past to control the Ebola and Zika virus epidemics and other disasters worldwide [7][8][9]. It has been used in surveillance, contact tracing, raising awareness among the public, improving access to care, referral linkage, follow-up, and delivery of other services.
Various functionalities of the mobile applications like risk assessment, self-reporting of COVID-19 symptoms, testing and reporting, contact tracing, tracking the patients or quarantined people, mapping the patients and healthcare facilities, connectivity with other devices (using Bluetooth), face recognition, booking the appointment and online consultations, knowing the status of vacant beds and availability of oxygen cylinders, and other features have been used to tackle the COVID-19 pandemic [10]. In addition to availing the necessary medical consultations, telemedicine, especially integrated with mobile applications, has expanded the accessibility of healthcare and reduced SARS-CoV-2 exposure among healthcare workers and patients [11,12]. The use of advanced digital strategies/solutions like artificial intelligence (AI)-based algorithms facilitated the instant contact tracing and tracking of high-risk populations for COVID-19 [13] . Furthermore, non-COVID-19 related healthcare services are delivered using similar advanced algorithms, such as a virtual mental health assessment and an AI-based chatbot [14].
With its own Aarogya Setu COVID-19 contact tracing software, the Government of India launched an ambitious attempt to halt the spread of the COVID-19 epidemic [15]. The Aarogya Setu application has features for selfrisk assessment, current risk status, detection of a nearby positive or high-risk person, a chatbot to clarify common doubts, location-specific COVID-19 updates, emergency COVID-19 helplines and e-Pass for movement from one place to another. This application uses AI-based algorithms to assess contact with high-risk people and instruct individuals about changing risk status and current best practices. The application also included a list of COVID-19 testing centers, navigation, test results, and instructions depending on the results. The government made the installation and use of the application mandatory for all government employees, travelers, people who tested positive and quarantined, and people involved in public utility services. Analytics of Bluetooth contacts and location data has also assisted in identifying potential COVID-19 hotspots, allowing state governments, district administrations, and health authorities to implement the appropriate COVID-19 pandemic containment measures. In line with Aarogya Setu, several state governments developed their mobile applications with add-on features since health is a state subject in India.
Despite several mobile applications with different functionalities, only a few studies reviewed mobile applications related to COVID-19. The available studies are limited to the non-use of validated quality assessment tools and assessment of a limited number of mobile applications, especially systematically missing the applications developed by various Indian states [10,16,17]. Furthermore, they did not report the quality of COVID-19 data reporting in these mobile applications. Although Vasudevan et al. assessed the quality of COVID-19 data reporting using a novel tool, "COVID-19 data reporting system (CDRS)", they included only the data extracted from websites and not mobile applications [18]. With this background, we conducted the current study to assess a) the profile (features/functionalities) and quality of mobile applications for COVID-19 applicable to India; and b) the quality of COVID-19 data reporting in these mobile applications.

Materials and Methods
We conducted an analytical cross-sectional study through a market review of all free mobile applications for COVID-19 applicable in India. We searched the Apple App Store (iOS applications), Google Play Store (Android applications), and Google engine including searching COVID-19 websites of all Indian states to identify the mobile applications [19][20][21]. We used the following key terms to search COVID-19 related applications in the stores: "Coronavirus", "Covid", "COVID-19", "SARS-CoV2", "Coronavirus app India", "Corona Contact Tracing", "Corona", "Coronavirus Symptom Tracking", "Coronavirus testing", and "Symptom tracking". And search phrase "Applications for COVID-19" was used to explore the applications in the Google engine. The applications found in the app stores or Google engine were enlisted and screened for eligibility based on the title and description provided by the developer as of October 10, 2020 ( Figure 1). We excluded the a) paid applications, b) non-working applications, c) duplicates, or; d) the applications developed only to collect relief funds for COVID-19. All the eligible applications were downloaded and checked for features related to COVID-19 and excluded the applications with download issues or restricted access for evaluation. We used MARS to assess the quality of all the included mobile applications [22] . MARS is a validated tool used to evaluate mobile applications [22]. It is a 23-item tool with (objective) app quality rating and app subjective quality subscale. The objective app quality rating tool consists of four subscales: engagement, functionality, aesthetics, and information. These four subscales consisted of nineteen items. The engagement subscale includes five items, i.e., fun/ entertainment, interesting, customizable, interactivity and appropriateness for the target group. Functionality was rated based (four items) on the application's performance, ease of use, navigation, and gestural design. The aesthetic subscale consisted of three items, i.e., layout, quality of the graphics and visual appeal of the app. The information subscale contains seven items, app description accuracy, specific goals, quality, and quantity of information, visual information, the credibility of the app source, and scientific testing of the app (evidence-based). A fivepoint Likert scale (1-Inadequate, 2-Poor, 3-Acceptable, 4-Good, 5-Excellent) is used to rate each item. The five items (specific goals, quality, and quantity of information, visual information, and scientific testing of the app) in the information scale also have the option of "Not applicable". The items marked with "Not Applicable" were not included in the score calculation. Each subscale's mean score is calculated, and a total objective app quality score is obtained by combining all four subscales together. The app subjective quality is calculated based on the mean of the four items in the subscale. The intraclass correlation coefficient (ICC), which assesses the interrater reliability, was 0.79, and the internal consistency was 0.90 (Cronbach alpha). Internal consistency of individual subscales is also very good (Cronbach alpha: engagement-0.89, functionality-0.80, aesthetic-0.86, information-0.81 and app subjective quality-0.93) [22]. In addition, there are six App-specific added items to assess the impact of the app on changing the knowledge, attitude, intentions or behaviors which are not part of the latent structure of the MARS. Two reviewers (AP and AG) independently rated all the applications using MARS. The third reviewer (SK) adjudicated the dispute in any variable among them. The reviewers agreed most of the time (95%) while rating the COVID-19 related applications using MARS.
We used the CDRS to assess the quality of COVID-19 data reporting in mobile applications which had dashboard features (showing COVID-19 data). CDRS is a novel metric table used to assess the quality of reporting of COVID-19 data. It consists of 12 metrics grouped into four scoring categories: availability, accessibility, granularity and privacy of the data. The availability category consisted of the total number, daily number and historical data metrics. The ease of access, availability of the data in English, the total and daily trend graphics are the four metrics under the accessibility category. The granularity included the data reporting stratified by age, gender, comorbidity and district. The compromise in privacy is the only metric included under the privacy category. For each specific metric, we assessed the status of reporting the confirmed cases, deaths, recovered, cases in quarantine and intensive care unit and scored '0' or '1' based on unreported or reported items, respectively. In the privacy category, we provided a score of '+1' if there was no compromise with privacy and '-1' otherwise [18]. Two independent reviewers (AP and SR) assessed all eligible mobile applications using CDRS. We have observed a good level of agreement between the reviewers (Cohens Kappa-0.755). A third reviewer (SK) resolved all the disagreements by re-evaluation in discussion with the primary reviewers. We calculated the raw scores for each category and normalized the scores between 0 and 1. We also looked at the existence or absence of various key features in the app, such as self-screening, risk scoring, contact tracing, information, education, the communication content on COVID-19, and helpline numbers, and compared the MARS scores using the Mann Whitney U test. A p-value <0.05 was considered statistically significant.

Results
The Apple App Store (n-176), Google Play Store (n-70), and Google (n-1) yielded a total of 247 applications.
After applying the inclusion and exclusion criteria and removing duplicates, we assessed 70 mobile applications related to COVID-19 applicable to India (Figure 1). Of them, 62 were Android applications, and 21 were iOS applications (Table 1). Thirteen (18.6%) applications were compatible with Android and iOS devices. The general population (48, 68.6%) followed by doctors or government officials (11,15.7%) were the primary target audience for the applications. A total of 55 (78.6%) of the applications could be used by children as young as three years old, whereas four (5.7%) of the applications were exclusively for persons aged ≥17 years.  M a h a K a v a c h ( M a h a r a s h t r a )
Of the 70 mobile applications, we have evaluated 42 applications in detail for quality using MARS. The mean engagement score was 2.26, the lowest of all the domains ( Table 2). The functionality, aesthetic, and information mean scores were 3.83, 3.76, and 3.88, respectively. The mean of the application mean quality score was 3.43, and the overall MARS score was 3.27. We have enlisted the COVID-19 related features available in all 42 applications and grouped the 28 identified features under COVID-19 risk assessment, COVID-19 related information, COVID-19 surveillance, updates and services and other features ( Table 3). The presence of features like information, education, and communication (IEC) on COVID-19, Interactive/Chatbot, COVID-19 case updates/Dashboard, COVID-19 testing labs, hospital bed status, FAQs and relief fund in the application was associated with a substantially high MARS score (p<0.05).  A total of 20 mobile applications had a dashboard providing COVID-19 data. The application-wise CDRS normalized score developed and used by various Indian states is given in Figure 2. Aarogya (Agra) Sarvam Setu, developed by Uttar Pradesh (UP) and Odisha covid dashboard, scored the top with 0.9. The MP COVID response app and WHO academy covid-19 had the lowest score (0.1). The normalized CDRS score for Aarogya Setu and Covid-19 Care (developed by Tamil Nadu) was 0.8. Chandigarh's CHDCOVID application got -1 in the private domain of the scale.

Discussion
Our study assessed the features or functionalities, and quality of the COVID-19 applications using MARS and the quality of COVID-19 data using CDRS. Of the identified applications, only 60% were functional and included in the MARS evaluation. The score for overall and individual subscales of MARS is more than 3.2, except for engagement (2.26) and app subjective quality subscale (2.95). We assessed 20 applications with dashboard features using CDRS. Aarogya (Agra) Sarvam Setu and Odhisha COVID dashboard had the highest CDRS score (0.9), and the MP COVID Response App and WHO Academy COVID-19 had the lowest score (0.1).
We assessed the quality of all the mobile applications related to COVID-19 in India using a validated tool, i.e., MARS. Davalbhakta et al. evaluated the quality of COVID-19 mobile applications around the world using the MARS. However, they could assess only 16 mobile applications from India compared to 42 in the current study. Eleven of the 16 applications assessed in the previous study were evaluated in the present study. We could not access four applications: Covid locator, Kavach, test yourself Goa, and Covid 19 West Bengal [17]. Similarly, we excluded an application (Trackmatic) since it did not provide COVID-19 related information. The low number of included applications in the previous study could be due to the conduction of the study during the early stages of the COVID-19 pandemic. Similarly, Simin Salehinejad et al. assessed 13 applications across the world and excluded those not available in English and needed a local phone number. [18] WHOinfo was the only application common between our and the later study. Both studies reported almost similar MARS scores for WHOinfo. Although Alanzi T et al. (assessed Arogya Setu) and Islam MN et al. (assessed COVA Punjab) also reviewed the applications, their primary objective was to assess the functionalities, features, and design characteristics rather than quality [10,16].
We observed the app quality mean score and the app quality subjective score similar to the scores reported by Davalbhakta et al. We found a good level of agreement (ICC: 0.793) between our study and the Davalbhakta et al in the overall MARS score [17]. However, the individual subscale scores, i.e., engagement and functionality scores, are lower, and the aesthetic and information scores were higher in our study compared to Davalbhakta et al. The difference in subscale scores could be due to the varying number of applications used between the studies. The engagement score was low compared to other subscale scores in all the studies, indicating low or poor entertainment features, strategies to engage the audience, interactiveness and fit for the target audience. The low engagement score could be due to the non-commercial purpose of the applications, the novel nature of the disease and the availability of limited information, purely health-related and developed during an emergency in a limited time. Improving interactiveness through enabling reminders, sharing options, timely feedback, improving the aesthetics or graphics, providing more customizable options and improving the focus on the target audience could increase the engagement score.
We checked for twenty-eight features listed during our review of applications and found that COVA Punjab with most features and high overall MARS score. Overall .60) were present compared to applications without the respective feature. There have been instances of overlapping functions and non-comprehensibility between applications used in the same settings [16]. As a result, there is a need to upgrade or integrate the existing applications with new features to solve all COVID-19related difficulties that the maximum target population faces [23].
The assessed mobile applications using CDRS normalized score showed a gap in COVID-19 data reporting, similar to the result of a study conducted by Vasudevan V et al [18]. However, the latter study evaluated the data presented only on the websites of various Indian states, whereas we assessed the COVID-19 data presented in the mobile applications. Hence, there is a difference in the highest and lowest normalized scores observed for different Indian states between the studies. The COVID-19 data provided on the websites of Karnataka and UP had the highest and lowest normalized CDRS score. However, we observed that UP and Odisha had the highest normalized score in providing COVID-19 data on the mobile application. The contrasting difference in CDRS normalized score for UP could be due to the non-representation of the assessed mobile application for the whole state (applicable only to Agra). The difference was also observed between the website and the application of Odisha, as the CDRS score of the website was mediocre, whereas the application had an excellent CDRS score. Notably, the primary function of the Odisha covid dashboard (App) is to present the dashboard only, which could be the reason for the high CDRS score. We have also observed a higher CDRS score in the mobile application of Punjab compared to its website. It could be due to the absence of privacy issues in the mobile application which is present on their website.
GoI and various Indian states have developed many mobile applications to manage COVID-19 in the country.
Though the overall quality of the mobile applications is good, the engagement aspect of the mobile application quality needs improvement. Applications providing comprehensive COVID-19 related services are still lacking in the country. The necessity of the hour is to assess the user's perspective and the impact of application features on COVID-19 prevention and control, either individually or in groups.
Our study had the following limitations. We didn't have access to some of the mobile applications, which were restricted to specific populations like quarantined people or Government officials using specialized login access or had geographical restrictions. Though the current study included all available and accessible applications in India, the number of applications with a dashboard representing the daily COVID-19 data was low compared to the total number of states and Union Territories in India (n-36). Furthermore, we were unable to examine the actual user's perspective of the applications because they were produced by separate state governments, user inaccessibility, and assessment by a restricted number of reviewers. However, we did collect the user's perspectives indirectly in the form of ratings provided by them. Though it was mandatory for people to install and use mobile applications like Aarogya Setu during certain period of COVID-19 pandemic, the views of non-users and rural vs urban population irrespective of having a smart phone needs assessment. Future studies involving the various population subgroups (gender, age group, education, or digital literacy levels) for rating the quality of the applications from various Indian states is needed.

Conclusion
The Union and State governments of India have developed many mobile applications to manage COVID-19 in the country. Though the overall quality of the mobile applications is good according to the MARS tool, the engagement aspect of the mobile application and app subjective quality score still needs improvement. Mobile applications providing comprehensive COVID-19 related features are still lacking in the country. Absence of a listed feature in COVID-19 related information, and related updates and services in the mobile application impacted he MARS score. However, such absence of a feature did not impact on MARS score for COVID-19 risk assessment and surveillance. The necessity of the hour is to assess the user's perspective, addition of new features and the impact of application features on COVID-19 prevention and control, either individually or in groups.