Abstract
The COBATEST Network of Community-Based Voluntary Counselling and STI/HIV Testing (CBVCT) services was created to standardise monitoring and evaluation of CBVCT services across Europe. This study aims to assess the quality of data collected in the network from 2015 to 2016. A survey was completed by 34 COBATEST Network members and an evaluation was performed of data quality based on three dimensions: transcription validity, completeness and consistency. The weakest area that we identified was data management processes. Only 8.8% of services had a written procedure to address data quality errors, 29.4% had procedures in place to resolve discrepancies and 35.3% performed quality control. We found that 41.2% of services utilised the COBATEST data, 11.8% made decisions based on the COBATEST data and 61.8% analysed their data in an independent manner for internal purposes. We conclude that while services have reliable data to support planning and management of services, improvements to quality procedures would ensure data are translated into evidence. This evidence would support further expansion of CBVCT services in the EU/EEA, including the integration of CBVCT-generated data into national surveillance systems.
Introduction
In recent decades, different modalities of HIV testing services have been developed to reach most-at-risk populations. Among these approaches, community-based voluntary counselling and testing (CBVCT) services have been recognised as a model that promotes early HIV diagnosis among key populations. These services, when coupled with effective linkage to care, play an essential role in improving outcomes in the HIV care continuum. 1 In 2009, the COBATEST Network of CBVCT services was created with the purpose of sharing procedures to monitor the activity of CBVCT services across Europe to recognise the value of testing in this setting and ultimately promote HIV testing and counselling, early diagnosis and care for hard-to reach groups.2,3 This network was created in the context of the HIV-COBATEST project (HIV Community-based testing practices in Europe 2010–2014), and by 2016 included 41 CBVCT services from 20 different European countries. 4
One of the main objectives of the network is the monitoring and evaluation (M&E) of HIV testing activities performed in the participating CBVCT services. To do this, a standardised protocol, including a core set of indicators to monitor HIV testing activities, was defined and a standardised data collection form and a COBATEST web-based data entry system were created to collect and analyse the data. For participating CBVCT services unable to use the web-based data entry system, an alternative data collection process was created whereby services could submit a minimum set of data or aggregated CBVCT core indicators. The COBATEST Network data have proven to be of strategic value to evidence the need for community-based service delivery models as an integral part of the HIV strategic investments, and in supporting quality services along the HIV care cascade. 4 This strategic information also informs a deeper understanding of the context of the epidemic, by describing the vulnerabilities that certain communities or sub-groups face and the risks to which these populations are exposed.5,6
Given the importance of the information generated by the CBVCT services and its potential utility, it is paramount that the data accurately reflect what is being monitored and evaluated, and is timely and relevant. High quality data are critical for decision-making processes and for the accurate assessment of the impact of CBVCT services in order to maximise their effectiveness, responsiveness and cost-effectiveness. With this objective, the COBATEST Network is continuously striving to improve the quality of the reported data.
Demonstrating that data collected by the COBATEST Network are reliable may significantly influence how member organisations respond to data and how local, national and European stakeholders perceive and value the strategic information collected by the CBVCT services. Confidence in the data would promote its integration into the formal surveillance system at national and regional levels to contribute to comprehensive monitoring of testing interventions. 5 To this end, this study aims to perform a quality assessment of the data collected by the COBATEST Network from 2015 to 2016 and to assess to what degree the CBVCT services’ data were integrated into the national surveillance data.
Methods
Study design and population
A descriptive observational study was undertaken, including an assessment of the functional components of CBVCT service data management based on a survey of all COBATEST members and a descriptive analysis of the quality of the COBATEST Network data based on three dimensions. Ethical approval was obtained from the Hospital Germans Trias i Pujol healthcare ethics committee.
Assessment of the functional components of CBVCT services, data management
Each CBVCT service’s manager completed a questionnaire, which was a piloted, structured ad hoc instrument hosted by the Survey Monkey website. All the CBVCT services that were partners of the COBATEST Network in 2017 were invited to complete the online survey. The survey was sent in Spanish to Spanish members and in English to all other members (online Annex 1). Invitation emails were sent out on 26 April 2017, two interim email reminders were sent and the survey was closed on 3 July 2017. The questionnaire included questions related to the main administrative and organisational CBVCT services’ characteristics (typology, funding, number of people working at the CBVCT service including volunteers, settings where the CBVCT service activities are implemented, key population targeted by the CBVCT service, type of HIV tests used, place where the confirmation of a HIV-positive test is performed, type of referral procedure for HIV diagnoses), as well as questions to capture six functional areas of a data management and reporting system:(1) M&E capabilities, roles and responsibilities, (2) Indicator definitions and reporting guidelines, (3) Data collection and reporting tools and forms, (4) Data management processes, (5) Links with the national reporting system and (6) Data use.
Assessment of quality of data based on three dimensions
All the data collected by the COBATEST Network between 1 January 2015 and 31 December 2016 were evaluated, encompassing data collected through three possible modalities: (a) Web-based data entry system that uses a standardised questionnaire and has a centralised database. This system uses a unique client identification code that ensured the anonymity of the client. (b) Disaggregated anonymised data submitted by members that use their own data entry system. (c) Aggregated data (indicators).
Data quality was measured using three dimensions:
Data transcription validity was calculated by dividing the number of records deemed inaccurate by the number of non-empty records that should be reported according to each variable. Records were considered inaccurate if data did not conform to the syntax (format, type, range) of its definition.
Data completeness was calculated by dividing the number of incomplete records by the number of records that should be reported according to each variable. Fields were considered incomplete if left blank and considered complete if there was a value reported.
Data consistency was evaluated by grouping together variables that were dependent on each other and dividing the number of inconsistent records in each group by the total number of records.
Statistical analysis
Three tables are presented to describe the results of the survey. The first summarises CBVCT services in the COBATEST Network (2015–2016) by geo-distribution, data submission mode and format, indicating the frequency for each. The second presents administrative and organisational characteristics and procedures in all responding CBVCT services with the frequency and proportion of respondents for each category. The third table summarises the functional areas of a data management and reporting systems, with functional components categorised into six areas and the frequency and percentage of respondents reported for each component (yes completely, partly, no not at all, not applicable). The percentages sum to 100 for each component in the case that all 34 services answer.
Each functional component of the data management was scored from 1 (No, not at all), 2 (Partly) or 3 (Yes, completely). Responses coded ‘Not Applicable,’ were not factored into the score. An average for all individual questions in each of the six functional areas was obtained to give six scores with a maximum of three points. The scores allow comparison across functional areas as a means of prioritising system-strengthening activities. The scores are comparable to each other and are most meaningful when comparing the performance of one functional area to another. A spider-graph presents the results of this scoring and the graph is used to prioritise areas for improvement.
The assessment based on the analysis of three data quality dimensions was performed on all data submitted to the COBATEST Network during the study period for the total of the CBVCT services, then by year and by selected variables. In order to compare between CBVCT services and the entire COBATEST dataset, two indices were constructed from the weighted average of the proportion of correct transcriptions and completeness calculated for each variable. The indices were measured first for all COBATEST variables and then for the core set of variables to monitor HIV testing at the European level as defined in the document. 5 The weights were defined according to the degree of importance of the variable. Data analysis was performed using Stata13 College Station, TX, StataCorp LP and SAS® (SAS Institute, 2011).
Results
Assessment of the functional components of CBVCT services, data management
Table 1 shows the number of CBVCT services participating in the network, their geo-distribution and the method used to send data. Thirty-four of 39 CBVCT services responded to the survey, giving a response rate of 87.2%. Twenty-four of 34 respondents (70.6%) sent data through the data entry tool, two (5.9%) sent disaggregated data and five (14.7%) aggregated data (COBATEST indicators). Three CBVCT services (9.0%) reported not being able to send data for 2016.
Community-based voluntary counselling and testing services by geolocalisation, data submission mode and format, COBATEST Network 2015–2016.
aServices who do not use the COBATEST web-based data entry tool, enter data in disaggregated format according to the COBATEST requirements in an Excel file via email or aggregated data prepared according to the COBATEST indicators and submitted in Excel format via email.
CBVCT: community-based voluntary counselling and testing.
Table 2 shows the assessment of the functional components of CBVCT services’ data management and the administrative and organisational characteristics of the participating COBATEST Network services in 2017. Overall, most of the CBVCT services were based in NGOs (82.4%), funded by a public/governmental organisation (41.2%) or co-funded by public and/or private entities (47.1%). The primary key population in the COBATEST Network were men who have sex with men (MSM) (79.4%), followed by male sex workers (55.9%) and the transsexual/transgender population (55.9%).
Administrative and organisational characteristics and procedures in CBVCT services, COBATEST Network, 2017.
NGO, non-governmental organisation; MSM, men who have sex with men; FSW, female sex workers; MSW, male sex workers; PWID, people who inject drugs.
Table 3 shows the functional components of data management and data quality from the COBATEST Network members, 2017. When the M&E capabilities, roles and responsibilities were self-assessed in the survey, half (52.9%) of CBVCT services reported having a documented organisational structure that identifies roles for data management. More than half (61.8%) of the services have training plans on data management for both staff and volunteers, 55.9% of the services have someone responsible for reviewing the quality of data and 41.2% have designated staff responsible for reviewing the quality of aggregated numbers or the digitalisation of data prior to their submission. Services that send disaggregated data have no one designated to review the data quality and oversee the submission of data to the COBATEST Network (online Annex 2). For services sending aggregated data, four had a documented organisational structure, with designated people for reviewing data quality, and three had someone responsible for reviewing aggregated numbers prior to submission (online Annex 2).
Functional components data management and data quality from the CBVCT services of the COBATEST Network, 2017.
aM&E: monitoring and evaluation.
b34 respondents, proportions given as % of 34.
Overall, 82.4% (n = 28) of the services reported having instructions on how to complete the data collection and reporting forms, and 76.5% (n = 26) use standardised reporting forms/tools. The data management process is the weakest functional area, as just 8.8% (n = 3) of services in the COBATEST Network had a written procedure to address incomplete reports, 35.3% (n = 12) performed quality control after digitalisation of data and 29.4% (n = 10) had an established procedure to resolve any data discrepancies or inconsistencies. Most sites use paper questionnaires, which double the reporting burden, as data have to be transcribed into the electronic systems.
From the surveyed CBVCT services, 16 services (47.1%) reported data to the national information system or national authority. It is important to bear in mind that 11 centres also belonged to the ‘DEVO Network,’ a network founded in Catalonia-Spain with the help of the Regional Surveillance System, which has been collecting centralised data since 1994. 7 All CBVCT services, which send disaggregated data and three services that were sending aggregated data, also submitted this data to their national surveillance systems. Of all CBVCT services, 61.8% (n = 21) reported that the service analyses the collected data independently of the network for internal purposes, 41.2% (n = 14) used the information generated by the network and 11.8% (n = 4) made decisions based on the collected data. Overall, the COBATEST Network assessment of data management and reporting systems showed broadly low scores across the six assessed functional areas (Figure 1).
Score for each one of the functional components of data management and data quality for the overall COBATEST Network Assessment, 2017.
Assessment of quality of data based on three dimensions
The COBATEST Network received data on a total of 95,493 clients who were tested for HIV in the participating CBVCT services/networks in 2015 and on 72,916 clients in 2016. For the individual CBVCT services/networks the number of clients varied from 8 to 43,097 in 2015 and from 7 to 38,658 in 2016.
Analysing all COBATEST services together (n = 39) by year, data completeness remained stable in the total set of selected COBATEST variables, whereas among the core variables decreased slightly from 99% to 97%, from 2015 to 2016 (Figure 2). Transcription remained stable year on year in the core variables, and decreased in the whole set of variables, from 99.6% to 95.8%, from 2015 to 2016. Consistency was only evaluated for the total number of variables, had the lowest average (93.4%) of the three dimensions and did not change year on year. When the assessment was performed for each variable (Table 3), the completeness of the core variables remained stable over the study period. However, important variables such as city of the CBVCT site and date of receiving screening test result had less than 90% completeness. Regarding CBVCT core variables, those used to build the definition of migrants (year of arrival to the country of CBVCT and country of origin) as well as history of drug injection were incomplete. Important variables such as ‘client received the screening HIV test result’ and ‘date of receiving screening test result,’ had a completeness rate of around 85%. This low percentage of completeness could be explained by the fact that most of the COBATEST CBVCT services use a rapid test, which means that clients received the result during their visit. The CBVCT service name and unique identifier are variables that both show low transcription quality, as many sites manually enter their names, rather than use a drop-down list (Table 4). Comparison by mode of data submission has been included in online Annex 3. These tables show that when the data entry tool is used, data quality improved.
Completeness, transcription and consistency by selected core variables for the COBATEST Network and for the basic variables likely to be included in the core set of variables to monitor HIV testing at the European level, 2015–2016.
Data quality index score for completeness and transcription of selected COBATEST core variables in CBVCT services using the website data entry tool, by year.
Proposed core variables to monitor HIV testing at the European level.
CBVCT, community-based voluntary counselling and testing; MSM, men who have sex with men; PWID, people who inject drugs.
Discussion
This assessment has identified a number of weaknesses in the data quality of the COBATEST Network. Many services do not designate M&E roles and responsibilities among the CBVCT services’ staff, and demonstrate little evidence of clear written procedures to address any quality errors, discrepancies or inconsistencies found in the datasets. Aside from CBVCT services which form part of the DEVO Network, services do not have clear guidance on sending their data to the national/regional surveillance systems. However, in terms of transcription validity; completeness and consistency, COBATEST Network data quality is high, especially if the web-based tool is used. Therefore, the findings from this evaluation indicate that CBVCT services have reliable data to support quality planning and management of the services. In order to produce quality data that translate into evidence to support further expansion of CBVCT service in the EU/EEA, CBVCT services should improve quality procedures, which would pave the way for CBVCT-generated data to be integrated into national surveillance systems.
Findings from this assessment have also demonstrated that among the CBVCT services, dedicating human resources to maintain data quality is a considerable challenge. This is in line with the findings of similar studies that have assessed data quality in healthcare services including community settings, that found that a workforce already burdened by service provision might give less thought to the quality of data.8,9 Therefore, CBVCT services require approaches that minimise and streamline data collection tasks, offer centralisation of data for long-term sustainability, are low-resource and take into account the high variability of personnel in the CBVCT services. These strategies should include centralisation of training and mentoring initiatives, development of Standard Operating Procedures, which clarify staffing needs and processes for the generation and use of strategic information as well as for the set-up of standardised quality assurance processes in CBVCT services.
Strengthening electronic health information systems and harmonising data collection systems/methods may also contribute to improving data quality. In the case of CBVCT services, a web-based data entry tool has the potential to dramatically reduce the data collection burden by automating data aggregation and reporting, therefore improving data quality. 10 In the context of the COBATEST Network it would be advisable for all CBVCT services to use the data entry tool as this will reduce error, data collection burden and allow for real-time access to data. The findings indicate that the existing platform should be improved and adapted to services’ needs. This assessment indicates that the platform’s potential has not yet been reached as the platform produces some transcription errors and creates a double reporting burden for services using paper questionnaires.
An additional, relatively high-resource investment suggested by the current assessment is the harmonisation of data collection in all the CBVCT services. This would include the development of essential data sets, which to be effective requires a timely, intensive consultative process. Such data sets would include a selection of key variables, included in the core set of variables to monitor HIV testing at the European level. At the same time, this selection could serve the data needs of the CBVCT services to avoid double reporting and a double burden of work. Streamlining of indicators and the selection of a core set of variables to monitor HIV testing at the European level is needed. Therefore, the COBATEST Network is working on creating a process to select key variables, serving both the purposes of supporting service monitoring and quality improvement process and contributing to European-level monitoring, while reducing the data management burden for the CBVCT sites.
Finally, excluding the centres participating in the DEVO Network (Catalonia, Spain), the linkage of CBVCT data with the local/regional/national surveillance HIV systems is very limited. Some CBVCT sites describe engaging in preliminary attempts to integrate their data but with no defined processes. Therefore, this assessment identifies a key task for the COBATEST Network; consolidating and expanding the network as an important source of strategic information for governments and connecting to the national HIV surveillance systems.
One limitation of this assessment was restricting it to CBVCT services participating in the COBATEST Network. Despite the lack of representativeness of the COBATEST Network in Europe, we believe that this network serves as a sentinel source of HIV testing information and provides a unique EU-level perspective. As participants were fully aware of the purpose of the assessment, they may have exhibited social desirability bias, answering questions with what they know to be the correct policies concerning data management and reporting, rather than explaining the current procedures at their site. Despite the difficulties in assessing a large quantity of data from multiple sources, the evaluation exercise succeeded in making a data quality analysis using two data quality assessment methods.
In conclusion, this study reports the first attempt to assess the quality of routine HIV testing data in community settings in the European region. We identified several gaps related to the functional components of CBVCT services’ data management and data quality; however, the findings showed that the majority of CBVCT services are collecting and using their data effectively. The COBATEST data could contribute to increasing the evidence community-based service delivery models as an integral part of the HIV strategic investments, and could be used as an important source of information to support high-quality services along the HIV care cascade and as important information to be collected by the data surveillance systems.
COBATEST Network
IP: Jordi Casabona (Centre d’Estudis Epidemiològics sobre les Infeccions de Transmissió Sexual i Sida de Catalunya: [CEEISCAT]-CIBERESP), Coordinator: Laura Fernandez (CEEISCAT), Field coordinator: Anna Conway, CBVCT services: Isabell Eibl (AIDS Hilfe Wien), Tomáš Čech, Ivo Procházka and Robert Hejzák (Czech AIDS Help Society, Prague), Zoran Dominković and Kristina Sekulic (ISKORAK), Francois Pichon (AIDS Fondet), Lionel Fugon (AIDES), Lella Cosmaro and Sabrina Penon (Fondazione LILA Milano ONLUS), Inga Upmace (Baltic HIV Association), Loreta Stoniene (Demetra), Iwona Wawer (CBVCT centres Poland), Daniel Simoes (Checkpoint LX, IN-Mouraria and MOVE-Se), Mitja Ćosić (Legebitra), Pamela Biot, Manuel Gomez (ADHARA), Esteban Brook-Hart (AVACOS-H), Juan Ramón Barrios and Aitor Calvo (OMSIDA), Pere Salmerón (Lambda), Eva Ma Prado Cuervo (ACCAS), Sara Solier, Elena Luque (IEMAKAIE), Alexandra Pérez (CAS Gibraltar), Gema Herrero (GADES), Juan José Reyes, Sonia Reyes and Rocío García (SILOÉ), Marcela Macheras (ACAVIH), Manu de Gregorio (CASDA), Maria José Oltra (CIBE Marítim), Loly Fernández (Comité Anti-Sida Asturias (CCASiPA), Adriana Morales and Luís Villegas (STOP-SIDA), Jaime Quezadas (ACASC), Jordi Baroja (CJAS), Joaquim Roqueta and William Mejías (Gais Positius), Anna Rafel (Associació Anti-SIDA de Lleida), Julia Collignon and Elena Griñán (Actuavallès),Mercè Meroño (Àmbit Prevenció), Lluís Romero and Toni Aguilar (AssexoraTgn), Anna Lara (ACAS Girona), Patrícia Colomera, Jessica Camí, Lorena Andreo and Elena Adán (SAPS-Creu Roja), Araceli Otón (Creu Roja Tarragona), Andrii Chernyshev (Alliance Global), Paqui Cantudo (Concordia Marbella).
Footnotes
Acknowledgements
We would like to acknowledge the ECDC for their support, especially Andrew Amato for his comments. We also extend our appreciation to the CBVCT services involved in the study, to all the CBVCT services which are part of the COBATEST Network and especially the service managers who agreed to participate in the survey.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
This work was supported by European Centre for Disease Prevention and Control (ECDC): [Service contract: ECD.6543]. The views expressed in this document do not necessarily reflect those of ECDC.