Sensitivity and Specificity of the HIV Risk Assessment Tool Used by PEPFAR Partners in Edo, Bayelsa and Lagos States, Nigeria

,


Introduction
This operational research study was conducted under the USAID-funded Strategic HIV/AIDS Response Program (SHARP) TO2, which aimed to identify and support proven interventions for improving HIV/AIDS health service delivery and strengthening health systems with the Government of Nigeria (GoN) in Bayelsa, Edo and Lagos States. SHARP was implemented by Family Health International (FHI360) with the support of four partners: Achieving Health Nigeria Initiative (AHNi), Howard University Pharmaceutical and Continuing Education Center (HU-PACE), Abt Associates, and Khulisa Management Services.
Owing to multiple implementation challenges and a drive for more data-driven and targeted HIV/AIDS programming in Nigeria, SHARP identified a need for routine operational research to guide project implementation for effective outcomes. Khulisa the consortium partner responsible for conducting operational research conducted this study, which focused on determining the sensitivity and specificity of the Risk Assessment Tool (RAT), used in HIV testing and screening services, with specific emphasis on identifying the predictive power of specific questions This particular study aims to assess the Risk Assessment Tool (RAT) used by PEPFAR partners in Edo, Bayelsa, and Lagos States, in Nigeria to determine the tool's ability to predict HIV-positive clients as positive (sensitivity) as well as predict negative clients as negative (specificity). The study also provided recommendations for PEPFAR partners' HIV programs to improve their RAT.

Problem Statement
HIV testing is a critical screening and entry point for patients to access HIV treatment. With the pressure to implement Universal Testing and Treatment for reducing HIV new infections and AIDS-related deaths, HIV programs worldwide require practical and cost-effective screening and testing strategies for meeting the UNAIDS target of testing 95% of individuals living with HIV by 2030.
Robert Trevethan (2017) highlighted that there are arguably two kinds of tests used for assessing people's health: diagnostic tests and screening tests. Diagnostic tests are regarded as providing definitive information about the presence or absence of a target disease or condition. In our case, the diagnostic test is the actual HIV testing kit. Evans et al. (2005) explain that screening is broadly based and aimed at identifying those at high risk of a disease or condition. Considering the cost of HIV tests, identifying those at high risk has the advantage of helping streamline HIV diagnostic testing. In the context of HIV screening, screening tools are a set of questions administered to the clients to assess the level of risk their behavior and lifestyle could expose them to contracting HIV. The Association of Nurses in AIDS Care (2013) noted that HIV risk assessment also facilitates discussion of risk reduction behaviors to reduce HIV transmission and provides the opportunity for timely HIV antibody counseling and testing. Trevethan (2017) noted further that screening tests have some advantages over diagnostic tests such as, they have fewer demands on the healthcare system for their implementation and are more accessible, less invasive, less dangerous, less expensive, less time-consuming, and less physically and psychologically discomforting for clients. Trevethan (2017) stressed however that screening tests are well-known for being imperfect and often ambiguous. The author highlighted, therefore, the importance to determine the extent to which these tests are able to identify the likely presence (sensitivity) or absence (specificity) of a condition of interest so that their findings encourage appropriate decision-making.
Worldwide in the HIV programming context, the use of risk assessment screening tools is becoming more common practice among HIV programs, due to the presumed advantage of being an efficient and cost-effective strategy for predicting and prioritizing clients most at risk of testing positive. Most HIV programs use tools with questions that are tailored to their community and local context. However, the scarcity of literature on this topic shows and testifies to the very few tools that have been assessed for their sensitivity and specificity in predicting true positive and negative HIV patients. This was also noted by a recent study on screening in HIV testing in Malawi by Corrina Moucheraud et al, (2021).
To effectively inform and strengthen its HIV screening and testing activities in Nigeria, the SHARP TO2 project commissioned this study of the Risk Assessment Tool in use in public sector health facilities in 3 states of Nigeria by PEPFAR partners.

Study Design, Setting and Research Theoretical Framework
This study used a two-step approach theoretical framework: Step one: Comparing the RAT's screening results with the HIV test results to determine the tool's sensitivity and specificity.
Step two: Determining which RAT variables/questions are relevant and could contribute to increasing the sensitivity and specificity of the tool.
A cross-sectional study design was used to achieve the above two study's primary objectives. Quantitative data on a number of people screened and tested was analyzed using 2x2 contingency tables to determine the tool's sensitivity and specificity. Secondly, a penalized logistic regression model was used to identify questions on the RAT that are predictors of True Positive.
This study used data collected from 20 FHI360-supported clinics in Edo, Bayelsa, and Lagos .

Population, Sample Size Determination, Sample Technique, Inclusion Criteria and Data Collection
For the purpose of this study, a non-probability convenience sampling was used to select the 20 FHI360-supported health facilities and clients to include in this study. The SHARP TO2 M&E team collected data from facilities and clients that were easily accessible and had enough HIV testing kits. A total of 4623 client records were collected for

Assessment of tool questions link to the HIV test results
•Assessment of each variable relevance in increasing the tool sensitivity and specificity gjhs.ccsenet.org Global Journal of Health Science Vol. 14, No. 12;2022 HIV screening and testing services delivered over a three-month period. Edo accounted for 52% of all records (2389) while Lagos accounted 31% (1412) and Bayelsa 18% (822). Due to data quality issues (missing test or screening results, incorrect result using the wrong algorithm, etc.) only 3988 patient records were usable and included for predictive analysis.

Data Preparation and Analysis
The data were managed using Stata version 15. The master dataset was an excel spreadsheet with 4736 observations from the 20 selected health facilities. 51 observations were excluded because the patient's previous HIV test result was already known as positive prior to the RAT and 62 duplicates were also excluded. Thus, we considered a total of 4623 client records for this analysis.
The raw dataset was cleaned to assess the level of data quality (completeness and accuracy/precision etc.). This step was crucial in determining the final sample size. We established that the records had a lot of missing key data elements such as the HIV Risk Assessment results and/or the HIV test results. Out of the 4623 records, 60% (2775) had a RAT result, 86% (3992) had HIV test results, but only 50% had both. To increase the number of records that could be included in the analysis, we applied the RAT's internal algorithm to the records with missing RAT results, but also to all other records to cross-check the final results captured and to correct them if necessary. As shown in Table 1, this operation managed to increase the number of usable records to 99% (4578 records). For data analysis, we firstly generated 2 x 2 contingency tables based on the prediction of the RAT and the confirmed HIV test results ( Table 2). The values in this 2 x 2 contingency table were used to calculate sensitivity and specificity using the following equations:

Ethical Consideration
Considering patient-level data was used for this study, the study team applied and obtained the ethical clearance from the National Health Research Ethics Committee of Nigeria (NHREC) to ensure the study aligned with the accepted scientific principles and the Nigerian National Code for Health Research Ethics.
Patient names and personal identifying information (PII) were not used in the analysis. Using unique identifiers (Unique ID Number from the dataset) was sufficient for analysis and avoiding duplicating patient records. Data was shared from fieldworkers to the study core team through a secured and encrypted electronic data sharing platforms (email). Data cleaning and analysis was done on a password-protected computers and applications. All study team members accessing the data agreed to a confidentiality agreement, i.e. not to share, disclose or use the data for any other purpose than the study. Figure 1 presents the sample of records by gender and State.

Sensitivity and Specificity
The 2x2 contingency table (  These calculations indicate that the RAT used by FHI360 and its local partners had poor accuracy (sensitivity of only 54%), meaning the current RAT was able to correctly identify 54% of people who have HIV, but it also failed to identify 46% of people who have HIV.
The RAT's specificity (77%) indicates that the tool correctly identified 77% of people who do not have HIV, but it also erroneously identified 23% of people as having HIV when they did not.
These results are based on 3988 records or 86% of the total collected dataset records with the correct algorithm of RAT result strictly applied across all records.

RAT Questions Descriptive Analysis
Examining the specific questions in the RAT (Table 4) shows that clients who reported having unprotected sex in the last 6 months accounted for 51% of those who tested positive to HIV. Those who reported having vaginal or urethral discharge accounted for 11% and those who were diagnosed with tuberculosis or had tuberculosis symptoms accounted for 8% of the clients who tested positive to HIV. These three (3) questions yielded the highest predictive values of clients who were likely to test positive.

RAT Questions Inferential Statistics/Predictive Analysis
The penalized logistic regression model was used to identify questions on the RAT that are predictors of True Positive. A univariate analysis was conducted to identify risks factors associated with testing positive for HIV.  Vol. 14, No. 12;2022 All variables with a p-values <= 0.2 were included in the regression model. Painful swelling and having a sexual partner who has had chronic ill health or died in the 5 past years had a p-value > 0.2 and were excluded from the multivariate model. Genital ulcers had a p-value of 0.160 but showed no difference between those who had ulcers and those who did not have ulcers and was excluded as well.
The findings of this model were based on 1,157 observations with complete values for sex, unprotected sex, genital discharge, tuberculosis and severe illness. Table 5 below presents the predictive model results. The univariate analysis shows that being a female or having a "yes" response for unprotected sex, genital discharge, TB, or severe illness increase the odds of testing positive. However, after adjusting for every other question, unprotected sex and genital discharge greatly increases the odds (three to four-fold) of testing positive for HIV (3.19 and 4.09 respectively). The predictive analysis, therefore, confirms that questions related to unprotected sex, genital discharge, Tuberculosis, and severe illness/hospitalization were the only questions that had some predictive power. However, it should be noted that, only questions on unprotected sex and genital discharge are strong predictors of True Positive HIV results.

Discussion
Although very few studies are conducted to assess the specificity and sensitivity of HIV Risk Assessment tools, the results are similar to two other studies. A recent study of HIV screening in Malawi (Corrina Moucheraud et al, 2021), used exit data (n = 1038) collected at outpatient departments to estimate the sensitivity, specificity, and negative and positive predictive values of screening tools. The authors compared a full tool (seven relevant questions) to a reduced tool (five questions, excluding sexual behavior measures) and to the standard of care (two questions, never tested for HIV or tested > 12 months ago, or seeking care for suspected STI). The full tool achieved 55.6% sensitivity and 84.9% specificity; the reduced tool achieved 59.3% sensitivity and 68.5% specificity while the standard of care two-questions tool achieved 77.8% sensitivity and 47.8% specificity. All these tools had poor accuracy.
A similar study by Tsitsi Bandason et al (2018) assessed the specificity and sensitivity of an HIV screening tool to identify adolescents living with HIV in a community setting in Zimbabwe. This was a community-based HIV prevalence survey conducted among youth aged 8-17 years residing in 7 communities. Participants without a previous diagnosis of HIV were evaluated for the probability of having HIV using an HIV screening tool developed by the researchers. The authors' 4-item screening tool had poor accuracy, its sensitivity was 56.3% (95% CI:44.0-68.1%), and specificity of 75.1% (95% CI:73.9-76.3%).
Both the Zimbabwe and Malawi studies corroborate the findings of this study which established that the risk assessment tools used in Nigeria's HIV programs have poor accuracy. The HIV Risk Assessment Tools' poor accuracy in all these studies is the direct result of the inclusion of many questions that have low predictive value for HIV. While J. J. Ong et al (2022) reported that several recent studies suggest that there may be value in risk-based tools to improve testing efficiency (i.e. identifying those who need to be tested) they also warned on the same limitation we are highlighting here that, there have not been any systematic reviews of the literature to synthesize these studies. The authors also noted that screening tools may be helpful in settings where it is not feasible or recommended to offer testing routinely but caution in the need for screening tools, where there is a trade-off between reducing costs of testing with missing cases of people living with HIV.
On another note, clients who reported having unprotected sex in the last 6 months held the highest prediction of HIV positivity (51%). Several other studies have linked unsafe sexual behaviors as one of the top factors associated with high-risk of contracting HIV. Although it is well documented that the Risk of HIV transmission was greatest for blood transfusion, followed by vertical exposure, sexual exposures, and other parenteral exposures (Patel et al 2014;CDC (2022), most studies report that sexual activity continues to be the primary route of HIV transmission worldwide WHO (2022), Adedimeji (2016), Ping Du et al (2016), and Cohen (2007). Understanding the highest risk behaviors of the population of interest is very important for streamlining tools questions, inform the tool design and allow to increase the tool predictive power.

Conclusions
HIV Risk Assessment Tools are often insufficient to predict clients who are at a high risk of testing positive for HIV. It is important to assess the sensitivity and specificity of these Risk Assessment Tools to ensure that they incorporate questions with the highest predictive value to streamline the testing of true positive patients and reduce the risk of false negative patients prior. Unfortunately, relevant studies have supported that HIV Risk Assessment Tools used by HIV programs often have poor accuracy. This is due to poor tools design practices which often lack methods to streamline the tool to only include highly predictable questions that are specific to the population of interest.

1) During tools design, it is a good practice to include interviews with people who tested positive for HIV.
To make questions highly predictable, there is no better audience than people who tested positive. Ask them about the risk behaviors that they believe led them to becoming HIV positive and build your tool's questions based on their answers. This will allow a streamlining of tool questions to incorporate those questions that have the highest predictive values. Use the data collected from the interview exercise to generate HIV risk questions that are tailored to that specific community.
2) Tools should be specific to each community. Given that many public health conditions are linked to risky community behaviors, the tool needs to capture community-level specific risky behaviors. For example, the LGBTQI community, polygamist communities, swingers' communities, and people using injectable drugs each have very different risky behaviors that need to specifically captured to increase the tool's predictive power.
3) Design a user-friendly tool that will reduce data collection errors: a. For example: rather than two columns for gender (e.g. one column for male and another for female with answers Yes and No in each column), use a single column with dropdown options for that variable (e.g. male and female; Yes and No; or Positive and Negative options).
b. Avoid paper data collection. Excel allows automation but where possible, use a platform that allows better design, automation, control, and submission of the data.
c. Automate answers and make compulsory blocks: This will avoid incorrect capturing and missing data. For example: automating the tool's backend algorithm will automatically insert the correct final RAT result based on responses to questions, instead of leaving it open to the data collector to complete, leading to errors. 4) Lastly but very important, pilot the tool prior to implementing it and test its sensitivity and specificity to ensure a high predictive power.