Validation of the STRATIFY Falls Risk Assessment Tool in a Japanese Acute Care Hospital Setting

Patient falls are the most frequent adverse events that occur in a hospital. Prevention of inpatient falls is performed by a strategy to target patients at high risk for falls determined by a falls risk assessment system such as the STRATIFY tool. However, the performance of the STRATIFY tool in a Japanese hospital setting has not been determined. We tried to verify the performance of the STRATIFY tool for predicting falls in acutely hospitalized patients in Japan by a multi-center study. A total of 113,413 patients admitted to four acute cares national university hospitals during the period from April 2010 to March 2012 were studied. Inpatient falls per 1,000 patient-days varied from 1.42 to 2.92 in the four hospitals. The STRATIFY score was calculated on the basis of data extracted electronically from the hospital information system. Although the distribution of STRATIFY scores differed significantly among the four hospitals, logistic regression analysis and survival analysis showed that the proportion of high-risk patients who fell was significantly larger than the proportion of low-risk patients in all of the four hospitals. The odds ratio and hazard ratio for high-risk patients versus low-risk patients were 2.5 to 4.3 (combined estimate, 3.9 (95% confidence interval (95% CI), 2.1 to 7.6) and 1.8 to 5.1 (combined estimate, 3.1 (95% CI, 2.1 to 4.6)), respectively. The results suggest that the STRATIFY tool can be used as a screening tool to detect patients at high risk for falls in a Japanese acute care setting as used commonly in other countries.


Introduction
Patient falls are the most frequent adverse events that occur in a hospital.About 3% to 10% of patients who experience falls sustain severe injuries such as bone fractures and intracranial hemorrhage (Toyabe, 2010(Toyabe, , 2012a(Toyabe, , 2014)).Inpatient falls are associated with additional healthcare costs, prolonged length of hospital stay and psychological distress for the patients.This situation might result in complaints and litigation from the patients and their families (Oliver, Killick, Even, & Willmott, 2008a).Prevention of inpatient falls is performed by a strategy to target patients at high risk for falls determined by a falls risk assessment system (Gates, Fisher, & Cooke, 2008).However, there has been no standard risk assessment tool for falls in Japanese hospitals (Izumi, Makimoto, Kato, & Hiramatsu, 2002).In Japan, each hospital uses its own fall risk assessment system specific for the hospital, and different hospitals use different risk assessment tools for falls.Moreover, several risk assessment tools are used in some hospitals to obtain better performance by reflecting patient characteristics and the environment of the ward.The lack of a standard risk assessment tool for falls is an obstacle for a multi-center study on inpatient falls.Adjustment of fall risks between hospitals is difficult under the condition in which each hospital uses its own risk assessment tool.Risk adjustment by using a standard risk assessment tool is necessary for a multi-center study.
There are various tools for assessing the risk of falls including the Downton scale (Downton, 1993), the Morse Fall Scale (Morse, Morse, & Tylko, 1988), the St. Thomas Risk Assessment Tool in falling elderly (STRATIFY) (Oliver, Britton, & Seed, 1997), and the Hendrick II Fall Risk Model (Hendrich, Bender, & Nyhuis, 2003).Some of these tools were verified in environments other than those for which they were developed.The STRATIFY tool was the most frequently used tool in previous studies on inpatient falls, and the STRATIFY tool showed greater diagnostic validity than that of other tools as proved by systematic reviews and meta-analyses (Aranda-Gallardo, Enriquez de Luna-Rodriguez, Canca-Sanchez, Moya-Suarez, & Morales-Asencio, 2015).The performance of assessment tools for fall risk varies considerably depending on the patient population and the environment (Oliver et al., 2008b), and their performance should be tested prior to implementation (Aranda-Gallardo et al., 2013).Since there have been few reports on the use of the STARTIFY tool in a Japanese acute care setting (Toyabe, 2010), we tried to verify the performance of the STRATIFY tool for assessment of fall risk in acutely hospitalized patients in Japan by a multi-center study.

Setting and Patients
This study was designed as a multi-center, retrospective observational study at four acute care national university hospitals with 613 to 1,035 beds.All of the patients who had been admitted to each hospital during the period of April 2010 to March 2012 and who were aged from 40 to 90 years at admission were studied.A total of 113,413 patients (1,862,271 patient-days) were admitted to the four hospitals during that period.

Data Collection
Each hospital has a hospital information system (HIS) that contains electronic medical records (EMR) and an online incident reporting system.Information on patients' background including age, gender, cognitive status, major diagnostic category of the patient's principal diagnosis, admission ward, admission day and discharge day was obtained from the HIS of each hospital.Information on falls recorded by incident reports was obtained from the online incident reporting system.We used the definition of falls by the World Health Organization, that is, any unintended descent to the ground, floor, or other lower level (Kalache, Fu, & Yoshida, 2007).Incident reports contained information on the degree of injury of the patient and type of event, essential information on the event such as the name of the patient, the name of the medical staff involved, and the exact time and place that the event occurred, and detailed information on the course of the incident, action taken by medical staff and outcome of the event.

Risk Assessment Scores
Each hospital used its own risk assessment tool for inpatient falls, and none of the hospitals used the STRATIFY score.We therefore recomputed the STRATIFY score based on data extracted electronically from the HIS of each hospital.The STRATIFY score is based on five items, including history of falls, agitated confusion, visual impairment, urinary frequency and high transfer/mobility score using the Barthel scoring system (Oliver, Britton, & Seed, 1997).Each item is scored with a "yes" or "no", with "yes" being scored as 1 and "no" being scored as 0. Data concerning the five items on patients' admission were used for recomputing the STRATIFY score.The score varies from 0 to 4, and a score of 2 or more was considered as a high risk for falls in the original study.We confirmed that a score of 2 or more could be used as a cutoff point also in a Japanese acute care hospital setting from the results of a survey conducted in a single institute (Toyabe, 2010).

Statistical Analyses
The rate of inpatient falls that occurred in each hospital was expressed by the number of falls per 1,000 patient days, and the rates were compared by using the chi-square test with Ryan's method for multiple comparisons.Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and odds ratio (OR) to detect inpatient falls were calculated with 95% confidence intervals.The ORs that were calculated for the four hospitals were combined by using a bivariate random effects model (Borenstein, Hedges, & Rothstein, 2009).A test for heterogeneity was performed to compare the ORs among the four hospitals.The occurrence of falls over time was also analyzed by survival analysis in which the time between admission and the fall events during the hospital stay was considered as survival time, since the length of stay in acute care hospitals in Japan is very long compared with that in other countries (OECD).Discharge of the patient without fall events was considered as censoring.The Kaplan-Meier method was used for analysis, and the log-rank test was used to examine whether there was a difference in the occurrence of falls over time among the four hospitals or between patients at high risk and patients at low risk patients for falls.Cox's proportional hazard model was used to calculate the hazard ratio (HR) for the high-risk group versus the low-risk group for falls that were evaluated by the STRATIFY score.The HRs calculated for the four hospitals were combined by using a bivariate random effects model (Borenstein, Hedges, & Rothstein, 2009).A test for heterogeneity was also performed to compare the HRs among the four hospitals.All statistical analyses were performed using IBM SPSS Statistics 23 (SPSS Japan Inc., Tokyo, Japan) and R package version 3.2.4,and a p-value less than 0.05 was considered significant.

Ethical Consideration
All data were analyzed anonymously.The Ethics Committee of each hospital or university gave ethical approval for the study.

Comparison of the Rates of Falls among Hospitals
The rate of falls differed significantly among the four hospitals (p<0.001, Figure 1), and multiple comparisons between each pair of hospitals indicated that there were significant differences between the four hospitals (p<0.001,respectively).Hospital A showed the minimum fall rate of 1.42 per 1,000 patient days, and Hospital B showed the maximum fall rate of 2.92 per 1,000 patient days.There was almost a two-fold difference between Hospital A and Hospital B. The rate of falls over time in each hospital was estimated by the Kaplan-Meier method, and the rates of falls over time were compared among the four hospitals by using the log-rank test.There was again a significant difference in the rate of falls among the four hospitals (p<0.001, Figure 2).Multiple comparisons between each pair of hospitals indicated that there were significant differences between the four hospitals except for the difference between Hospital A and Hospital C. The rate of falls per 1,000 patient-days in each hospital is shown.There was a significant difference in the rate of falls among the four hospitals (p<0.001), and multiple comparisons between each pair of hospitals indicated that there were significant differences between the hospitals (p<0.001,respectively).

Figure 2. Comparison of the rates of falls over time among the hospitals
The rate of falls over time estimated by the Kaplan-Meier method showed a significant difference among the four hospitals (p<0.001)except for difference between Hospital A and Hospital C.

Comparison of Fall Risk Stratification among the Hospitals
The STRTIFY score was recomputed using data on patients' admission that were extracted from the HIS of each hospital.The risk stratification for falls differed significantly among the four hospitals (p<0.001), and the results of post-hoc multiple comparisons showed that there were significant differences among the four hospitals.When a patient with a STRATIFY score of 2 or more was considered as a patient at high risk for falls, the proportion of patients at high risk for falls also differed significantly among the four hospitals (p<0.001, Figure 3).The results of multiple comparisons showed that there were significant differences among the four hospitals (p<0.001,respectively).The proportion of patients at high risk for falls was lowest in Hospital B (13.1%) and was highest in Hospital C (36.4%).There was an almost three-fold difference between Hospital B and Hospital C in the proportion of patients at high risk for falls.The STRATIFY score was used to evaluate patients' risk for falls, and the score was recomputed using data on patients' admission.The proportion of patients with each score in each hospital is shown.The proportion of patients stratified by the STRATIFY score differed significantly among the hospitals (p<0.001), and multiple comparisons between each pair of hospitals indicated that there were significant differences between the hospitals (p<0.001,respectively).The values in the graph indicate that the proportion of high-risk patients with a STRATIFY score equal of 2 or more.

Relationship between Risk for Falls and Rate of Falls in Each Hospital
The performance of the STRATIFY score when the score was applied to each hospital was validated, and the results of sensitivity, specificity, PPV, NPV and their 95% confidence intervals are shown in Table 1.The ORs ranged from 2.5 to 8.0 among the four hospitals, and the ORs were combined into a summary estimate of 3.9 (95% confidence interval (CI), 2.1 to 7.1) (Table 2).There was a significant difference in the ORs among the four hospitals by a test for heterogeneity (p<0.001).Kaplan-Meier plots for falls over time in the four hospitals are shown in Figure 4.There were again significant differences in the rate of falls over time between the high-risk and low-risk groups in the four hospitals (p<0.001,respectively).Cox's proportional hazard model was used to calculate HR of high-risk group versus low-risk group for falls (Table 2).The HRs ranged from 1.8 to 5.1 among the four hospitals, and the HRs were combined into a summary estimate of 3.1 (95% CI, 2.1 to 4.6).There was again a significant difference in HRs among the four hospitals by a test for heterogeneity (p<0.001).The STRATIFY score was recomputed using data for inpatients that were extracted from the HIS of each hospital.A STRATIFY score of 2 or more was considered as a high risk for falls.Sensitivity, specificity, PPV and NPV were calculated with their 95% confidence intervals (CI).The patients in each hospital were divided into a high-risk (H) group and a low-risk (L) group.ORs and HRs were calculated with their 95% confidence intervals.P-value was calculated for the hypothesis that there was no association between risk categories and fall status.The ORs and HRs of the four hospitals were combined into summary estimates by using a bivariate random-effects model.The patients were divided into low-risk and high-risk groups based on the STRATIFY score.A score of less than two was considered as a low risk for falls, and score of 2 or more was considered as a high risk for falls.In each hospital, there was a significant difference in rate of falls over time between the low-risk and high-risk groups (p<0.001,respectively).

Discussion
In the present multi-center study, we tried to validate the performance of the STRATIFY score for predicting inpatient falls and to determine whether the STRATIFY score can be used as an effective risk assessment tool for falls in Japanese acute care hospitals.Our results showed that patients at high risk for falls as assessed by the STRATIFY score had a significantly higher rate than did low-risk patients in all of the four hospitals.When the risk of falls was assessed by the STRATIFY score, the ORs and HRs of high-risk patients versus low-risk patients as well as their 95% CIs exceeded 1.0 in all of the four hospitals.These results suggest that the STRATIFY score is an effective tool for prediction inpatient falls in the four hospitals.Since there has been no report on the performance of the STRATIFY score in a Japanese hospital setting, including both acute care and post-acute care hospitals, this report is the first report to show the effectiveness of the STRATIFY score in a Japanese acute care setting.The STRATIFY score is the only tool that was verified in country other than the country in which the tool was developed (Aranda-Gallardo et al., 2013), and our results also suggest that the STRATIFY score is useful for identifying patients at high risk for falls in a Japanese acute care hospital setting.
Our results suggest that the STRATIFY score can be used as a standard risk assessment tool for a multicenter study on falls in Japanese acute care hospitals.
Although the results of our study showed that high-risk patients were more prone than low-risk patients to falls in the four hospitals, the ORs and HRs of high-risk patients versus low-risk patients were significantly different among the four hospitals.The exact reason for this difference is uncertain, but there are several possible reasons.
Although the four hospitals in our study have similar characteristics as national university hospitals, the hospitals were located in diverse settings including an urban area (Hospital D), suburban area (Hospital B) and rural areas (Hospitals A and C).Patient background, procedure to assess the risk for falls, fall prevention program practiced by the staff, rates of reports of fall events and data extraction efficacy from the HIS might be different among the four hospitals.In addition, timing of risk assessment was not ideal in this study because the STRATIFY score was recomputed based on data at patient admission in this study.It is ideal for risk assessment to be performed just before the patients fall (Swartzell, Fulton, & Friesth, 2013).In relation to these possible reasons, the risk stratification and proportion of high-risk patients were significantly different in the four hospitals.
Another reason why the ORs and HRs were significantly different in the four hospitals is the difference in the rate of falls among the hospitals.The rate of falls showed an almost two-fold difference between the hospitals with the lowest and highest rates.Previous studies indicated that the rates of falls in acute care hospitals varied between the hospitals, ranging from 2 to 10 per 1,000 patient-days (DiBardino, Cohen, & Didwania, 2012).The difference in the rate of falls among the four hospitals might again be due to the differences in patient background, fall prevention program performed in the hospitals, and underreporting bias of the falls.Underreporting bias of adverse events in a hospital is a well-known problem of incident reporting, and it is the case for falls that occurred in acute care hospitals (Hill et al., 2010;Healey et al., 2008;Grenier-Sennelier, Lombard, Jeny-Loeper, Maillet-Gouret, & Minvielle, 2002).About 25% of inpatient falls recorded in the EMR were not reported by incident reporting in a Japanese acute care setting (Toyabe, 2012b(Toyabe, , 2015)).The speculation that underreporting is a reason for the difference in the rate of falls between the hospitals is supported by the paradoxical result that the rate of falls was the lowest in Hospital B, in which the proportion of high-risk patients was the highest among the four hospitals.
Although patients who were judged as being at high risk were significantly more prone to fall than were low-risk patients in the four hospitals, the PPV of high-risk patients versus low-risk patients was very low in all hospitals.
The low PPV problem in the STRATIFY score has been pointed out in a previous report (Webster et al., 2010).
Since PPV and NPV are directly related to the prevalence of problems in the cohort (Hendrie, Hall, Arena, Legge, 2004), the low prevalence of falls in the four hospitals might also have caused the low PPV in our study (Webster et al., 2010;Oliver et al., 2008b).Most of the patients who were judged as being at high risk by the STRATIFY score did not fall, and intervention for the high-risk patients often ended up being wasted.Therefore, cost-effectiveness analysis is necessary to use the STRATIFY score as a tool for target intervention for patients who have been assessed as being at high risk for falls (Cumbler, Simpson, Rosenthal, & Likosky, 2013).On the other hand, NPV of high-risk patients versus low-risk patients was high in all of the four hospitals.The high NPV and low PPV of the STRATIFY score suggest that the score might be suitable as an initial screening tool but not suitable to target patients for fall preventing intervention.Billington et al. performed a systematic review and meta-analysis of the STRATIFY score and suggested that the STRATIFY score should not be used in isolation (Billington, Fahey, & Galvin, 2012).Rather, they suggested that the tool could be used in step 1 of a fall management strategy.Since our results were the same as the results of their review, the STRATIFY score should be used as an initial screening tool for identifying patients at high risk for falls in clinical practice.The relatively low performance of the STRATIFY score in our study might be related to the design of our study.Hains et.al reported that 'prospective' evaluation of a hospital fall risk screening tool showed lower performance than that with a 'retrospective' study design (Haines, Hill, Walsh, & Osborne, 2007).According to their definition of the term 'prospective', the following must have been collected or selected before evaluation of the risk assessment system: data required for assessment, items of the risk assessment system, weighting of each item and cut-off point for the risk assessment system.Checking the design of our study by the definition, the design of our study seems to fulfill 'prospective' despite the retrospective collection of data from the HIS.The design of our study might be a reason for the relatively low performance of the STRATIFY score in our study.

Weakness
Although our results suggest that the STRATIFY score can be used as a screening tool to detect patients at high risk for falls in a Japanese acute care setting, the risk stratification and the rate of falls significantly differed among the hospitals.Further research is needed to determine whether the results can be generalized to all acute care hospitals in Japan.

Figure 1 .
Figure 1.Comparison of the rates of falls among the hospitals

Figure 3
Figure 3 Comparison of stratification of patients at risk for falls among the hospitals

Figure 4 .
Figure 4. Relationship between risk for falls and rate of falls over time in each hospital

Table 1 .
Validation of the STRATIFY score in each hospital

Table 2 .
OR (A) and HR (B) for high-risk versus low-risk group for falls