Tackling variation in the early stage evaluation of a state-wide integrated care program

Introduction: Under the NSW ICS, the NSW Government has committed $180 million over six years to implementing innovative and locally led models of integrated care in NSW[1]. A challenge has been how best to approach the evaluation of these projects in order to capture learning and outcomes. The situation is confounded by the variation in care models and target populations observed across the demonstrator and innovator sites. Literature suggests that system level outcomes can take years to appear however there is a call for evidence on the impact integrated care investment has made to date.

Methods: The study comprised all IC sites for which data was available – five in total (2960 patients). Using ANOVA and paired comparisons, we evaluated year-on-year differences in (1) the number of ED visits, and (2) length of stay for hospital admissions, for each patient, taking enrolment date as the point of reference and going back 4 years prior to that date. Year-on-year differences, as a self-referential metric for individual patients afford a more direct measure of efficacy in a heterogeneous cohort setting. It also results in a technically preferable Gaussian distribution as opposed to a Poisson distribution.

To address the potential inferential complications arising from not having a control group, we devised a second study taking our inspiration from propensity matching, and capitalising on the opportunity created by the (quasi) step-wedge nature enrolment.  Patients who have been in the program for 10 months or over were classified as the “intervention group” and those who have been in for 3 months or less were classified as the “control group”. A multiple linear regression was then performed with ICOD status and IC site as predictors.

Results: In four of the five IC sites we studied, cohorts had an accelerating pattern of ED presentations for the four years prior to the IC program.  In each case this accelerating pattern came to a halt with IC.

This result was validated by the second study for all four sites.  A multiple linear regression model for differential ED visits (against a pre-enrolment baseline) with intervention status and IC site as predictors confirmed IC intervention as a negative predictor for ED visits. As a rule, being in the intervention group meant a reduction in ED visits compared to the control group.

For two of the four IC sites that showed a halt in ED presentations, similar analyses of unplanned overnight hospital admissions reveal year-on-year increases not only come to a halt but are reversed with intervention. Total length of stay is also reduced for these cohorts.

Conclusion: The results provide evidence of early stage impact by the IC program in four of the five LHDs we evaluated.

The study also illustrates a practical application of statistical learning methods capable of extracting evidence of early stage impact from a state-wide heterogeneous set of care models.

References:

1- Available from: http://www.health.nsw.gov.au/integratedcare/Pages/default.aspx