In the semi-darkness of the evening a professor walks in the wood. He sees a bird. It flies like a duck. It swims like a duck. It quacks like a duck. Back at home he tells his children: “I saw a duck!’.
Later that same evening the professor settles down to write a paper on a case study of a telehealth innovation he has been investigating—one that provided a device to patients with chronic obstructive pulmonary disease (COPD) so that professionals could monitor and manage patient care remotely, and one where patients could manage care better by themselves. The results seemed promising: patients and carers were enthusiastic; self-reported health outcomes had improved; professionals supporting the delivery of the innovation were highly satisfied and felt care was provided in a more integrated way; and hospital admissions and lengths of stay per patient enrolled in the study had decreased compared to previous levels. The professor concluded that the innovation was a great success and that the deployment of new technologies had significant promise in delivering more cost-effective care to people with COPD.
On publication of the professor's research his critics hit back. To them, it was unclear from the professor's study whether the telehealth intervention reduced overall costs to the local health system. They pointed out that COPD-related admissions to the local hospital continued to rise at the same rate, suggesting that the interventions were targeted at the wrong people. They queried whether it was really the technology that was important since patients received active nurse-led case management alongside it which they had not benefited from previously. They thought the sample size was too small to be meaningful. They concluded the research was flawed, could not be applied to other settings, and should not be included as ‘high-level’ evidence to influence whether telehealth solutions should be adopted more widely.
Lack of robust evidence is often cited in telehealth as a barrier to adoption, as it is with many integrated care innovations more generally. In our hypothetical example, the level of uncertainty in the professor's observations provided the excuse needed to ignore his results, rather than to take on board the positive noises being produced. It is as if his children had said, disbelievingly, “Are you sure what you saw was a duck? It was very dark outside. I think you may have seen that small swan that we saw last week, or even that strange bird that visits our tree-house!” Without a sound methodology, results and observations are open to different interpretations.
The question we pose in this editorial is what level of evidence is good enough to provide a convincing case for adoption of, in this case, telehealth? This question lay at the heart of discussions at the recent International Congress on Telehealth and Telecare, streamed live from The King's Fund in London to over 2000 people from 60 countries (proceedings will be published in a forthcoming IJIC conference supplement, but the debates can be revisited at: www.kingsfund.tv/telehealth).
This international congress was built around the early findings from the UK Department of Health's Whole System Demonstrator (WSD) Pilot Programme—the largest cluster randomised control trial (RCT) of telehealth ever conceived. The field trial was established to provide a ‘proof of concept’ on the legitimacy of telehealth as a way of caring for patients with chronic illness. Established in early 2009, the trial has involved more than 6000 patients with heart failure, COPD or diabetes whom were randomly allocated to either an experimental group or care-as-usual group. The complex study design sought to examine cost-effectiveness, changes in utilisation of care facilities, health outcomes, impact on carer burden, utility of the technology from a patient and professional point of view, plus issues related to organisational development. To account for local context, the trial was further sub-divided across three different localities—Cornwall (rural), Kent (suburban and wealthy), and Newham (a deprived and multi-ethnic inner-city London borough). The overall cost of the trial was reported to be £31 million sterling of which about 12% was allocated to the evaluation itself.
Those working in delivering care during the trial were positive in their feedback at the conference. Positive experiences from patients, carers, nurses and family practitioners were reported in Newham. In Cornwall, it was reported how the technology worked safely, protected privacy and had developed good interoperability between the different e-health applications and data sets required. In Kent, local evaluations concluded the scheme to be cost-effective and especially beneficial in relieving the burden on carers. Results presented by the Department of Health reported ‘encouraging’ early findings, including (unspecified) reduced hospital admissions for people with COPD and a ‘positive’ impact on system cost-effectiveness.
However, amongst the attending delegates and those watching remotely, there was considerable doubt as to the overall value of RCTs. First, as proven in the UK context, they are expensive and time-consuming. Second, RCTs remain contextually-specific and problematic to generalise their findings in other regions and countries. Third, new assistive technologies are emerging rapidly meaning that current evaluations will soon be out-of-date since the future format of care will change; for example, through the use of social media, mobile phones, remote sensors, and ingestible monitoring devices. Large academic studies take 3–5 years to complete from formulation of a research question to publication in an academic journal; and then even longer for them to become adopted in practice (if at all). Such a glacial process in the production and adoption of ‘high quality’ evidence simply does not fit well in the innovative and fast changing field of telehealth.
During the debate at the conference it was argued that evidence was essential, but that this should not get in the way of promoting technological innovation and investing in the necessary redesign of care services. Ideally, it was argued, research and service innovation should run conjointly as party of a continuous process of innovation and improvement. But how could this be achieved?
In an attempt to solve this problem on the utility of research findings, the Danish Health Technology Assessment (HTA) researcher Kristian Kidholm presented to the conference a new research model to evaluate telemedicine innovations (MAST, accessible at http://www.mast-model.info). The purpose of MAST has been to provide a structured framework for assessing the effectiveness and contribution to quality of care of telemedicine applications, based on users need for information as a basis for decision-making. The evaluation methodology promotes a ‘transferability assessment’ based on seven standard questions to ensure for comparability in the design of research. It also provides a platform through which to enable routine outcome monitoring (‘romming’) that allows real-time data to be collected, aggregated, continuously statistically analysed, and reported over time to evaluators and decision-makers alike. Such an approach is currently popular in the Dutch mental health services (see www.routine-outcome-monitoring.nl).
The conclusion we reach is that research and evaluation needs to be much more closely aligned with innovators and decision-makers to enable them to utilise the best available evidence in ‘real-time’. However, traditional research and evaluation studies are not conducted to facilitate such rapid knowledge-transfer. In particular, the adoption of new technologies is a fast-moving area that cannot sit back and wait for the evidence as traditionally gathered. New methodologies and approaches are needed and this will only come to fruition through better cooperation between scientists, professionals, product developers, and policy-makers.
As a scientific journal dedicated to the promotion of innovative methods, research papers and case studies of approaches to integrated care such as telehealth we invite readers with expertise in this field to submit articles and to stimulate debate. As an e-based journal, submissions are accepted at all times and are peer reviewed and published on a continuous basis. This means that your work does not lie on a shelf waiting for publication since this is guaranteed between three and eight weeks after acceptance.