Figure 1
Dimensions of the B3-MM (retrieved from: http://www.scirocco-project.eu/maturitymodel/).
Table 1
Search terms used in narrative literature review.
| Component | Terms | Remarks |
|---|---|---|
| Construct | Integrated care, coordination of care, continuity of care, patient centered care | Based on the work of Uijen et al. [47] modified by Bautista et al. [23] |
| Instrument | Questionnaire, measure, survey, instrument | User-defined based on Terwee et al. [48] |
| Feature | Degree, maturity model, level, phase | Terms reflecting “maturity” |
Table 2
Criteria for the level of evidence and overall assessment of measurement properties.
| Criteriaa | Overall assessment | Level of evidence |
|---|---|---|
| Consistent findings in multiple studies of good methodological quality OR in one study of excellent methodological quality | +++ or – – – | Strong |
| Consistent findings in multiple studies of fair methodological quality OR in one study of good methodological quality | ++ or – – | Moderate |
| One study of fair methodological quality | + or – | Limited |
| Conflicting findings from multiple studies | +/– | Conflicting |
| Only studies of poor methodological quality OR only indeterminate results from multiple studies regardless of methodological quality | ? | Unknown |
| Measurement property not assessed | 0 | Not assessed |
[i] a Adapted from Uijen et al. [47].
Table 3
Criteria for rating the adequacy of the reported measurement properties.
| Measurement property | Reported Result | Quality criteria [47] |
|---|---|---|
| Content validity | + | The target population considers all items in the questionnaire to be relevant AND considers the questionnaire to be complete |
| ? | No target population involvement | |
| – | The target population considers items in the questionnaire to be irrelevant OR considers the questionnaire to be incomplete | |
| 0 | Did not assess content validity |
Table 4
List of experts in the first Delphi round.
| Types of experts | Number of experts selected | Experts retrieved from |
|---|---|---|
| Corresponding/first author of scientific articles (researchers with experience in the measurement or development of integrated care) | 10 | Articles included in the literature review used in the study |
| Experts with practical experience in the development, implementation and/or monitoring of integrated care interventions | 10 | SCIROCCO consortium partners* |
| Experts from the B3 Action Group on Integrated care | 11 | SCIROCCO consortium partners* |
| Experts with experience in the field of Information and eHealth services in the field of integrated care | 10 | SCIROCCO consortium partners* |
| Members of the SCIROCCO advisory board | 5 | SCIROCCO consortium partners* |
| Researchers with expertise in measurement of development of integrated care | 9 | A convenience sample provided by one of the researchers |
[i] * Basque Country (ESP), Norrbotten Lans Landsting (SE), Puglia region (IT), Olomouc region (CZ) and Scotland (UK).
Figure 2
Flowchart calculation of consensus.
Figure 3
Flowchart narrative review process.
Table 5
Oversight narrative review search terms and hits.
| Database | Final used search term combination/string | Date search | Hits | Filter | Selected articles based on title/abstract | Selected articles after full text selection | Grey literature | Peer-reviewed literature | Total included in review |
|---|---|---|---|---|---|---|---|---|---|
| IDEA | “integrated care” | 26-7-2016 | 126 | None | 2 | 1 | 0 | 1 | 1 |
| integrated care or coordination of care or continuity of care or patient centered care and measure or instrument or survey or questionnaire and degree or maturity model or level or phase | 1-8-2016 | 164 | English only | 6 (1 duplicate with Google Scholar) | 2 | 0 | 2 (1 dissertation) | 2 | |
| Google Scholar | (“integrated care” or “coordination of care” or “continuity of care” or “patient centered care”) and (measure or instrument) | 29-7-2016 | 141 | None | 3 (1 duplicate with Google) | 1 (Retrieved from the article of Bainbridge et al. [68]) | 0 | 1 | 1 |
Table 6
Overview of articles matching descriptions with B3-MM.
| Dimensions and related indicators as described in B3-MM [30] | Number of article(s) [Reference] |
|---|---|
| 1. Readiness to change to enable more integrated care | 8 [12, 55, 64, 65, 67, 69, 70, 71] |
| 1.1 No acknowledgement of crisis | |
| 1.2 Crisis recognized, but no clear vision or strategic plan | |
| 1.3 Dialogue and consensus-building underway; plan being developed | |
| 1.4 Vision or plan embedded in policy; leaders and champions emerging | |
| 1.5 Leadership, vision and plan clear to the general public; pressure for change | |
| 1.6 Political consensus; public support; visible stakeholder engagement | |
| 2. Structure and Governance | 6 [12, 55, 64, 67, 69, 71] |
| 1.1 No overall attempt to manage the move to integrated care | |
| 1.2 Change underway, but with fragmented organisations & plans | |
| 1.3 Formation of task forces, alliances and other informal ways of collaborating | |
| 1.4 Governance established at a regional or national level | |
| 1.5 Roadmap for a change programme defined and broadly accepted | |
| 1.6 Full, integrated programme established, with funding and a clear mandate | |
| 3. Information and e-Health Services | 11 [12, 55, 62, 63, 64, 65, 66, 67, 69, 70, 71] |
| 1.1 No connected health services, just isolated medical record systems | |
| 1.2 No integrated services used, only pilots/local services | |
| 1.3 eHealth deployed in some areas, but limited to specific organisations or patients | |
| 1.4 Voluntary use of regional/national eHealth services across the healthcare system | |
| 1.5 Mandated or funded use of regional/national eHealth infrastructure across the healthcare system | |
| 1.6 Universal, at-scale regional/national eHealth services used by all integrated care stakeholders | |
| 4. Standardisation & Simplification | 7 [12, 64, 65, 67, 69, 70, 71] |
| 1.1 No systematic attempt to standardise the use of citizen health & care data, or to simplify systems in use | |
| 1.2 Debate on information standards (e.g., coding, formatting); exploration of options for consolidating ICT | |
| 1.3 A recommended set of agreed information standards at local level; a few local attempts at ICT consolidation | |
| 1.4 A recommended set of agreed information standards at regional/national level; some shared procurements of new systems at regional/national level; some large-scale consolidations of ICT underway | |
| 1.5 A unified set of agreed standards to be used for system implementations specified in procurement documents; many shared procurements of new systems; consolidated data centres and shared services widely deployed | |
| 1.6 A unified and mandated set of agreed standards to be used for system implementations fully incorporated into procurement processes; clear strategy for regional/national procurement of new systems; consolidated datacentres and shared services (including the cloud) is normal practice. | |
| 5. Finance & Funding | 8 [12, 55, 63, 64, 67, 69, 70, 71] |
| 1.1 No special funding allocated or available | |
| 1.2 Fragmented innovation funding, mostly for pilots | |
| 1.3 Consolidated innovation funding available through competitions/grants for individual care providers | |
| 1.4 Regional/national (or European) funding or PPP for testing and for scaling-up | |
| 1.5 Regional/national funding for scaling-up and on-going operations | |
| 1.6 Secure multi-year budget, accessible to all stakeholders, to enable further service development | |
| 6. Removal of inhibitor | 7 [12, 55, 64, 67, 69, 70, 71] |
| 1.1 All projects delayed or cancelled due to inhibitors | |
| 1.2 Some projects delayed or cancelled due to inhibitors | |
| 1.3 Process for identifying inhibitors in place | |
| 1.4 Strategy for removing inhibitors agreed at a high level | |
| 1.5 Solutions for removal of inhibitors developed and commonly used | |
| 1.6 High completion rate of projects & programmes; inhibitors no longer an issue for service development | |
| 7. Population Approach | 5 [12, 66, 69, 70, 71] |
| 1.1 No consideration of population health in service provision | |
| 1.2 A population focus of risk stratification but no risk stratification tools | |
| 1.3 Individual risk stratification for the most frequent service users | |
| 1.4 Group risk stratification for those who are at risk of becoming frequent service users | |
| 1.5 Population-wide risk stratification started but not fully acted on | |
| 1.6 Whole population stratification deployed and fully implemented. | |
| 8. Citizen empowerment | 7 [12, 62, 65, 66, 67, 69, 71] |
| 1.1 No systematic plan for empowerment | |
| 1.2 Citizens are not involved in decision-making processes and do not participate in the co-design of their services | |
| 1.3 Policies to support citizens’ empowerment and protect their rights, but may not reflect their real needs | |
| 1.4 Incentives and tools to motivate and support citizens to co-create health and participate in decision-making processes | |
| 1.5 Citizens are supported and involved in decision-making processes, and have access to information and health data | |
| 1.6 Citizens are involved in decision-making processes, and their needs are frequently monitored and reflected in service delivery and policy-making. | |
| 9. Evaluation methods | 6 [12, 64, 67, 69, 70, 71] |
| 1.1 No routine evaluation | |
| 1.2 Evaluation exists, but not as a part of a systematic approach | |
| 1.3 Evaluation established as part of a systematic approach | |
| 1.4 Some initiatives and services are evaluated as part of a systematic approach | |
| 1.5 Most initiatives are subject to a systematic approach to evaluation; published results | |
| 1.6 A systematic approach to evaluation, responsiveness to the evaluation outcomes, and evaluation of the desired impact on service redesign (i.e. a closed loop process) | |
| 10. Breadth of ambition | 11 [12, 55, 62, 63, 64, 65, 66, 67, 69, 70, 71] |
| 1.1 No level of integration | |
| 1.2 Services in silos; the citizen or their family as the integrator of services | |
| 1.3 Integration within the same level of care (e.g., primary care) | |
| 1.4 Integration between care levels (e.g., between primary and secondary care) | |
| 1.5 Integration includes both social care service and health care service needs | |
| 1.6 Fully integrated health & social care services | |
| 11. Innovation management | 4 [12, 64, 69, 71] |
| 1.1 No plan for innovation management | |
| 1.2 Isolated innovations across the region/country, but limited visibility | |
| 1.3 Innovations are captured and published as good practice | |
| 1.4 Innovation is governed and encouraged at a region/country level | |
| 1.5 Formalised innovation management process in place | |
| 1.6 Extensive open innovation combined with supporting procurement & the diffusion of good practice. | |
| 12. Capacity building | 8 [12, 62, 63, 64, 65, 67, 69, 71] |
| 1.1 No plan for capacity-building | |
| 1.2 Single organisational initiatives engaged in process improvement | |
| 1.3 Some mechanisms for sharing knowledge among organisations | |
| 1.4 Systematic learning about IT; integrated care and change management | |
| 1.5 Knowledge shared, skills retained and lower turnover of experienced staff | |
| 1.6 A ‘learning healthcare system’ involving reflection and continuous improvement |
Table 7
Number of validation studies, the methodological quality of the studies, the direction (positive or negative) of results of the measurement properties and overall quality measurement property content validity score.
| Instrument (data derived from Bautista et al. [23]) | Author (name of first author only used) [reference] | Number of validation studies | Methodological quality of studies on content validity (COSMIN checklist [51]) | Direction of results (Table 3) of measurement property content validity | Overall quality measurement property content validity score (Table 2) |
|---|---|---|---|---|---|
| Scale of Functional integration | Ahgren[55] | 1 | Fair | a | ? |
| DELTA service user assessment | Ahgren [62] | 1 | Fair | a | + |
| Human Service Integration Measure | Browne [63] | 1 | Excellent | a | ? |
| Unnamed1 | Lukas [64] | 1 | Fair | a | + |
| Dual Diagnosis Capability in Health Care Settings (DDCHCS) | McGovern [65] | 1 | Not assessed | a | 0 |
| Patient Perceptions of Integrated Care Survey (PPICS) | Singer [66] | 1 | Fair | a | + |
| Unnamed2 | Uyei [67] | 1 | Good | a | ? |
| Instruments (derived from the narrative review) | |||||
| HCP integration survey | Bainbridge [69] | 1 | Fair | ? | ? |
| Unnamed3 | Calciolari [70] | 1 | Fair | ? | ? |
| Development Model of Integrated Care (DMIC) | 5 | +++ | |||
| Minkman [12] | Excellent | + | |||
| Minkman [12] | Excellent | + | |||
| Minkman [12] | Excellent | + | |||
| Minkman [12] | Excellent | + | |||
| Longpré [71] | Fair | ? |
[i] a Data on direction of results per instrument was summarised in the review of Bautista et al. [23]. No individual data per instrument was provided.
Table 8
Characteristics of experts in Delphi rounds 1, 2 and 3 (in % unless stated otherwise).
| Characteristic | Category | Expert group first round (n = 26) | Expert group second round (n = 13) | Expert group third round (n = 10) |
|---|---|---|---|---|
| Age (year) | Min–Max | 36–71 | 36–71 | 36–71 |
| Average (sd) | 49.23 (11.73) | 52.69 (13.22) | 52.60 (13.43) | |
| <40 | 23.1 | 23.1 | 20 | |
| 40–50 | 30.8 | 23.1 | 30 | |
| >50 | 46.2 | 53.8 | 50 | |
| Gender | Male | 30.8 | 46.2 | 50.0 |
| Female | 69.2 | 53.8 | 50.0 | |
| Country | Belgium | 3.8 | 7.7 | 10 |
| Canada | 7.7 | 7.7 | 10 | |
| Czech Republic | 3.8 | 7.7 | 10 | |
| Finland | 3.8 | 0 | 0 | |
| Germany | 3.8 | 0 | 0 | |
| Italy | 15.4 | 15.4 | 0 | |
| Luxembourg | 3.8 | 0 | 0 | |
| Netherlands | 7.7 | 0 | 0 | |
| Netherlands and USA | 3.8 | 7.7 | 10 | |
| Portugal | 7.7 | 7.7 | 10 | |
| Spain | 7.7 | 15.4 | 20 | |
| Sweden | 7.7 | 0 | 0 | |
| UK | 15.4 | 23.1 | 20 | |
| USA | 7.7 | 7.7 | 10 | |
| Professional Affiliation | Medicine | 15.4 | 15.4 | 20 |
| Nursing | 7.7 | 7.7 | 10 | |
| Policy | 7.7 | 15.4 | 0 | |
| Managerial | 15.4 | 23.1 | 20 | |
| Research | 46.2 | 30.8 | 40 | |
| Other | 7.7 | 7.7 | 10 | |
| Years of experience | <1 | 0 | 0 | 0 |
| 1–5 | 38.5 | 23.1 | 30 | |
| 5–10 | 26.9 | 23.1 | 20 | |
| >10 | 34.6 | 53.8 | 50 |