Consensus group sessions are useful to reconcile stakeholders’ perspectives about network performance evaluation

Introduction

Over the last decades, health care systems have faced important challenges in dealing with pressures to increase the performance of the services they offer while the allocated resources to offer such care have remained stable or have decreased. Integration has been presented as an efficient solution to improve service delivery by tackling fragmentation and efficiency issues. One of the many forms of service integration is inter-organizational networks, which could be defined from a sociometric perspective as a set of three or more organizations linked together using different strategies [1, 2]. Other theorists, however, suggest that it is the mode of governance, neither hierarchical or competitive, that defines a network [2, 3]. Despite their apparent simplicity, networks are complex organisations that vary greatly on many aspects. Indeed, networks can involve various organizational levels of the organization, such as administration, governance, caseload, management, etc. [4, 5]. They may be created at various levels, bringing governmental agencies, regional or local facilities or programs [4] to work together using one or many integration strategies, such as communication, resource sharing, joint programs, client transfer agreements and so on [1]. Depending on the intensity of the links created, networks can be cooperative, coordinated or collaborative [6, 7], the latter being the most intensive and the most demanding. According to the theoretical perspective fostered, networks can also be thought of as a source of knowledge and information, a system of power regulation or a social structure created by environmental tensions [2]. A very common perspective of networks in the field of health services is that of the services perspective, where network partners work together to create a seamless continuum of care. Because they are viewed by many service professionals, policy makers and researchers as a potential integration strategy that could reduce service fragmentation and improve the coordination of services and effectiveness [8], networks have become a popular way to reconfigure health service delivery systems [9–11] found in many countries. In addition, the challenging context in which health care systems must operate has generated a growing commitment to the evaluation of health services. Indeed, programs, facilities and systems are increasingly required to demonstrate their performance. The rising popularity of networks and the renewed importance of health services evaluation have bred new interest in the evaluation of integrated networks.

The evaluation of integrated networks is complicated by the fact that the concept of network performance is poorly defined [3]. This is due in part to the fact that the advantages of networks have been predominantly studied with regard to their effect on individual organizations, and not with regard to the whole network [3, 10]. To date, network performance has been mainly appraised using performance indicators traditionally used for individual organizations. At best, this may lead to partial and inaccurate evaluations of network performance [10, 11]. The lack of clarity about the concept of network performance is likely to induce difficulties in network performance evaluation, and consequently limits our ability to use evaluation results to improve network performance [12].

In addition, the evaluation of integrated networks is complicated because networks involve many organizations that may have different values, cultures and mandates. Consequently, network members could have different perspectives about what constitutes the performance of their network [11, 13, 14]. Moreover, Provan et al. [15] stated that network stakeholders tend to see the network and its performance from the perspective of their own organization. Huxham [14] also mentioned that the variety of organizational and individual agendas that are present in collaborative situations make it difficult to agree in practice. This author further suggests that when partners do not completely agree on a shared purpose, they may not be able to agree on the next steps of a collaborative process [16]. Indeed, if they do not agree on what constitutes new network performance, network partners could be in conflict on the evaluation process and indicators and interpret differently the evaluation results. A lack of a common vision risks disintegrating the collaborative advantage of network.

To address various stakeholders’ perspectives of organizational performance, the multiple-constituency approach to evaluation has been proposed [17]. This approach states that the multiple constituencies or stakeholders of an organization could hold different perspectives about what constitutes its performance, and that it is unjustified or arbitrary to choose one perspective over another for evaluation purposes. According to the multiple-constituency approach, these perspectives should be incorporated into a larger vision reflecting the sum of the evaluative criteria applied by the various stakeholders involved. Several studies have used this approach to build performance frameworks for various organizations [12, 18–20], most of them having found differences in perceptions and priorities among different types of stakeholders. However, the use of this approach within a performance evaluation process presents two major weaknesses. The first weakness is that the multi-constituency approach does not explicitly explain how to deal with divergent perceptions. Without further interaction among stakeholders, each group maintains its own different understanding of performance despite being embedded within a larger framework. Deciding upon what to include in a performance evaluation framework, based solely on a multi-constituency approach, will thus likely fail to harmonize the standards and values of the different stakeholder groups and may be unsuccessful in promoting collaboration and mutual trust between network members [21].

A second challenge associated with using the multi-constituency approach for performance evaluation is that the methods frequently used within this approach (written questionnaires) provide only a limited and thus superficial understanding of the underlying rationale for the differing opinions of stakeholders. Moreover, responding to a questionnaire does not provide stakeholders with an opportunity to share and to discuss their preferences to better understand their partners’ vision. Many researchers [20, 22, 23] have emphasized the need to go beyond simply reporting differences in stakeholder perceptions and to engage stakeholders in meaningful discussions to arrive at a common understanding of an organization’s performance. This may be particularly important when integrated network links partners from different settings with different values and priorities [24]. However, few authors have proposed ways to do so. Indeed, Zinn and colleagues [25] repeated the steps of a Delphi technique to obtain consensus within homogenous stakeholder groups regarding laboratory management performance indicators. They found the technique useful for reaching a consensus within groups of stakeholders of a similar type (e.g. hospital executives, managed care executives, referring physicians, laboratory regulators and laboratory managers). However, they did not explore whether the technique was effective in reconciling the perceptions across various stakeholder types. The Delphi technique has certain benefits (e.g. anonymity and consultation of persons over large distances) but it does not provide opportunities for meaningful discussions between stakeholders [26]. Discussion could be useful especially when the subject under study is complex and abstract [26], such as organizational performance. Such subjects might thus be better explored through methodology involving group discussions. Studies of collaboration highlight the importance of a drafting process that is highly participatory and involves key stakeholders [27].

The creation of a common vision between partners sets the foundations for collaborative efforts and enhances the collaborative advantage of network. However, traditional approaches integrating multi-constituency perspectives fail to reconcile partners’ visions and involve them only superficially in the process. To gain a better understanding of what constitutes network performance and to improve the performance evaluation process, there is a need to explore innovative approaches to reconcile stakeholders’ perceptions while giving them an opportunity to discuss and share ideas. Falling short of this may result in fragmented network efforts, disinterest of network partners with regard to the vision, evaluation process and evaluation outcome, they deem as not responsive to their reality.

TRIAGE, a consensus technique, could be a viable method to create a shared vision in a participatory way. The general goal of this study was to explore whether TRIAGE group discussions could reconcile the perspectives of network members with regard to the importance of performance dimensions. Specifically, we 1) determined the initial degree of consensus existing among individual TBI network organization members with regard to importance of performance dimensions for performance evaluation, 2) explored the usefulness of consensus group sessions to reconcile these perceptions, by comparing the importance paid to the different dimensions of performance using questionnaires and group discussion.

Method

We conducted the present study from November 2006 to November 2008 during the accreditation process of network organizations providing services to persons with traumatic brain injury in the province of Québec (Canada). We used the TRIAGE consensus technique [28, 29], a method that combines an individual data collection phase using questionnaires with a group data collection phase involving group consensus sessions.

Results

Discussion

In this study, we first observed that when obtained through individually answered questionnaires, the extent of consensus about the importance of performance dimensions is low and network members have different perceptions about the importance of at least 50% of the performance dimensions. This suggests that any methodology that does not involve interactions between stakeholders is not likely to produce consensual results, and that a performance framework based on survey results could fail to promote a shared vision among network stakeholders. We also observed that the extent of consensus varied across networks and according to the dimensions examined. Several studies using a multi-constituency approach involved participants from multiple settings [18, 20, 25]. However, to our knowledge, our study is the first to examine the variability in the extent of consensus across integrated settings (e.g. networks). Larger networks exhibited more differences in stakeholder perceptions, while smaller ones were typically (but not systematically) more consensual. This is not surprising since networks linking a large number of participants are more likely to include heterogeneous organizations. Our findings thus support Huxham’s [14] theory, in that the complexity of partnerships seems to influence mutual understanding. However, it is also possible that participants of smaller networks had the opportunity to be in closer proximity to each other and developed stronger links. The small size of the network could thus facilitate the adoption of a common vision. Even in smaller networks, we observed an absence of consensus for at least half of the dimensions surveyed. In combination, these results suggest that consensus group sessions could be beneficial for organizations of all sizes, but especially for larger ones, regrouping many members potentially holding different perceptions of what is important to consider for performance evaluation.

The results of the survey also indicate that the dimensions deemed more important by network participants are more consensual than others. This result suggests that the participants intuitively agree on the higher importance of a core set of dimensions often qualified as fundamental to the networks when discussed in groups. The fact that a high importance was paid to dimensions such Ability to attract the clientele (Accessibility) and Continuity supports the idea that the performance of an integrated network of care is different from the performance of a single organization [7, 10, 11], but shares the goal of many integration initiatives [4]. Because the concept of network performance is relatively new and yet not fully shared by all network members, its specificity should be made explicit to all stakeholders before any evaluation process takes place to avoid different visions tainted by the characteristics of a single organization (rather than of a whole network).

Another important result of our study is that overall, consensus group sessions did not influence the quantitative importance attributed to performance dimensions by the member participants. Given the small changes in importance scores provided by group participants, one might question the usefulness of consensus group sessions over the simple computation of median importance scores compiled using an individual data collection phase. We argue that the added value of consensus group sessions is evident in the discussions made possible between participants. Indeed, qualitative analysis of the group discussions showed that participants expose the reasons that motivated the initial and group scoring. The exchanges help other network members understand their visions and it helps sustain discussions leading to a shared understanding of network performance. Such harmonization of values and standards helps increase mutual trust and confidence among partners, which are deemed essential to improve collaboration [6, 14] and to increase network performance. A second important added value of the consensus group session is that it represents a bottom-up approach to evaluation, in that it involves stakeholders in the design of the evaluation process. Because group discussion takes into account the values and interest of networks partners, they are likely to improve the responsiveness and the usefulness of the evaluation process as well as stakeholders’ adherence to the resulting performance framework. Consensus group discussion can also help to empower network participants who usually have less power in the network [14], by giving them an opportunity to take part in the decision about the network performance. This could facilitate the ‘collaborative advantage’ and neutralize the ‘collaborative inertia’ that are often observed in collaborative efforts [14], such as networks.

In general, the results suggest that consensus group sessions are a useful step in the process of developing a TBI network performance evaluation framework and that the reconciliation of stakeholder perspectives could be a logical complement to multi-constituency approach studies. They could be used in any type of network, but would be most suitable for service networks, such as the one studied here and they may be more useful to participants who already have minimal experience of working together [14]. Consensus group sessions may be useful to clarify network performance frameworks, but could also sustain goal or process clarification. As such, the TRIAGE technique may represent a tool for integrated care, to improve a common vision and language. For an example, it could help practitioners who work collaboratively in a joint program to explore their perceptions of the program’s goals, and to discuss the ones to prioritize. It constitutes a collaborative and reflexive tool that taps into benefits from the multiple perspectives and empowers the participants into establishing a shared construction of a common vision. It could be beneficial to the effectiveness and to the climate of a network, two important dimensions of network performance [12].

Group sessions allow participants to decide upon objectives they deem important, while using minimal time and human resources. With regard to the effectiveness of TRIAGE, the technique allowed the participants in each network to discuss their perceptions and to agree upon the importance they attributed to the dimensions discussed. This surpasses the 80% cut-off typically used in consensus methods, such as the Delphi technique. TRIAGE, however, requires travelling for some participants, which could be difficult when a network is geographically spread out. Group discussions could be limited by time restrictions or influenced by the level of expertise or the personality of participants.

We acknowledge that this research has some limitations. We used an arbitrary cut-off of 20% to determine the dimensions requiring discussion during the sessions. Using another cut-off may have led to different results. Only a few representatives from each individual facility could participate in the group sessions and we could not verify whether their participation was representative of the opinions expressed by all of their team members. The high participations rates of individual organizations and networks support the generalizability of the results.

Conclusion

This study reports the usefulness of consensus group sessions to reconcile the perceptions of stakeholders about the important dimensions to include in an evaluation of network performance. Since our study is the first to illustrate such a process, repeating similar experiences is required to determine the advantages and drawbacks of different processes to reconcile divergent stakeholder perspectives. Moreover, it remains to be demonstrated how implicating stakeholders from an early stage affects the evaluation process. In the future, the exploration and reconciliation of stakeholder’ perspectives about performance evaluation could become an important step in the planning of any performance evaluation process.

Reviewers

Edgar Borgenhammar, PhD, MBA, DSI, Professor of Health Services Management, Sweden

Louis Demers, Prof, École nationale d'administration publique [National School of Public Administration], Québec, Canada

Robyn Keast, Associate Professor, School of Management, Faculty of Business and Research Director, Airport Metropolis Project, Faculty of Built Environment and Engineering, Queensland University of Technology, Australia.

Figure and Tables

Figure 1.