Measuring Progress: Resources for Developing a Mental Health and Addiction Performance Measurement Framework for Canada

This resource was published in 2018. The data may be out of date.

Over the last few decades, more and more countries have come to recognize the immense impact of mental health problems and illnesses — including addiction — on individuals, families and communities. This has driven changes to policy and practice in Canada and other jurisdictions. It has also triggered calls for better data to determine if such steps are having a positive impact and to inform further system change.

This report identifies strategic areas of focus for developing a pan-Canadian information and performance measurement framework specific to mental health and addiction (MHA). Such a framework could be parlayed into a performance measurement system that would make it possible to measure and report on MHA outcomes across the country — stimulating data-informed service change, enabling greater collaboration and making a difference in the lives of Canadians from coast to coast to coast.

The Mental Health Commission of Canada (MHCC) initiated the project that led to this report with four main goals:

1. systematically compile resources to support the development of a pan-Canadian, system-level performance measurement framework for MHA
2. map common policy priorities across provinces and territories
3. learn from existing performance measurement frameworks and related development processes
4. formulate recommendations for an engagement and consultation process to drive framework development

Making the case for a framework
A pan-Canadian MHA framework would help focus performance measurement on aspects of Canada’s complex system that are key to effecting change. It would articulate a “system” vision that, ideally, aligns with common policy directions and shared stakeholder values — increasing the likelihood of impact. It would also help ensure balance across sets of indicators, protect against unintended effects, clarify relationships among indicators and reveal gaps (i.e., where indicator development is needed). It would be designed to complement, rather than duplicate, existing and planned provincial/territorial performance measurement initiatives.

The full benefits of a framework could be realized through a pan-Canadian performance measurement system with the capacity to compile and report on common system-level measures and with a forum for sharing improvement practices and solutions across jurisdictions. There are reasons to be optimistic this vision can be realized, including the fact that other federations around the world have established national, system-level MHA performance measurement initiatives. There is also a strong Canadian example in the pan-Canadian performance measurement framework of the Canadian Partnership Against Cancer (CPAC), which includes mechanisms for reporting and collaborative improvement.

This report provides insight into the five key steps involved in developing a performance measurement framework as follows:

1. Recognize and acknowledge key issues
   To develop a consensus-based system-level framework across 13 jurisdictions and in collaboration with key national level stakeholder organizations would be a complex undertaking complicated by differing values. It will be important to identify and address key issues and concerns inherent to performance measurement early in the process. These include determining the ultimate purpose of performance measurement, whose performance will be measured, whether comparisons will take local conditions into account and if there is sufficient capacity to generate quality measures and support action on findings.
2. Cultivate shared language and understanding of key concepts
   A critical starting point is to establish shared understanding of key terms and concepts. This is especially important in Canada due to the complexity of the Canadian context. While a pan-Canadian performance measurement framework would not require partners to have the same service models or approaches, shared understanding would facilitate consensus on the framework’s main features. Establishing shared principles and values is also important. The research team assembled an initial set of concepts as a starting point, but stakeholders must be able to suggest additional concepts, frameworks and world views — especially those whose views have not been well represented to date in work related to performance measurement.
3. Define the overall scope of performance measurement
   Scope refers to the breadth of services/interventions, levels of service and service recipient populations covered by the framework. The framework should be reasonably aligned with the parameters of the MHA service system model, but there may be good reasons for its scope to be developed in stages,e.g.,60 or even to have separate but related performance measurement frameworks for service components or special service populations. The important thing is that decisions about scope (and the costs and benefits of those decisions) be explicit.
   
   The provincial/territorial policy documents reviewed for this report had broad predominantly models that included the social determinants of health as well as a full continuum of interventions. Twelve of the 13 policy documents included substance-related problems/addiction, which subject-matter experts (SMEs) agreed would be important in a pan-Canadian PM framework. Special consideration of populations including children and youth and Indigenous peoples were viewed as important as well, as was honouring Indigenous stakeholders’ values, principles and data standards.
4. Define the framework’s key dimensions and domains
   The literature is unequivocal that performance measurement frameworks and indicators should relate back to strategy aims. Provincial and territorial policy priorities — especially those held in common — offer key information for framework development. The top five policy priorities in common across provinces and territories identified through the research described in this report were access, promotion/prevention, children and youth, person-centred care, and Indigenous peoples.
   
   Notably, some SMEs felt reviewing past frameworks could result in a backward-looking pan-Canadian framework. It will be important to look ahead to build progressive frameworks with forward-looking indicators. SMEs also noted it will be important and desirable for stakeholders, in particular Indigenous and other equity-seeking groups, to lead the aspects of the work that relate to their communities.
5. Select indicators
   Four general approaches to choosing performance measurement indicators emerged from the literature: simple selection by a project team, selection by an expert panel, systematic ranking approaches (e.g., Delphi methods) by experts or broader stakeholders and approaches like care mapping/concept mapping. There is wide variability in indicator selection processes across Canada, from gradual, informal operational processes to expert panel/research team processes, to formal processes that included local stakeholders engaged via consultation meetings, and further to research-based, multi-stakeholder, multi-stage Delphi rating processes.
   
   SMEs did not consider existing indicators to be sufficiently visionary, arguing that they will only result in “more of the same.” Aspirational indicators that could move the system more quickly in the desired direction are needed, as well as indicators based on the voices of people with lived experience and those that capture information for Indigenous and other equity-seeking groups.

Systematic engagement and consultation processes
The task of developing a pan-Canadian MHA performance measurement framework is value-laden, which creates a divergence of opinions. A systematic and intentional approach is necessary to develop a framework and set of indicators that stakeholders can endorse despite their plurality of views. The process of deliberative dialogue used to create the Mental Health Strategy for Canada is an important general model. The research team also found five specific initiatives that involved and reported on systematic processes for MHA service or performance measurement frameworks, which can inform pan-Canadian framework development.

SMEs stressed that engagement must be “meaningful,” which includes a co-designed/participatory process with reflection on and acknowledgement of social-historical context and biases, as well as recognition of who gets to decide and who benefits. This requires time and resources.

Getting from framework to system
Performance measurement has been criticized for stalling at the conceptualization stage. The capacity to regularly generate and report on existing indicators, and resources to develop strategic, aspirational indicators are required to realize the ultimate objective — in this case, to improve Canada’s MHA system. While the plurality of data systems across Canada’s makes coherent performance measurement as challenging within this country as it is between countries in other parts of the world, there are exemplary models for a pan-Canadian system. These include CPAC’s Cancer System Performance initiative, which has been reporting by province/territory since 2009, with indicator values available to all stakeholders online. While the cancer field has some distinct advantages that MHA lacks, it is reasonable to aspire to a system with similar features.

Thinking big, starting small, acting now
The resources SMEs compiled for the current project can inform and support a plan for developing a pan-Canadian MHA performance measurement framework. They are grounded in provincial and territorial policy priorities, features of existing frameworks and lessons learned from systematic developmental processes used for framework development in Canada.

While there is substantial diversity among existing performance measurement frameworks, there is also a richness and depth of approaches that, with thoughtful selection, could inform an effective process and a quality outcome for MHA.

In the research team’s view, work should begin immediately on developing a pan-Canadian, policy-driven performance measurement framework with key stakeholders. Once a framework is in place, existing strategic indicators that fit that framework can be selected and critical gaps for immediate indicator development work identified. With the necessary capacity for indicator development and reporting, the production of a first collaborative pan-Canadian report on MHA system performance is an attainable goal. Ideally, this would include capacity for a mechanism that enables logically connected and collaborative activities aimed at system improvement.