The advice presented here rests on a combined logic and implementation analysis. Both evaluation approaches aim to assess the potential value of a given intervention but with a different focus.

On the one hand, "Logic analysis is an evaluation that allows us to test the plausibility of a program’s theory using available scientific knowledge—either scientific evidence or expert knowledge" (Brousselle & Champagne, 2011, p. 70). On the other hand, implementation analysis relies mostly on empirical observations to identify factors that actually enhanced or impeded the implementation of the intervention or the production of its effects (Champagne et al. 2001). Combining these two evaluations approaches allows building a comprehensive understanding of the factors related to the intervention and the contextual characteristics. In turn this allows providing evidence-based advice to optimize the intervention and hence, maximizing its effectiveness. Furthermore, implementation analysis helps in identifying which determinants of intervention's effectiveness are more important than others while putting the intervention in place.

At the operational level, the research team firstly, conducted a logic analysis on nurse practitioner deployment and practice based on a realist review of the literature and on expert advice. Secondly the team conducted an implementation analysis using a case study research design (n=6) in three health regions of Quebec. The evidence derived from both logic and implementation analysis was then integrated as practical advice pertaining to five core themes that structure the NP integration process.

Logic Analysis

Logic analysis is a three-step approach (Brousselle & Champagne, 2011; Rey et al., 2012): Building the logic model, developing the conceptual framework and evaluating program theory. The logic model was built using grey literature and consulting experts from the Health Ministry that participated to the elaboration of the NP program in Quebec. We then used a realist review approach (Contandriopoulos et al., 2010; Greenhalgh et al., 2005; Pawson, 2006; Popay, 2006) with the aim at understanding the factors and mediating variables that influence the effectiveness of a complex organizational level intervention.

Practically, the starting point was the advanced-practice nursing database (Advanced Practice Nursing (APN) Literature Database part of the work of the Canada Research Chair in Advanced Nursing Practice held by Alba DiCenso at McMaster University) which systematically compiled all the scientific literature published on advanced practice nursing between 2000 and 2009. We relied on the theme structure of the database to select 3,674 references and manually assessed their relevance based on titles and abstracts. To be retained, documents had to address nurse practitioner deployment, practice models, or integration process. This allowed the selection of 159 documents, which were then independently reviewed by two members of the research team. Documents were summarized using an abstraction sheet and given a relevance score (1 to 3) and a scientific validity score (1 to 3). Only documents with a combined score (relevance and validity) higher than 4 were retained. This allowed the selection of 43 documents as primary sources for analysis. In a second phase, the search syntax used by the Research Chair on Advanced Nursing Practice to compile the APN Literature Database was reproduced in the health search engines (MEDLINE, CINAHL, and EMBASE) to identify articles published between 2010 and 2012. The same sorting methods were applied to this second corpus of articles, and 53 articles were retained for full textual analysis. Of these, 15 were added to the 43 documents selected in the first phase. Overall 58 documents were selected for in-depth analysis through systematic processes.

The documents where then iteratively read, often several times, and analyzed to build a preliminary conceptual model according to the realist review approach (Contandriopoulos, et al., 2010; Greenhalgh, et al., 2005; Pawson et al., 2005; Popay, 2003). The model's was focused on structuring available evidence on practical advice to support PHCNP integration. Five major themes were selected as core elements to structure the advice: planning the integration, role definition, patient management, collaboration and support to the team. For each theme, a draft narrative summary of the information collected in the systematic review was produced. At this point the team's interdisciplinary expertise (see Team section) was also used to identify new documents on an ad-hoc basis. The draft narrative summary on each theme was then used as the analytical framework for the implementation analysis.

Implementation analysis

The second data source upon which the advice provided here is based is derived from an implementation analysis through 6 qualitative case studies. Each case was defined as a clinical team into which one or more PHCNPs have been integrated in Quebec. Potential cases were identified in collaboration with Quebec's Ministry of Health and with the Regional Health and Social Services Agencies involved. Cases selection was based on two criterions. The first was to be a team where PHCNP integration was successful in order to identify and analyze functional integration models. The second criterion was based on the aim of maximizing variation in regard to the environment (rural, suburban, urban) the organizational setting (GMFs, CLSCs or UMF, (see Preamble)) and the stage of the PHCNP integration. Analyzing the same intervention in varied contexts improves the external validity of the findings (Denzin, 1978; Patton, 2002; Yin, 1989).

The case studies are based on semi-structured interviews conducted with members of the clinical teams and other key actors as well as analysis of available documentation. A total of 30 interviews were conducted. All informants gave informed consent and best practices regarding the ethical conduct of research were followed (see Ethics section). Data was again analyzed according to an iterative process. Each one of the six cases was under the responsibility of one team member. This researcher produced a narrative case summary structured around the five themes identified in the literature review. Those summaries where discussed in team meetings and cross-case insights were identified. In a second step, six narrative theme-based cross-analysis of the cases were produced in order to synthetize the contribution on empirical case-study evidence on each theme.

Combined analysis

After the completion of the implementation analysis, we revised the summaries built during the logic analysis phase to integrate empirical knowledge derived form the case study analysis. Ultimately, the evidence gathered from the logic analysis and from the implementation analysis was integrated into one single theme-based narrative and the specific expertise of each team member mobilized in the process. The objective of relying on two different approaches was threefold. Firstly to be able to cross-validate and inform the analysis by confronting the evidence derived from the literature to actual practices and reciprocally. Secondly, the objective was to assess the applicability and usefulness of the literature-based advice in real-world contexts and to identify the most important factors in supporting the implementation process of primary care NPs among those identified in the literature. Thirdly, to better appraise the dynamic of implementation, and understand how factors are intertwined during the implementation process.

Three important elements in the final integration of the material also need to be emphasized. The first is that the focus was to offer practical advice. A lot of the literature is structured on the identification of barriers and facilitators (Clarin, 2007; de Guzman et al., 2010; Gould et al., 2007; Poochikian-Sarkissian et al., 2008; Rapp, 2003) but often offer little to help teams to improve. It might also be worthy to stress that integration is a process and thus a dynamic phenomenon but that the identification of a list of barriers and facilitators is a very static analytic framework. We will get back to this idea later on. The second is that we aimed for a single set of theme-based recommendation for all team members, whether they are NPs, RNs, MDs, administrators or support staff. Finally, even within our cases all sampled in one Canadian province, the organizational parameters (ownership, size, staffing, services offered) were highly variable and the combined approach of logic and implementation analysis maximizes the generalizability of results beyond Quebec's or Canada's boundaries.

Back to top

For more information:

Brousselle, A., & Champagne, F. (2011). Program theory evaluation: Logic analysis Evaluation and Program Planning, 34(1), 69–78.

Clarin, O. A. (2007). Strategies to overcome barriers to effective nurse practitioner and physician collaboration. Journal for Nurse Practitioners, 3(8), 538-548.

Contandriopoulos, D., Lemire, M., Denis, J.-L., & Tremblay, É. (2010). Knowledge Exchange Processes in Organizations and Policy Arenas: A Narrative Systematic Review of the Literature. Milbank Quarterly, 88(4), 444-483

de Guzman, A., Ciliska, D., & DiCenso, A. (2010). Nurse practitioner role implementation in Ontario public health units. [I]. Canadian Journal of Public Health.Revue Canadienne de Sante Publique, 101(4), 309-313.

Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods. New York: McGraw-Hill.

Gould, O. N., Johnstone, D., & Wasylkiw, L. (2007). Nurse practitioners in Canada: beginnings, benefits, and barriers. Journal of the American Academy of Nurse Practitioners, 19(4), 165-171.

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005). Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review Social Science & Medicine, 61(2), 417-430.

Patton, M. Q. (2002). Qualitative Research & Evaluation Methods (3rd ed.). Thousand Oaks: Sage Publications.

Pawson, R. (2006). Evidence-based Policy: A Realist Perpective. London: Sage.

Poochikian-Sarkissian, S., Hunter, J., Tully, S., Lazar, N. M., Sabo, K., & Cursio, C. (2008). Developing an innovative care delivery model: Interprofessional practice teams. Healthcare Management Forum, 21(1) (pp 6-11), ate of Pubaton: Srng 2008. doi: Periodique - UdeM

Popay, J. (2003). Moving beyond effectiveness in evidence synthesis. Methodological issues in the synthesis of diverse sources of evidence. London: National Institute for Health and Clinical Excellence.

Popay, J. (Ed.). (2006). Moving beyond effectiveness in evidence synthesis: Methodological issues in the synthesis of diverse sources of evidence. : NICE: National Institute for Health and Clinical Excellence.

Rapp, M. P. (2003). Opportunities for advance practice nurses in the nursing facility. Journal of the American Medical Directors Association, 4(6), 337-343. doi: McGill

Rey, L., Brousselle, A., & Dedobbeleer, N. (2012). Logic analysis: testing program theory to better evaluate complex interventions. In J. Houle, N. Dubois, S. Lloyd, C. Mercier, Z. Hartz & A. Brousselle (Eds.), L'évaluation des interventions complexes (Vol. 26(3), pp. 61-89): Revue Canadienne d'Évaluation de Programme/ Canadian Journal of Program Evaluation.

Yin, R. K. (1989). Case study research: Design and methods (Revised ed.). Newbury Park: Sage.

Back to top