Meeting Summary
Advancing the Science of Implementation
October 28, 2004 – October 29, 2004
Washington, DC
There are major challenges in the mental health services research field, namely those that relate to implementing interventions into real-world practice. These include: 1) Building interventions that are more service-oriented, 2) Understanding optimal construction of interventions, 3) Improving the likelihood of implementation during intervention testing, and 4) Improving the science of implementation. Despite efforts to improve mental health services through the development, testing, and implementation of treatment and prevention interventions, a gulf remains between scientific discovery and service delivery. While much has been written about reducing the gap between science and practice, few tangible changes have occurred in the way intervention development and implementation have been offered. To respond to these challenges, the National Institute of Mental Health organized a workshop on October 28th and 29th, 2004, in Bethesda, MD, entitled, “Advancing the Science of Implementation: Improving the Fit Between Mental Health Intervention Development and Service Systems.” The purpose of the workshop was to (1) identify organizational theories, measures, and constructs relevant to implementation research; (2) discuss the likely fit of mental interventions with the service systems; (3) build integrative research designs around implementation of evidence-based treatments into intervention trials; and (4) build capacity in the field around multi-level models of intervention implementation.
Over the two-day workshop, a multi-disciplinary group of researchers, implementers, and clinicians was brought together in an “engaged scholarship” * format composed of small and large-group settings to discuss the development of a sound knowledge base on the implementation of evidence-based practices and integration of constructs across different types of interventions. Using three specific intervention categories—team-delivered interventions (e.g., Assertive Community Treatment), individual therapeutic interventions (e.g., Cognitive Behavioral Therapy for Trauma in Children and Adolescents), and medication management interventions (e.g., Medication Algorithms for Depression)—participants identified constructs seen to be important to the implementation of the model in real-world systems.
Following each breakout session, attendees reconvened for a full group discussion and brief presentations were conducted to highlight interventions in the areas of organizational measures, social network analysis and field opportunities.
Meeting Discussion
The first part of the meeting consisted of three different breakout groups to discuss the evidence about the implementation of interventions. Each breakout group identified lessons learned from efficacy/effectiveness research and real-world implementation about key constructs that impact the implementation of interventions, key organizational constructs that may relate to the adaptation of interventions during implementation, and ways of determining how available theories and measures can enable the inclusion of key constructs related to implementation. The breakout groups focused on the articulation of multiple levels of influence on the implementation of interventions (e.g., client, provider, state levels). The groups agreed that constructs at each of these levels were important and interdependent, and there was a need for collaboration with non-mental health service systems. Finally, the groups mentioned the need to develop an implementation framework that establishes the key non-clinical strategies and outcomes for implementation, and to identify data sources to address the influences at each level, as well as across multiple levels.
The second part of the meeting began with a presentation that focused on organizational measures used within multiple industries to assess different features related to the implementation of interventions. The review of measures included recent studies (since 1998), with search terms of “innovation,” “diffusion,” “culture,” and “innovation.” The results were then narrowed to eliminate innovation that was done “in house,” as it lacked information about how to incorporate external ideas. Most of the studies were based on adopting new technologies. The presentation reinforced the importance, above all, of ensuring that organizational measures aren’t just “taken off the shelf,” used as a quick fix and plugged into a study, but are carefully examined and tested for their appropriateness. The presentation followed with some reflections on the variety of measures used and how they might be relevant to implementation of mental health interventions.
A subsequent presentation provided a brief introduction to the use of social network analysis to look at relationships within and between organizations. The contribution of a social network perspective is that individuals are seen not in isolation, but in the context of the relationships in which they are embedded. “Actors” in a network can be individuals, groups, organizations, etc. “Ties” in a network can be communication, responsibility, trust, advice, workflow, resource allocation. The presenter posed questions to facilitate the discussion in determining what role the individual, team and organization play in the adoption of innovations. The three small group discussions centered around identifying measures that would help to augment current intervention efficacy and effectiveness designs, addressing how system-level constructs impact study design, highlighting research questions related to implementation that can be asked within existing trials, and identifying questions that require separate research studies. The discussion topics included designing a study to disseminate interventions, identifying major principles of a research (approach) protocol for implementation and corresponding measurement variables, focusing research on factors that distinguish adopters from non-adopters and organizational culture, and determining whether implementation could be viewed as a separate, yet strong component of effectiveness.
Lessons Learned and Next Steps
The final portion of the meeting provided an opportunity to hear about some successful models of implementation research, to highlight lessons learned and to propose recommendations/next steps for building capacity to advance the science of implementation research. Participants learned how mental health services researchers have been able to increase the relevance of research on interventions by working within real-world clinical settings and partnering with practice organizations to conduct research. The presenters described how the research team approached dissemination research as a flexible, collaborative process; and highlighted current work involved in measuring organizational factors promoting adoption and sustainability of interventions. The presenters concluded with a discussion regarding the lessons learned over the course of the center’s previous work in implementation research, including:
- Researchers should consider ways to address stakeholders’ need for control over the interventions and identify ways to use data on stakeholder priorities and resources as a guide for implementation and evaluation.
- It is important to have realistic expectations of organizational and client outcomes based on experience and evidence, as well as appropriate theoretical and conceptual context for the population, disease condition, service settings, and treatments.
- Partnership in organizational assessment and intervention implementation is feasible with diverse stakeholders, but each stakeholder has unique issues and adoptions are largely up to them. Impacts of organizational variation on adoption and effectiveness can be lessened by technology, flexibility, and partnership models.
Finally, the broad group discussion focused on what Federal agencies and researchers in the field could do to build capacity for conducting research on implementation, which is still a developing field with relatively few people engaged in studies. First, a review of current opportunities were presented, potential actions at the Institute were provided, followed by participant views of what the field could do to advance the science of implementation. In addition, several proposed action steps for continued dialogue with workshop participants were highlighted. Some examples included:
- Current Institute Contribution (e.g., Conference Grants-R13; Training Institute-R25; Exploratory Development Research on Implementation-R34; and Standing Program Announcement on Research Methods, Dissemination and Implementation)
- Potential Institute Contribution (e.g., supplements to existing studies to conduct implementation research, coordination of review committees to ensure appropriate expertise on scientific panels, and collaboration across NIH Institutes to complement their new mechanisms.
- Field Capacity Development (e.g., involve young scholar groups in organizational behavior in health care, provide experiential training to graduate students and conduct a dissemination and implementation research conference)
- Research Design and Measurement (e.g., measuring the success of implementation, development of a measurement system in service settings, and establishment of a process for sharing and/or accessing available measures)
- Research Methods and Models (e.g., measurement of organizational factors relevant to implementation, development of collaborative research centers, involvement of organizational experts in consultation, and the involvement of statisticians earlier in the research development process).
Program Contact:
Footnote
* Engaged Scholarship is defined as a collaborative form of inquiry where researchers and practitioners leverage their different perspectives and competencies in producing knowledge about a complex problem in reality. The framework views science-practice gap as not merely a knowledge transfer problem, but a knowledge production problem, in which knowledge applicable to both science and practice worlds is developed. It addresses the dual challenge of enabling academics to put theories into practice, and managers to put practice into theory. (Van de Ven and Johnson. 2005 Acad Mgmt Review, In Press)
