Evaluating success to accelerate impact

December 14, 2020

Evaluations can uncover why and how health programs work. By identifying the factors that contribute to success, we can adapt our approach, improve our work, and increase our potential impact.

Public health programs are increasingly complex and interconnected—making success harder than ever to measure.
Implementation science uses research methods to paint a picture of not only what happened, but why.
Bangladesh, DRC, Guatemala, Mozambique, Senegal, Uganda, Zambia
Gavi; Global Fund; IHME
Path Expertise Areas

Expertise in evaluation design and methods

Focus on strengthening in-country evaluation platforms

Commitment to learning

Data Analysis and Use

Tolbert Shaba, Acting District Medical Officer, Kazangula District Medical Office. Photo: PATH/Gabe Bienczycki

PATH works with local experts like Medical Officer Tolbert Shaba, to understand how and why public health programs work. Kazangula District, Zambia. Photo: PATH/Gabe Bienczycki.

The challenge

Understanding why a public health program works (or doesn’t) is a complicated task. In many cases, governments, implementing partners, and donors are all working together to address the same health challenge through a variety of related and complementary projects and programs.

Governments and donors have a vested interest in understanding how these programs work. How do they improve health outcomes? And what barriers are keeping them from even greater health impact?

“Accelerating impact requires real understanding of each rollout strategy, partnership, delivery mechanism, and behavioral change involved.”
— Emily Carnahan, Manager, Monitoring, Evaluation, and Learning at PATH

As new innovations are implemented, governments, donors, and partners need information to adjust in real time, reflect on successes and challenges, and improve design and scale-up of programs.

The solution

Implementation research (sometimes called implementation science) considers how our work translates from initial plans and ideas into real world health impact. By looking across the full spectrum of implementation—inputs, activities, outputs, outcomes, impact—using a suite of data sources and methods, implementation research helps us learn what is going well, what we can do better, and how we should continue to evolve.

Over the last seven years, PATH and partners have implemented prospective assessments in Bangladesh, the Democratic Republic of the Congo (DRC), Guatemala, Mozambique, Senegal, Uganda, and Zambia.

Gavi Full Country Evaluations

In partnership with the Institute for Health Metrics and Evaluation (IHME) and working with evaluation partners in Bangladesh, Mozambique, Uganda, and Zambia, we designed prospective evaluations (2013–2018) to support ongoing learning and improvement of Gavi and national immunization programs.

The aim of these evaluations was to identify drivers of vaccine equity and coverage.

The Gavi Full Country Evaluations consider many drivers of vaccine equity and coverage.

The Gavi Full Country Evaluations consider many drivers of vaccine equity and coverage. Source: How to evaluate the implementation of complex health programmes in low-income settings

The findings and recommendations from the Full Country Evaluations have been used by Gavi and country governments to refine their approach. For example, Gavi revised its policies around the introduction of human papillomavirus (HPV) vaccines to increase learning to inform scale-up.

Coverage of the DPT3 vaccine by district in the three Gavi Full Country Evaluation countries. Source: Evaluation Findings Brief.

Coverage of the DPT3 vaccine by district in the three Gavi Full Country Evaluation countries. Source: Evaluation Findings Brief.

Global Fund Prospective Country Evaluations

The Global Fund Technical Evaluation Reference Group commissioned a set of independent evaluations to guide the development and implementation of Global Fund investments in eight countries. PATH and its partners are conducting these evaluations in DRC, Guatemala, Senegal, and Uganda (2017–2021).

These evaluations create timely evidence to support program improvements and accelerate progress.

“In DRC, timely exchange of findings with the national malaria program helped resolve bottlenecks and improve data quality.”
— Dr. Salva Mulongo, Technical Coordinator, PATH

Findings have influenced the work of the Global Fund and its partners. These evaluations provide in-depth knowledge of the complexities of implementation that country stakeholders can use for learning and improvement. Comparing evidence across countries helps identify common implementation bottlenecks for the Global Fund to address through changes to its policies and processes.

Framework of the PCE’s Thematic Areas

In line with the objectives of the Global Fund 2017–2022 strategy, the evaluation is anchored in several key thematic areas. Source: Prospective Country Evaluation overview.

Why was PATH chosen to do this work?

PATH provides experience and expertise to conduct rigorous, high-quality evaluations:

  • Expertise in evaluation design and methods.
    PATH has deep technical expertise in conducting evaluations—from determining research questions and evaluation design to mixed-methods data collection and analysis.
  • Focus on strengthening in-country evaluation platforms.
    PATH has strong relationships with both global and in-country evaluation partners to foster country ownership and strengthen capacity for implementation research.
  • Commitment to learning.
    PATH continually strengthens and refines our approach to respond to changing needs and emphasize continued learning.

Our approach

So how do we do it? PATH combines a variety of research approaches that generate both quantitative and qualitative evidence. This mixed-methods approach ensures a breadth of evidence and a way to validate our findings.

Although the evaluations for Gavi and the Global Fund were multi-year, multi-country efforts, there is a growing need for fit-for-purpose, rapid evaluations in global health. PATH developed Rapid Evaluation, Action, and Learning Toolkits that allow us to apply implementation research principles in a flexible and rapid way through root cause analysis of barriers to implementation, followed by designing, testing and refining solutions.

“Our analysis provided the Senegal Ministry of Health with rapid feedback that led to corrective action and improved performance.”
— Ida Ndione, Senior Program Officer, PATH Senegal

PATH is committed to continual learning—including capturing lessons about how we learn. Sharing lessons about our evaluation process is another way to advance high-quality implementation science in public health.

Reflecting on the Gavi and Global Fund evaluations, PATH has shared these lessons:

  • Engage stakeholders early.
    Engaging governments, donors, and partners from the beginning leads to stronger evaluation design and greater participation.
  • Balance objectivity and learning.
    Including research components in ongoing public health projects creates opportunities for data collection and adaptive management—but implementation researchers should also ensure a level of objectivity to make findings more robust.
  • Design for flexibility.
    Public health is constantly changing, so evaluations should be designed with flexibility to meet changing needs and circumstances.
  • Build research teams intentionally.
    Team members that bring different skills and experiences—complemented with the right tools and methods—expand evaluation options and strengthens finding and recommendations.
  • Communicate the strength of evidence.
    In mixed-methods research, some evidence will be more rigorous than others. When translating evidence into recommendations, noting the strength of the evidence is an important guide for decision-makers.

PATH will continue learning, adapting, and evaluating in our efforts to strive for greater health equity.

Partners in the pursuit

In addition to PATH, the evaluation teams for the Gavi Full Country Evaluations and the Global Fund Prospective Country Evaluations included a coalition of organizations:

  • Institute for Health Metrics and Evaluation (IHME), University of Washington (USA)
  • International Centre for Diarrhoeal Disease Research (icddr,b) (Bangladesh)
  • Universidade Eduardo Mondlane, Health Alliance International, Manhiça Health Research Centre, and National Institute of Health (Mozambique)
  • Infectious Diseases Research Collaboration (IDRC) (Uganda)
  • University of Zambia (Zambia)
  • Centro de Investigación Epidemiológica en Salud Sexual y Reproductiva (CIESAR) (Guatemala)
  • Institut de Santé et Développement/Université Cheikh Anta Diop (ISED/UCAD) (Senegal)

Gavi, the Vaccine Alliance, and the Global Fund to Fight AIDS, Tuberculosis and Malaria provided support for the design and implementation of these evaluations.

Read more