The Evaluation

What evidence do we have that community-led transparency and accountability can improve health services and health outcomes?  What do we know about how transparency and accountability works, the mechanisms that are most effective, and in what contexts?  Designed in parallel with the T4D intervention, the integrated mixed-method T4D evaluation was developed to answer not only questions of whether transparency and accountability works – but also how. 

 

Evaluation Design

The T4D evaluation was designed to gather evidence at every step of the theory of change – from the CSO implementation of the intervention to community empowerment and social action, service delivery changes, and ultimately maternal and newborn health outcomes.  To understand the full picture of how the intervention works in practice, we employ the following evaluation techniques:

  • Randomized controlled trials.  We will utilize baseline and endline quantitative data collected from 100 randomly selected treatment villages and 100 randomly selected control villages in each country to analyze whether this community-led transparency and accountability intervention has an impact on healthcare decisions, health service quality, health outcomes, and citizen empowerment.
  • Quantitative and observational data from the intervention.  The success of the intervention is likely dependent on several intermediate outcomes and outputs that are immediately observable, such as community participation in intervention meetings and design of social actions that match actual health problems.  Further, we hypothesize that there are identifiable relationships between specific context factors, social actions undertaken by the community, and the success of those actions.  To better understand these relationships and intermediate outcomes, we will collect structured information on participation, reactions, and signs of empowerment during intervention meetings through external observation and CSO facilitator assessments.
  • Key informant interviews and analysis of social action plans.  Using key informant interviews of citizen intervention participants and informal and formal leaders, we will assess the social actions undertaken and the perception of these actions by key actors.
  • Ethnography.  Finally, to provide a full and nuanced picture that allows us to reveal additional important factors in the success of the intervention, we have placed trained ethnographers in several villages in each country for a period of 6-12 months to study the actions and experience of those undertaking and targeted by the intervention. 

This evaluation design builds on an adaptive learning model used in the iterative pilot for the project (findings forthcoming). 

 

Research Questions

The specific questions that we will answer in this phase of the T4D evaluation are: 

  • What is the effect of the community-led transparency and accountability intervention on the utilization of health care services related to maternal and newborn health?
  • What is the effect of the community-led transparency and accountability intervention on the content of health care services related to maternal and newborn health?
  • What is the effect of the community-led transparency and accountability intervention on health outcomes?
  • What is the effect of the community-led transparency and accountability intervention on citizen empowerment and efficacy, both perceived and actual? 
  • If there are significant effects, what are the mechanisms through which these effects occur?
  • What is the role of context in shaping or determining these mechanisms?

 

Evaluation Materials

Materials related to the evaluation in Tanzania and Indonesia are available here.