Methodology:
Development and implementation of the outcome monitoring
Use of outcome monitoring
focusing on participatory research processes
The African
Highlands Initiative (AHI) is an ecoregional programme that focuses on natural resource
management (NRM) research in the densely populated highlands of Eastern Africa.
Since its inception, AHI has made substantial efforts in promoting integrated
participatory research as a more effective approach to the development and
dissemination of NRM technologies by 1) integrating solutions to NRM issues by
adopting participatory and systems approaches; 2) strengthening partnerships
enhancing collaboration and building capacity of a wide range of institutions;
3) improving research through integration of biophysical and social sciences
research; and 4) linking local policy formulation and technology development.
Sustainable use
and management of NRM is essentially about people relating to each other and
their environment in a positive way. Therefore, outcome monitoring can be used
to characterise and assess in detail changes in behaviour of researchers and
farmers as they engage in community-based participatory research activities. At
the inception of the programme in 1995, a framework for performance monitoring
and evaluation was lacking, but it was later introduced in 1999. The innovation
in outcome monitoring is that it makes a shift from assessing only the
technical outputs of research programmes towards focusing on the changes in the
behaviour, relationships and actions of the people and organisations noting
“how” these came about (or not). These contribute and lead to desirable
outcomes.
The methodology
used in the research reported here followed a “participatory learning action
research” approach and involved teams of scientists in National Agricultural
Research Institutes (NARIs) working at eight benchmark sites in five countries
of Eastern Africa. They systematically monitored the outcomes of participatory
research, and its challenges, their experiences, lessons and behavioural
changes that have taken place as they try to apply participatory research
approaches. The outcome-monitoring methodology is used as a means to bring
about the desired changes and is part of the continuous research activities.
Preliminary
results show that the desired changes in the approaches used by research teams
to cope with NRM technology development have been realised in part. Researchers
are focusing on documentation of adoption trends and economic profitability of
technologies but are less engaged in documentation of the participatory
research process, changes in behaviour, and interactions that result
from using the process. Strongly rooted commodity approaches to research and
technology development and dissemination, and scepticism about participatory
research, remain some of the challenges; if not among the researchers
themselves, then in the institutional culture in which they are based.
Additionally, skills and competencies in conducting participatory research and
monitoring of the outcomes are new and developing. Increasingly, partnerships
and other institutional working arrangements among collaborating research and
development (R&D) organisations are influencing the research teams who are
starting to modify their approaches to include community-based research.
Rationale
for outcome monitoring
Analytical
framework of outcome monitoring
The highlands
of Eastern Africa are characterised by medium to high agricultural potential
(producing about 50% of staple foods), but diminishing resource bases. They
constitute about 23% of the total landmass in the region, yet house over 50% of
the population given their suitability to human habitation. Population
densities are already relatively very high (100–200 people per km2),
have risen over the last fifty years within this ecoregion, resulting to
critically small, often fragmented farms reaching 0.25 to 1.0 ha for an average
family of six (AHI 1998). There is a diminishing natural research base due to
declining ability to: maintain and improve soil fertility and erosion control;
intensify livestock feed and nutrient management systems; decrease in social
cohesion and positive arrangements to manage due to policies and increased
competition for scarce resources, distance from markets, lack of inputs and
credit, continued low local wage rates, and land inheritance practices.
Indicators of decline are: lower yields, more pests and diseases of poor
intensification, lowering income, fewer options for diversification, and
lowering general ability to cope (AHI 2001).
Concerns that
technologies emanating from agricultural research in the highland areas had not
yielded results commensurate with investments to improve and sustain
productivity and natural resource base led to the formation of AHI in 1995.
Studies had shown that limited adoption and impact was due to 5 major factors:
To address the
above problems African Highlands Initiative (AHI) was established as an
ecoregional programme focusing on the issues of natural resource management
(NRM) in the highlands of East and Central Africa. AHI operates in eight
selected benchmark sites in five counties (Kenya, Uganda, Tanzania, Ethiopia
and Madagascar). The programme is under the umbrella of the Association for
Strengthening Agricultural Research in East and Central Africa (ASARECA) and is
convened by the International Centre for Research in Agroforestry (ICRAF). It
also forms the East African component of the Global Mountain Programme (GMP), a
global CGIAR programme uniting mountain research. AHI’s guiding philosophy is a
client-driven approach using participatory methods and an effective research
development continuum where research partners, using collaborative, synergic
partnership can bring together diverse contributions to foster farmers’
innovations and collective action for design and dissemination of appropriate,
integrated technologies and methods for improving NRM in the diverse and complex
situation (AHI 1999). AHI’s philosophy and strategy have evolved rapidly
reflecting the dynamic and rapidly changing field of NRM. The current
programme’s outputs are shown in Box 1.
|
Box 1:
AHI’s purpose and five core outputs |
|
Purpose: Small-scale farmers and R&D
agencies have increased capacity to develop, adapt and use innovative
approaches to develop and disseminate technical, social, economic and policy
solutions to sustain and improve agricultural production. |
|
Output 1: Approaches, methodologies and
integrated technologies for participatory NRM research and development
increase the resource users’ capacity to innovate and manage their resources
and agricultural productivity issues in a sustainable way. |
|
Output 2: Selected cross-site research
conducted and syntheses are produced that improve decision making and
priority setting for diverse stakeholders. |
|
Output 3: Strategies for dissemination and
scaling up of NRM technologies and approaches are developed and tested. |
|
Output 4: Selected NARIs, IARCs and other key
partners’ capacity to carry out integrated, participatory NRM research and
development is enhanced across the ecoregion |
|
Output 5: Coordination, management and
synergies are strengthened through strategic partnerships building upon the
collaborative advantages |
Increasingly,
the field of NRM is giving considerable attention not only to the technology
developed but also and more importantly to the process of developing and
disseminating technologies. Consequently, the new focus requires that research
partners not only look at the technologies being delivered as an end in
themselves (THE WHAT), but also seek to understand the processes, strategies
and the means of developing and delivering the technologies (THE HOW) and the
outcomes and impacts of both technologies and approaches (FOR WHOM). In
recognition of these changing paradigms, AHI initiated a monitoring and
evaluation (M&E) system focusing on tracking and documenting processes and
outcomes of participatory research in NRM. The shifting paradigms in M&E
are cognisant of the fact that participatory research and development processes
have to be documented through intensive community-based research, periodic
reviews and reflection and open ended analysis of the research context (Mosse
et al 1998). Understanding the means through which research outcomes are
achieved calls for description and documentation of action and events arising
from planned activities. Looking at the desired outcomes implies that the
stakeholders’ attention goes beyond just monitoring the availability of input
and outputs created as though they are linear in relation, but includes the
process of tracking projects beyond the outputs generated per se.
The need for an
M&E system was brought to the forefront as a result of an internal
evaluation and regional workshop at the end of phase I of AHI. The lack of a
process to monitor and assess progress, changes and outcomes was one of the
weaknesses identified. This meant that research teams were not systematically
collecting and analysing information that provided feedback as to whether or
not they were achieving what they set out to do. Researchers tended to collect
typical technology performance information with less engagement in
documentation of the processes used.
Concerns with
outcome monitoring arose from a number of pragmatic and strategic reasons.
Recent shifts in AHI strategy have given more emphasis to processes and
methodologies development rather than the conventional focus on technology
generation, going back to the major deficits identified in research processes –
leading to poor adoption. Increasingly, participatory research is less and less
concerned about generating deliverable technologies (high-yielding varieties,
soil fertility recommendations, integrated pest management options) but is
becoming more concerned with behavioural and institutional changes necessary
for self-application and/or adaptation of information, materials, etc. to
improve their system which needs to be sustained over time. The focus on
outcome monitoring is justified by the fact that participatory research is
essentially a learning process. Outcome monitoring is therefore an alternative
M&E process that provides stakeholders with timely information about their
progress and achievements for systematic and collective learning, reflection
and corrective action. AHI then specifically sought and received financial
support (in 1998) from the International Development Research Centre (IDRC) to
use participatory research to develop a framework, processes and methods to
enhance M&E of research outcomes in NRM activities.
This paper
analyses and shares some of the preliminary experiences learned for programme
improvement. Resource persons from the International Center for Tropical
Agriculture (CIAT), ICRAF, IDRC Nairobi and researchers from the NARIs at the
eight benchmark sites are facilitating the implementation of the ongoing
M&E work in many ways. These range from literature search, awareness
raising at the various levels of the programme, tools development for initial
testing and further refinement, critical assessment of focus and content of the
M&E aspects, training and facilitation in workshops, and editing of site
reports and workshop proceedings.
Kibel (1999) defines
outcomes as changes in behaviour and interactions of those being affected by
development projects or programmes. Thus, for effectiveness, R&D programmes
must go further than information and technology creation and dissemination
(Kibel 1999, IDRC 1997, Earl et al 1999). Monitoring means systematic
collection, synthesis, storage and use of information about progress and
performance. Therefore, outcome monitoring is a continuous activity that
entails regular gathering and analysis of information. In the process of
collecting and documenting information, outcome monitoring helps researchers in
checking whether inputs, activities and outputs are proceeding according to
plan so that intended outcomes are realised. Therefore, the focus for AHI is on
the behaviours, relationships and actions of the people and organisations with
whom AHI is working with over the last five years.
Research
outcomes are monitored and evaluated in order to assess the extent to which
development actors in projects or programmes have contributed to transforming
and influencing desired changes in behaviour, knowledge, beliefs and relations
among the targeted communities. For example, human behaviour is important in
determining whether newly introduced interventions are being adopted, adapted
and modified to improve livelihoods when undertaking participatory research
activities. Information generated from outcome monitoring enables R&D
actors to make informed decisions and choices for strategic investment and
commitment of resources.
Broadly
speaking, outcome dimensions in development work introduce M&E
considerations to unite intervention processes and desired state. More
specifically, the outcome-monitoring methodology is useful at the planning
stage of research process so that projects set their overall intentions,
strategies and mechanisms for monitoring their contribution to achieving
outcomes and priority changes. By so doing, the stakeholders involved in
development work systematically think about how they intend to achieve results.
Hellawell (1991) describes monitoring as a process of providing information not
results and is a means to an end rather than an end in itself. Thus AHI is
investing in process-oriented research to enable attainment of the regional
purpose. Additionally, monitoring is periodic rather than one-off reassessment
of indicators that are chosen to determine effects of certain interventions, or
policies or changes in general (Abbot & Guijt 1998).
The
implementers of AHI (1999) recommended the introduction of outcome monitoring
as way to track progress. Three strategies were identified as key towards
achieving the desired outcomes (AHI’s purpose) and that were departures for
most researchers and their organisations: interdisciplinary research
(integrated team work), use of a participatory research approach, and stronger
linkages and partnerships with development and policy actors. These are
referred to as the “learning areas” because the programme and the researchers
are interested in assessing experiences in application. Researchers, like most
farmers (Richards 1989, Holland & Silva 2000), do not deliberately
systematise what they learn from the “process experiments”, but if this is
done, will adapt their performance in the light of the results. Hagmann (1999)
indicates that experiential learning is critical among the stakeholders
involved in development interventions so that they adjust their strategies and
context of operation.
Information
needed to monitor achievements in the direction of the desired outcomes was
identified and called “progress markers” also referred to as performance
indicators, which are similar to milestones and enable the users of the
methodology to track progress being made in the integration of the “new”
working strategies in the short, medium and long term. The progress markers are
statements that focus on describing how the behaviour, relationships,
activities and or actions of an individual, group or institution will change
over time in the process of using the new strategies to conduct research.
A key question
is: How will the behaviour, relationships, activities and or actions of
researchers be changed by their interaction and use of “new” AHI strategies?
The progress
markers describe what one would expect to see the stakeholders doing if
they paid attention to the AHI strategies, to what it would like to see
them actually doing, to what it would love to see them doing, thus
describes a pattern of behavioural changes taking place over time to reach the
desired state. Earl et al (1999) states that, “expect to see” progress markers
indicate passive learning by the stakeholders and are easy to achieve. The
progress markers that indicate more active learning or engagement are listed
under “like to see” category, while those markers that are transformative and
more difficult to achieve are listed under “love to see” (Appendix 1).
Over the last
three years, AHI has made substantial effort to build researchers’ capacities
in the use of participatory research approaches, in multidisciplinary teamwork
and in managing multi-institutional linkages, so that researchers can improve
the ways they interact amongst themselves, with farmers and other development
partners for the betterment of the farmers. The hypothesis being that
ultimately, there will be better adoption, feedback to research and better
returns to investment for solving agricultural productivity and NRM problems.
Therefore, tracking the progress made in these areas – in particular, how they
contribute to better implementation of participatory research process – has
been a critical component of the regional programme. Participatory research
processes entail the involvement of the relevant stakeholders in all the stages
of research. Ashby et al (1989) explain that participatory research is a
process “in which the farmer acts as a subject which investigates, measures and
studies in collaboration with researchers.” For the researchers to allow space
for the farmers to become involved in the research process, it means that
researchers have to change from the conventional ways of conducting research
where they consider themselves as “experts” to one where farmer innovations and
knowledge is valued and their involvement takes priority. This echoes peoples’
involvement in development process as being important so that they transform
their lives for their own benefit.
Regional
synthesis and scaling up
AHI uses
benchmark sites that were selected geographical areas for integrated,
participatory research concentration in the respective countries. Research
activities are undertaken with teams of national scientists in collaboration
with government line ministries, non-governmental organisations (NGOs) with
some input from International Agricultural Research Centres (IARCs) and
university scientists.
The development
and implementation of outcome monitoring followed a number of iterative steps.
First a regional workshop was organised with many of the implementing
stakeholders in AHI to develop a common understanding and definition of
mission, focus and purpose of AHI from the point of view of different
stakeholder. The core outputs of AHI as well as crucial questions for the
performance evaluation were developed and agreed upon by all the relevant stakeholders
Using the
regional workshop output, a small group of resource persons (from ICRAF, IDRC,
CIAT and AHI) was formed and consulted with NARI stakeholders to select the key
strategies or “learning areas” and to develop tools for monitoring these priority
learning areas: interdisciplinary teams, participatory research and
multi-institutional linkages. (Although distinctly handled, the three areas are
interrelated.) It was decided to be selective in the areas to monitor and to
start with one stakeholder group (researchers) to do the monitoring, given the
newness of the outcome-monitoring process and the recognised need to develop
and test methods first. It should be noted that the implementers (researchers)
had been exposed to these areas through training courses or workshops, and that
these were deemed key to AHI’s success. All these areas have stakeholder and
gender analysis embedded in them. An action plan was then developed and
reviewed by an AHI M&E working group (ICRAF, ILRI, CIP, IDRC, KARI) and by
AHI’s regional Technical Support Group (all site coordinators and some
representatives of the various AHI working groups).
Subsequently,
in-country and site workshops were organised where possible in conjunction with
annual planning to familiarise the site teams with the newly developed AHI
framework, to start to build a conceptual base for understanding M&E in a
new context, and to further develop strategies and steps for testing, adjusting
and institutionalising the M&E framework. All in all there were eight
workshops with 112 total attendees over a period of 18 months to date.
A first
workshop was held during the Kabale site team’s annual planning in 1999, where
monitoring participatory research comprised three impact areas: technology
outputs, participatory research process and outcomes in terms of behavioural
change (see Table
2). Based on their specific research protocols and activities, the
site teams defined their performance questions and identified performance indicators.
The framework also specified the types of data or information needed, who is
responsible and timeframes.
Then, it was
decided that the three strategies should become focal points (as means to the
end) rather than technology generation and dissemination itself. New tools were
designed to focus more on tracking desired changes in the three learning areas.
These were tested in the two test sites (Kakamega and Lushoto) and then
incorporated into site workshops held in other countries.
Thus, to start
off the monitoring process, each of the three learning areas (strategies) were
analysed by the researchers looking at the changes in the following: (i)
current status and experiences; (ii) their perception of the benefits and
shortcomings; (iii) practical examples of the effects of using the approach
(strategy) on their behaviour, interactions and research; (iv) suggestions on
how they, as research teams, can be assisted to improve on the learning areas
(approaches); and (v) future plans for using the approach (see Table 1).
Table 1: An example of the outcome-monitoring
tool
|
Learning area |
Status review |
Benefits |
Shortcomings |
Changes in behaviour |
Improvements needed |
Lessons |
|
Interdisciplinary
research |
|
|
|
|
|
|
|
Participatory
research |
|
|
|
|
|
|
|
Multi-institutional
linkages |
|
|
|
|
|
|
Table 2: An example of an M&E framework for
participatory research evaluation of climbing bean varieties disseminated in Kabale
(Southwestern Uganda) by December 2000
Impact categories
|
Indicators |
Information needs |
By whom |
Technical
At least 3
varieties being produced in farmers’ fields Increase
in yield per unit area Multipurpose
trees planted Recommended
practices adopted |
At
least 60%of target farmers grow one improved variety Target
farmers increase yield by 1500 kg/ha At
least 40% of target farmers grow multipurpose trees |
Seasonal
reports File
sampling and discussions |
Principal investigator Farmers NGOs in Kabale |
Process
Seed
multiplication Farmer
selection Farmer
training Tree
nursery establishment Follow-up
visits |
4 well-established seed multipliers Volunteers identified Curriculum developed 4 well-established nurseries Visits organised |
Farm
records Farmer
registry Training
booklets Field
reports, visitors book |
Researchers,
farmers Researchers,
extension, farmers Researchers Researchers Researchers,
extension |
|
Outcomes (behavioural changes) Farmers positive
on growing climbing beans Farmers
willing to pay for climbing beans Farmers
plant beans in fertile portions of their land Farmers
re-use and buy stakes for the beans Researchers
hold joint consultative meetings Researchers
and other stakeholders organise joint monitoring visits to farms Farmers
conduct experiments on own Farmers
adapt technologies proposed by scientists Increased
autonomy to engage in research options |
Enhanced
knowledge and positive attitude to growing climbing beans Rapport
among stakeholders |
KAP
(knowledge, attitude + practice) survey PRA Observation Quality
of reports Case
study |
Principal
investigator, farmers, extension Site
coordinator, researchers Researchers,
farmers, extension |
NB: The researchers in the site planning meeting developed this
framework. The above example represents one of the activities that the
researchers were conducting in the Uganda site of Kabale.
Although using
this tool was the main component of the first meetings, other sites requested
for more time to be spent on improving their understanding of participatory
monitoring and evaluation and the underlying concepts. This was built into
these workshops along with using this tool. In many of these sessions participants
share information and experiences about participatory research. Facilitators
have been used to help build conceptual understanding and arrive at more common
understanding of the concepts and to ensure a “harvesting” of ideas and inputs.
Occasionally, a knowledge assessment form has been used to assess the knowledge
levels of the participants in participatory research and outcome monitoring.
The participants are then taken through a SWOT (Strengths, Weaknesses,
Opportunities, Threats) session to gauge their experience and to draw lessons
for decision-making by the programme on the areas for improvement and
modifications.
The workshops
and meeting process have diverse methods employed: plenary discussions, groups
work with specific tasks, two-person buzz groups to define concepts, feedback
sessions in plenary, process group session that look at what went well, what
did not go well and suggestions for improvement in future. The workshops end by
the participants developing an action plan for follow-up and meta-evaluation of
the workshop process and content.
Examples of the
M&E tools and formats developed during the learning workshops as listed
below.
The information
gleaned from the workshops has been compiled and discussed with researchers in
order to design the next set of tools. In the second workshop in Western Kenya,
the resource people shared the output from the initial workshop and developed a
monitoring plan to follow up on the integrated multi-disciplinary teamwork
aspect to start with (see Table 3). Researchers were encouraged to try
new ideas and modify the tools to suit their information needs.
Table 3: Participants’ Action Plan from an
M&E
Name of the site ______________________________
The ultimate objective of the workshop is the
application of tools, concepts, lessons learned during the workshop period in
research activities and future plans. The action plan is a tool that that helps
the researcher consider specific applications of the tools learned in the
ongoing research activities.
1. What aspects of the workshop do you realistically
think are possible to integrate in your research activities and future plans?
|
Activities
(Indicate O if ongoing or N if new) |
What tools,
concepts & lessons learned are you going to apply? |
Reasons |
What new
information do you expect? |
How do you
intend to use the new information? |
Timeframe
(indicate when you plan to use it) |
Responsible
and collaborator |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. What are the
steps needed to execute your action plans?
3. What are the potential constraints you are likely
to encounter in the implementation of your action plan?
4. What are the technical aspects of the workshop
requiring strengthening or new areas for future learning?
|
Areas
requiring strengthening |
New
learning areas |
|
1. |
|
|
2. |
|