Document(s) 14 of 24

Introduction

INCREASING INTEREST AMONG DONORS in linking natural resource management with poverty alleviation has motivated concerns to fund research that builds on experiences of non-governmental organizations (NGOs) in participatory technology development. The goal of such research is to develop sustainable agricultural technologies appropriate for heterogeneous environments occupied by poor farming communities. Another research objective is to improve understanding of the processes involved, their outcomes and wider applicability. Partners in this type of research project are often state research institutions or extension departments, which are typically bureaucratic but are wide reaching. The research is likely to involve a range of stakeholders, especially when the research is initiated by external actors responding to identified local needs. Stakeholders include donors, researchers, local institutions, and farmers with particular interests or direct involvement in the research. Stakeholder interests and the state institutions that support the research become significant factors influencing the process of developing agricultural technologies, monitoring performance, and evaluating impacts. For instance, donor interests pressure researchers to produce generalizable results of farm trials in order to apply research outputs more widely.

Another important factor affecting technology development research is in the nature of the technology itself. Previous attempts to involve farmers in the evaluation of technologies generally dealt with simple change, for example, evaluating the impact of introducing a range of crop varieties (e.g. Ashby, 1990; Joshi and Witcombe, 1996). However, many of the technologies appropriate for sustainable or low-input agriculture involve more complex changes that affect management of the whole farming system.² One implication for monitoring and evaluation (M&E) involving more complex changes is that it may be difficult for one farmer to compare several interventions simultaneously, because systems changes as a result of introduced technology may affect the whole farm.

This chapter describes two projects – one in Bolivia and the other in Laos – that deal with these particular challenges in monitoring and evaluating impacts of new technology: those of addressing donor needs, state – institutional contexts, and farming system change. The projects were collaborative initiatives carried out by local research institutions, external researchers, NGOs, people's organizations (POs), and individual farmers (see Box 4.1).

The project context in Bolivia and Laos

Each project is characterized by its environmental diversity and the lack of available technology appropriate for the ecological zone (see Table 4.1).

The complexity of the local environment led researchers to propose a participatory approach for developing farming systems based on farmers' own knowledge and priorities. The project methodology adopted is similar for both projects, which facilitate a two-way learning process between farmers and technical researchers, and acquire government institutional

Table 4.1: Features of the projects in each country

support in designing trials based on local and scientific knowledge. The projects differ in terms of:

• their institutional contexts
  
  
• their project design (one is oriented towards production and the other towards resource conservation)
  
  
• ease of communications
  
  
• contact between local research institutions and NGOs (see Table 4.2).

Table 4.2: Characteristics of the partner research institutions in each country

Appreciating knowledge of men and women

Gender issues remain a neglected aspect of rural development in both cultures. External researchers were concerned to highlight gender issues and recognize the different perspectives of men and women. In Laos, after a situation analysis was conducted, local government staff realized that several different opinions existed within the community, especially between women and men. In previous experiences, village consensus would be obtained but this usually followed the senior-male point of view. While external researchers viewed participatory monitoring and evaluation (PM&E) as an essential tool for learning more about women's experiences and views, local staff also recognized that men and women differed in their knowledge and attitudes, particularly about technology development.

However, neither of the collaborating research institutions is accustomed to consulting women farmers. Staff tend to assume that women and men share the same knowledge about farming, that men eventually discuss farming issues with their wives, or that women are not considered relevant actors in technology development. Aquaculture in Laos and crop production in Bolivia are generally regarded as 'men's work'. However, in each case, technology development is likely to affect women and men differently. Furthermore, in Bolivia and Laos alike, women are found to be much more closely involved in farming systems than technical researchers or extension workers expect.

In this chapter we compare these two experiences and consider the implications for developing a PM&E research process. We focus particularly on indicator development and the merits of using matrix scoring as a method for evaluating the impacts of new farming technology.⁵ By looking at two projects from different cultures and institutional contexts, we can observe how evaluation tools are adjusted to local conditions and what participants have learnt from their use. We also further reflect on the wider applicability of information generated using a PM&E approach.

The methodology in practice

While both collaborating research institutions had little direct experience with PM&E, each adopted a project methodology that emphasized stakeholder participation. In Bolivia, the project involved farmers, scientists, and NGO development workers in identifying and sharing farming know-ledge. CIAT's non-hierarchical structure has allowed its staff considerable flexibility in working with farmers, so field staff implicitly base their research on close knowledge of farmers' priorities. The PM&E process began with a research planning workshop during which farmers planned their own trials based on their own knowledge and experience of soil conservation. Because various project stakeholders are involved, different types of information were needed for the monitoring process. Local staff were quick to recognize the need for a participatory approach to M&E, because their close interaction with local communities made them aware of farmers' needs and experiences of previous farm trials, and conscious that farmers use different criteria to judge the success of a technology.

In Laos, because working closely with farmers is still a novel approach within the collaborating institution, researchers decided to integrate PM&E more formally into work plans to fit local institutional practice. The project focused on increasing fish production in rice fields, and intended to establish institutional linkages between external researchers (from loA and AERDD), the LFS and the LWU at provincial and district levels. District-level staff were then supposed to work with farmers. From the outset, the knowledge and perspectives of the project stakeholders-including farmers, the various institutions, and researchers-were all considered important components of the research process. The local institutions (the LWU and LFS) had close contacts with the farming communities but little experience in sharing and recording information systematically together with farmers. LFS researchers had experience with data gathering but limited knowledge about local conditions, and they faced cultural and language barriers. After working closely with farmers in workshops, a participatory system was established for recording farm trials that would enable farmers to identify options for aquaculture development.

The process of developing indicators differed in each country, but in general the two projects used the following key steps:

1. situation analysis, to improve understanding of the interlinkages in farming systems and to identify farmer perceptions
   
   
2. discussions with farmers regarding individual experiences with systems changes as a result of incorporating new farming practices (i.e. soil conservation or fish culture)
   
   
3. discussions with other farmers (individually or collectively) regarding their expectations of incorporating new farming practices
   
   
4. indicator development together with farmers and researchers to show farming systems changes and impacts of new technologies
   
   
5. refining of systems indicators in the field by involving more farmers
   
   
6. use of matrix diagrams based on the identified indicators to rank and score changes.

In Bolivia work began with two to three farmers in each of the three communities; now work is continuing with ten farmers in each community and farm trials are planned for future seasons. In Laos the team is presently working with five to ten families in each of the six communities.

As researchers were interested in exploring the effects of new technologies on farmers' livelihood systems, they decided to adopt matrix scoring for monitoring and recording information. Matrix scoring has been used in participatory evaluation of simple technologies, such as testing crop varieties (Ashby, 1990). As mentioned earlier, it is generally more difficult to make straightforward comparisons when whole farming systems interventions are involved. Therefore, instead of comparing a range of technologies with each other, these projects use matrix scoring tools to compare how farm household members perceive their farming system as a whole both before and after the trial. The intention was that such matrices would use the indicators defined by farmers to explore the changes throughout the farming system.

Changing indicators in Bolivia

In Bolivia the process of developing indicators was less formal and less systematically documented than in Laos. This was because of CIAT's more flexible institutional structure and limited staff experience with documentation. Researchers began by asking farmers to assess their previous experiences of conducting contour hedgerow trials. The process allowed researchers to determine indicators through semi-structured interviews with farmers. Farmers' assessments focused on their perceptions of change as a result of the trials. Outside researchers inferred indicators from these, which were then used to find out how farmers' perceptions changed during the trial. However, as will be discussed further below, the implicit indicators used by farmers changed in the process of applying the new technology.

At an early stage, when the contour hedgerows were still small, farmers identified criteria that were important to them:

• time (labour) spent preparing the land for sowing (e.g. when the fallow or stubble is not burnt)
  
  
• palatability to livestock or risk of losing hedgerows through browsing
  
  
• compatibility with farming practices such as ploughing by oxen.

After eight months of growth, participating farmers and their neighbours began to notice how the soil was building up on the uphill side of the hedgerows, and how recent rain had left the soil around the trees damper than on the rest of the slope. Two new criteria were then added:

• moisture retention
  
  
• soil retention.

At this stage, more farmers began to express interest in establishing further trials. Initial caution regarding the potential negative impact on the farming system was being replaced by observations of potential positive impact. By documenting farmers' criteria for evaluating technology, researchers and farmers have improved their understanding of the role of soil conservation in the whole system. Identifying farmer's evaluation criteria also proved useful to researchers, who were then able to infer indicators based on these criteria and document how farmers' attitudes towards new technologies can change: initial doubts may give way to-wards greater enthusiasm for experimentation and identifying new production alternatives.

Later in the process, a research planning workshop was conducted during which farmers explicitly identified evaluation indicators. These expanded on the implicit indicators which researchers had inferred up to that point. The question posed to farmers was: 'How will I know if my experiment is working out well?' Farmers' evaluation indicators included:

• dark soil colour
  
  
• abundant vegetation
  
  
• no bare soil
  
  
• humidity in the soil
  
  
• increased production
  
  
• our neighbours copy what we are doing
  
  
• water flow increases
  
  
• increased soil organic matter
  
  
• the soil will stay on the hillside and not wash away
  
  
• animals with increased weight.

Farmers' criteria indicated both causes and effects of improved soil fertility and conservation. The indicators themselves informed researchers that farmers understand linkages between soil cover, soil colour, soil organic matter, humidity, fertility and productivity. They also highlighted the fact that farmers prioritized the improving of fertility and crop and animal production over preventing soil erosion.

The process of developing indicators highlighted important differences in evaluation perspectives between men and women. Women were more concerned about the suitability of contour species for fuel and their palatability to sheep, while men emphasized species palatability for cattle over meeting fuel needs. Previous trials experimented with grasses and leguminous trees for creating contour barriers, which satisfied most male farmers. However, women's evaluation criteria revealed that women farmers were more interested in planting grasses and trees in pure pastures or plantations, highlighting their interest in livestock nutrition and fuel production.

More formal methods in Laos

In contrast with Bolivia, partner institutions in Laos had much less experience with participatory research or evaluation, and language barriers made communication amongst stakeholders more difficult. Consequently, the outside researchers initially played a greater role in establishing the PM&E process and supported project staff in developing tools. A more structured and formalized approach was adopted, partly because institutional staff were more accustomed to following fixed guidelines in conducting most of their activities. Project staff decided to use matrix scoring tools and work only with participating farmers with experience in fish production - to avoid the complications of recording information in larger group discussions. This more structured process helped ease language difficulties and staff's limited confidence in using flexible methods.

Since staff did not have much experience with semi-structured interviewing, the external researchers suggested adopting bio-resource flow diagrams as a tool for identifying local indicators of farming systems change.⁶ Farmers involved in fish production drew diagrams to illustrate their farming systems before and after introducing aquaculture activities. Discussions with farmers revealed that biological and physical inputs and outputs changed, as did cash flow, labour and family nutrition. Figures 4.1 and 4.2 show two diagrams drawn by Mrs Nouna, from Nyangsoung village, who illustrated the complexity of her current rice-fish system (Figure 4.2) in contrast with her former system. This led to a realization that wild fish populations increased as a consequence of digging ditches in rice fields to help fish cultivation.⁷

Farmer observations of changes based on the systems diagrams were then used by farmers and researchers to identify criteria for evaluating rice–fish trials. Discussions in selecting indicators focused on weighing the benefits and risks of fish-in-rice – for instance, gaining or losing food, money, work, and land. Indirect effects, such as time saved looking for food, money saved from buying food, nutrition, rice production, and wild fish numbers, were also considered. Nine indicators were finally identified: time, investment, labour, land, rice production, wild fish yield, cultured fish yield, technical knowledge, living expenses. After interviewing other farmers, two more indicators were added: improved family diet and income (see Figure 4.3).

Figure 4.1: Resource flows on Mrs Nouna's farm before introducing fish to her rice field

Selected indicators were incorporated into matrices for monitoring and evaluating results. The first attempt to use a matrix involved ranking the indicators, but was not very successful due to some confusion over its use. The second attempt used the conventional matrix scoring method, which local staff applied more successfully (Figure 4.3). Staff used matrix scoring with great enthusiasm during the situation analysis, and therefore had gained familiarity with applying the tool. They used stones to indicate their perception of the quantitative value of each factor (many, middle or few), both before and after the trial. The evaluation was only conducted after farm trials in order to look at changes of farmers' perceptions.⁸

Matrix scores indicate perceptions of large increases in both fish and rice production. The matrix was used during interviews with both men and women in each participating household. Matrix results suggest that men and women differed in terms of their assessment of the amount of labour required by the new technology (women usually indicated more) and perceptions of resulting fish yields (there were variable but no consistent differences between men's and women's views). Discussions provided further insights similar to those found in Bolivia: farming households (including men and women alike) prioritize indicators that represent risk or costs and, hence, are regarded as potentially negative, over indicators that represent potential benefit or positive impacts. As was the case in Bolivia, farmers' priorities may very well change if they perceive fish production to be successful.

Figure 4.2: Resource flows on Mrs Nouna's farm after introducing fish to her rice field, showing 'multiple simultaneous innovation' and a variety of factors changing as a result of the innovation

Despite the more formal use of tools in Laos, the process helped to involve farmers actively in analysing changes on their farms. Local staff were particularly impressed by the effectiveness of using systems diagrams for technology evaluation, and have adopted the method in other research. Diagrams facilitated communication between researchers and farmers, who found it much easier to discuss experiences by using visual illustrations.

Scope for increasing stakeholder participation

In both projects, outside researchers (loA and AERDD) and donors played a key role in promoting a participatory approach to evaluating technology. While partner institutions in Bolivia and Laos were both interested in using PRA methods, they initially did not regard 'participatory research' as going beyond the situation analysis or diagnostic phase. The process has encouraged a wider range of stakeholders to become involved and learn from the research process, including government and non-governmental institutions. This, in turn, has helped to widen project-reach and involve a greater number of participating farmers.

Figure 4.3: Form for monitoring of fish-in-rice (Savannakhet, Laos)

However, the team's experience suggests that it is mainly the research staff (from external and local research institutions) who may derive more benefits from the process than farmers. For instance, in Bolivia, CIAT researchers emphasized the need for evaluating trials with farmers using farmers' criteria. They felt that this process would help them carry out future research and extension work, and refine technology further. In Laos, there was incentive to build local institutional capacities, as local government research staff have limited experience in working closely with farmers, and wanted to learn more about using participatory research tools. As a result, researchers in Bolivia and Laos have learned a great deal about farmers' perspectives and priorities. They better understand local evaluation criteria and how these influence farmers' adoption or development of new technologies. However, in terms of providing direct benefits to farmers, it is not yet so evidently demonstrated that such a participatory process is useful.

During the self-evaluation of the entire research process (including PM&E), the research teams felt that greater farmer participation could still be achieved. For example, in Bolivia, involving women in the research process remains a challenge; both institutional and community perceptions hinder their more active participation and the fuller appreciation of their knowledge and recognition of their priorities. In Laos, focusing PM&E on participating farming households has meant that the technology impact on non-participating farmers has been ignored. Including non-participating farmers in the evaluation of farm trials would improve equity and provide information useful to international researchers and national policy makers concerned with agricultural development. In addition, local staff in Laos generally treated matrices merely as forms for recording information, rather than as tools for actually stimulating farmer analysis, recognizing different stakeholder perspectives, and assessing impacts on men and women.

Although there remain limitations to farmer participation, there is some indication that farmers can and do benefit from the PM&E process. For instance, in Bolivia, several farmers now use some of the indicators developed in documenting their own trials. Because of the more informal, personal mode of interaction between CIAT researchers and farmers, it has become rather artificial to document these indicators more explicitly into matrices. Instead, farmers participating in trials are designing their own evaluation forms to monitor indicators (such as soil loss, quantity of fodder produced, and crop yield) and to compare results before and after the trials.⁹

Using participatory indicators for wider application of results

The research process in Bolivia and in Laos led to farmer-designed and-managed experiments that involved several stakeholders with common aims but also with stakeholder-specific objectives. For institutional stakeholders (including donors), part of their interest in the research lies in the wider usefulness of findings for informing policy and recommendations in other comparable zones of the world (Lawrence et al., 1997b). These include generalizing about farmer strategies, decision making, local M&E criteria and data gathering methods (i.e. the use of matrices). The next section focuses on how external researchers can use formal research principles and methods to address these stakeholder interests.

Who is carrying out the experiments?

Research objectives of these projects make clear that farmers play a central role as experimenters in designing farm trials, evaluating and comparing results, while researchers serve as facilitators and observers. These different roles and objectives, in turn, affect how research results are analysed and used by the different stakeholders. For instance, each farmer's experiment will yield information specifically relevant to the farmer in terms of the technology's direct benefits to his/her enterprise. Without external intervention, farmers are likely to monitor and evaluate results on their own but through more informal mechanisms (i.e. through daily observations). However, external researchers are interested in recording these M&E processes, and hence developing and using information gathering instruments in partnership with farmers. In Laos in particular, external researchers were eager to develop evaluation matrices which they used for recording relevant M&E criteria for each experiment and for formalizing the M&E methodology across farmers. Using M&E matrices can then later help external researchers to integrate results and make more formal/quantitative comparisons.

Information collected in the matrices will, to some extent, allow external researchers to make comparisons across time and between farming strategies. However, the highly variable and risk-prone nature of the farming systems make direct comparisons between strategies difficult. Comparisons are less problematic when an individual farmer decides to conduct an experiment that tests at least two types of 'treatments'. This was not popular amongst farmers in Laos because fish culture technology is new and the main treatment – incorporating fish production into the farming system – was mostly adopted on a small scale as a single treatment in part of the landholding. In contrast, some Bolivian farmers have experimented with more than one treatment on the same farm, mainly by using different species for hedgerows and cover crops. The problem in comparing single-treatment experiments stems from the difficulty of separating the effects of local variability from the possible impacts of the treatments, as well as expecting that individual farmers will use similar criteria for evaluating treatments.

In other instances, comparisons are possible when a relatively large group of farmers decides to experiment with a relatively small number of treatments. Research results derived in such contexts can provide a general idea of the main effect of a treatment but also indicate how variable this impact might be. Analysing the variability of impact, especially in relation to individual farmer characteristics or recommendation options, might provide a sound basis for generalizations. In Bolivia we expect that by the end of the third year of the project, there will be around 50 farmers conducting experiments.¹⁰ Depending on the results of the different treatments, CIAT could undertake an analysis of this sort in the mid-term. Replicating experiments would make findings amenable to using statistical techniques for establishing generalizations. However, our ability to measure variability will depend on the number of technologies adopted by farmers, the number of farmers conducting experiments, and the assumptions necessary in order to decide whether comparisons can be made (i.e. whether or not a farmer's strategy is similar enough to another farmer's strategy).

Representativeness

General statements can be made only if farmer participants adequately represent the farmer population and farming systems about which we attempt to generalize. But how can we ensure representativeness of farmers within a fluid, participatory research process? One way is to select farmer participants randomly at appropriate stages, but this may prove difficult – especially when aiming to carry out participatory research. Another alternative is to check after farmer selection whether farmer participants are representative of the wider population, using generally accepted characteristics (i.e. levels of wealth, ethnic composition, characteristics of the unmodified farming system, etc.).

Using ranks or scores?

In developing M&E matrices, the use of ranking or scoring will allow different types of analysis. Ranking is useful for identifying farmers' prioritization of criteria, but is less useful for making generalizations. Ranks contain less information than scores since they depend on the number and type of criteria determined by the participants in the ranking exercise. Consequently, it is difficult to use rankings of one group in combination with rankings produced by other groups, unless exactly the same criteria are imposed on the groups. This is a less desirable alternative, especially in the context of participatory research. On the other hand, scores have an advantage over ranks because scoring contains more information.¹¹ While ranks can be constructed from scores it is impossible to construct scores from ranks. Scoring allows the scores of different farmer groups to be combined and compared, with fewer conditions imposed on the participatory process.

In order to make generalizations or quantitative comparisons, measuring variability is important. However, group discussions that take place during participatory matrix scoring often tend to smooth out the variability of individual perspectives and experiences, resulting in group consensus or compromises. In Laos one way diversity and variability was ensured was to develop indicators based on group consensus but to ask each individual to evaluate their own experiments independently. The resulting sets of scores provide external researchers with a basis for further quantitative analysis, which will be useful in reaching wider conclusions about their own research questions.

In searching for ways to generalize about results obtained from participatory processes, research teams felt the need to use more quantitative measures and to take advantage of the pool of statistical ideas often used in other areas of research. The challenge lies in maintaining the real advantages of participatory methods, while at the same time incorporating principles such as replication, independent observations and representativeness that allow the use of statistical methods of generalization. Using matrices opens possibilities for arriving at a balance between these two research objectives.

Lessons learned

This chapter has identified some of the challenges presented by donor-funded participatory research projects:

• taking into account different stakeholder needs
  
  
• generalizing about project results and findings
  
  
• developing appropriate participatory methods and institutionalizing participatory approaches within government research institutions
  
  
• identifying indicators for measuring systems level changes affected by participatory technology development.

The next section elaborates on these challenges to PM&E research.

Usefulness of the PM&E process to different stakeholders

Researchers sometimes assume that asking farmers to evaluate new technologies is a process that is intrinsically useful to farmers. While our experience does not negate that assumption, it does indicate that participatory methods involving farmers in documenting change (even in using a shared, visual method such as matrices) may be of more value in facilitating communication between farmers and researchers, than in enabling farmers themselves to arrive at dramatic new insights. The more formal approach and forms used in Laos, in particular, limit the method in terms of providing in-depth, meaningful data. It becomes all too easy for government officials who are accustomed to collecting census data to fall into the mode of merely recording views without generating local analysis and reflection.

The method has also been invaluable to researchers in terms of drawing out the different perceptions between women and men. Through external facilitation, local staff were encouraged to compare the evaluation matrices of men and women. In both projects in Laos and Bolivia, government staff are now much more aware of the value of women's perspectives on the impact of new technology. Particularly in Bolivia, staff, despite initial reluctance, eventually appreciated the different views of men and women farmers and that each were equally valid. As a result, the value of women's knowledge of livestock forage preferences is now much more acknowledged by CIAT staff.

This raises the question of who is benefiting from the PM&E process. Our experience suggests that farmers may not immediately value nor derive direct benefits from indicators, forms and matrices used as evaluation tools, because many already have informal ways of assessing their own experiments. The tools are more useful in that they help extension agents and researchers better understand farmers' needs and perceptions, and the costs and benefits of farming experiments. Nevertheless, using participatory evaluation tools can place local staff and farmers in a better position to make decisions about new technologies on local farms. Overall, however, our experience shows that the process of learning from farmers' indicators and their evaluation of those indicators has been most valuable in helping outside researchers, e.g. in thinking about replicability, institutional appropriateness, and institutionalization.

Adapting methods for different institutions or cultures

CIAT and LFS have quite different institutional cultures, which in turn have implications for the way PM&E methods are used and adapted. CIAT staff in Bolivia tend to adopt a more informal, flexible approach to decision making and working. Because most of their time is spent in the field, staff have a very good understanding of farmers' perspectives and ideas about technology development and are quick to support them. However, CIAT staff are less interested in formal documentation and reporting. Matrices and forms have been introduced into workshops but are not widely used. On the other hand, LFS staff in Laos respond to a more centralized model of decision making and accountability and have adopted a more structured approach to documenting results. District staff wanted to record quantitative data and use forms for recording information, pointing out that farmers were able to quantify changes and values more often than PRA methods allowed them to. These observations led them to develop more structured methods, such as matrices, for monitoring and registering feedback.

While matrix scoring was promoted in both institutions, differences in institutional working styles necessitated that the tool be adapted and supplemented. Staff in Laos used matrices for recording information but found resource flow diagrams helpful in facilitating communication between farmers and researchers. Because of language barriers and staff's limited experience with open-ended group discussions, resource flow diagrams made it easier to identify farmers' evaluation criteria – which were eventually converted into indicators on the matrices. On the other hand, in Bolivia semi-structured interviews between farmers and researchers sufficed.

Potential for comparisons and applying the results elsewhere

The way research was conducted in Laos and in Bolivia, in turn, affected the potential for comparing and generalizing results. As mentioned previously, in Laos, local staff paid more attention to detail and documentation. In Bolivia, by contrast, while staff were enthusiastically committed to helping farmers, they did not see much value in filling in evaluation forms but invested in developing more personal interactions and informal discussions with farmers.

The Laos approach led to a data gathering method that was more amenable to statistical analysis and generalization than in Bolivia. Once sufficient data has been collected in Laos, it will be possible to link the results to factors such as gender, the agroecological system and individual wealth, and to draw conclusions on how these factors affect farming strategies. However, the validity of the data collected through the Laos form-filling approach has yet to be verified. Furthermore, while the more formal Laos approach led to meticulous quantitative documentation of farmers' evaluations, there was limited explanation of why different farmers rated change in different ways. By contrast, the more haphazard, informal approach in Bolivia – while perhaps more frustrating to donors and others seeking more systematic procedures – provided a better understanding of why farmers were developing technologies in a particular direction. Limited documentation in Bolivia, nevertheless, prevented the further sharing of experiences amongst other staff and farmers. These institutional differences are cultural – an aspect of PM&E which has been little explored but has significant implications for the way information is obtained and used.

Evaluating farming systems change

In both countries the research process made new attempts to explore the range of factors affected by farming systems development. Through resource flow diagrams and semi-structured interviews, farmers were able to identify indicators which pointed out systems impacts that researchers had been unaware of. For example, in Laos management of cultured fish can affect wild fish populations. In Bolivia, growing contour hedgerows for soil conservation could affect cattle nutrition, or be affected by browsing cattle.

Farmers' indicators were themselves a valuable product of the research. In both projects indicators revealed farmers' understanding of ecological and economic processes and interactions. In particular, indicators of success identified by Bolivian farmers (described above) show that they understand the role of organic matter in conserving nutrients, humidity and soil.

The use of indicators in a matrix improved comparability before and after trials and across farming households. However, the research team found it more useful to complement the more rigid matrix method with more open methods that helped reveal unexpected outcomes or benefits, even though results may be less comparable and generalizable. For instance, the more open-ended use of methods in Bolivia showed that indicators can change over time, as farmers' experiments produced results which farmers and researchers did not expect.

Towards institutionalization: building on participatory evaluation of technologies

In both projects, an iterative approach to the research process incorporated stages of self-evaluation and learning, which led to local staff defining their own needs for PM&E. In Bolivia, workshops to share the experience with other CIAT staff and a range of NGOs have helped to draw out stronger conclusions about the usefulness of the research, including those reported in this chapter. In Laos, a key feature of PM&E was that it incorporated methods that staff had learnt and used in conducting other PRA work, thus building their confidence and understanding in applying the tools more flexibly. In both countries, staff have strengthened their understanding and capacities to plan, monitor and evaluate new technologies together with farmers, and to apply what they learn in other aspects of their work.

Document(s) 14 of 24