Document(s) 17 of 24

Introduction

'While institutional capacity development is strongly assumed to be beneficial, there has been relatively little systematic analysis of institutional capacity and its growth subsequent to intervention.'

(Lusthaus et al., 1995: 2)

THE ADOPTION OF a learning-based approach to evaluation presents special challenges to a research funding agency with a mandate to strengthen research and research capacity with partners in less industrial countries, or in the South. While the general practice has been to evaluate funded projects, there is increased recognition that the project may be the wrong unit of analysis. Projects are a way of organizing work, but they are not the end in development. They do not, in themselves, serve the purpose of building institutional capacity, and their implementation and evaluation may in some cases be detrimental to the strengthening of an institution. The adoption of a learning-based approach to evaluation within a funding agency leads to the realization that there is also a need to apply this evaluation approach within recipient organizations and potential benefit from so doing. This highlights a significant change in perspective on the use of evaluation for both the donor and the recipient. Such an approach presents significant challenges and opportunities to increase participation in the evaluation process. Giving evaluation away to those most directly affected calls for new approaches to evaluation, which both recognize the need for accountability and quality control and build the internal capacity of organizations for using evaluation for their own organizational planning and management purposes.

In 1995, the International Development Research Centre (IDRC) published a framework for institutional assessment for research organizations (Lusthaus et. al., 1995) which was originally commissioned to meet the needs of the centre in assessing the organizations it funds. It was quickly recognized that this framework had considerable potential as a participatory self-assessment tool and as a mechanism to assist organizations in building evaluation into their planning and management systems. Trials were carried out in several organizations in West Africa and South Asia.

In this chapter we explore the background to the development of a model for institutional assessment at IDRC, to support our interest in strengthening capacity with our partner institutions. We focus on perspectives from a funding agency because that is where our experience lies, but also because funding agencies have driven a significant part of the evaluation agenda in development work for the past 20 years.

IDRC is a public corporation funded mainly by the Government of Canada. IDRC was established in 1970 and funds research and research capacity building in developing countries, with a view to supporting local capacity building for scientific research in support of development. While in the early years of IDRC the primary focus was on building individual research capacity, there has been increasing emphasis on building strong research systems, organizations and institutions.

Background

The field of international development has a particular relationship with evaluation. Evaluation has been used primarily by donors to assess the utility of their projects in countries they are assisting. In this context, donor agencies generally set the evaluation measures and establish criteria based on donor agency programmes. This approach to evaluation remains an important dimension of accountability for any donor agency, whether in the public sector or a non-governmental organization (NGO). From the point of view of recipient organizations, evaluation has thus been viewed largely as a policing mechanism, and in donor agencies its implementation has largely been on a compliance basis. What is assumed in this approach, is that good projects were selected to begin with and that these projects will lead to an overall beneficial effect. Evaluation of projects often serves as a proxy to assess executing agencies: if 'good' projects are happening, then the executing agency is considered good (and vice versa).

Frustration with this donor control of the evaluation agenda, together with an early recognition by community groups and community voices that there was an essential role for the community in evaluation, has led to the development of a number of approaches to evaluation based in the community, such as participatory rural appraisal, among others. While the donor community has been slow to deal with this issue, it is increasingly recognized that the current approach to project evaluation has not yielded the most beneficial results, either for the donors themselves or for their recipients. It has not been particularly helpful to donors because the focus has been primarily on individual projects, without recognizing overall contributions to development. As we are pushed increasingly to demonstrate results, there is an emerging realization that the results are not evident solely in the projects, but also in the environments where the projects are implemented. Because results are generally translated into short-term measurable impacts of projects, the very nature of research for development to build capacity for the future is at risk.

Project evaluation is also less useful to recipients because this approach remains focused on donor funding agendas, without taking into account the local context in which projects are implemented. As Bajaj (1997) noted, donors and recipients want very different things out of an evaluation. Recipients want to learn about how their objectives are being supported by this work, and what they can learn about their progress in evaluating a given project. Donors want to learn about the project itself, and then relate it back to their programming objectives. As the same study noted, the lack of involvement of recipients in the design stage of evaluation studies, or even the data gathering stage, means that the needs and interests of the donor dominate the evaluation agenda. Recipients only tend to be brought into the evaluation to help with the logistics and to hear the results. If they have not been actively involved in the design of the evaluation process itself, it is hardly surprising that most evaluation results are irrelevant to the recipient organizations.

In summary, the project may be the wrong unit of analysis. Rather, the analysis should be more specifically focused on the results we are trying to achieve, whether to strengthen a field of research or to contribute to a domain (such as health, employment, food security) in national development. In other words, instead of regarding projects as the end, they should be viewed more as the vehicles to achieving larger development objectives. That is certainly the intention in funding the work in the first place; however, the evaluation process does not reflect that reality. If we move in this direction, results are then measured in terms of progress towards the objective, not only in terms of the (project) vehicle's successful performance.

That projects should be regarded as a means rather than the end is not a novel concept. It is in the implementation that projects have become the focus and for many purposes, the end point. As Najam (1995) notes in a review of the literature on project and policy implementation, only when the actors are viewed as the unit of analysis and implementation is seen as a political process do we begin to build an understanding of the enabling and constraining factors in any initiative. In contexts where there are many actors, both individual and institutional, the process is even more complex; hence, a project-focused evaluation approach will take one further away from a clearer understanding of the interactions and interests driving the success or failure of an initiative. Both the problem area and the project context are critical in the evaluation process, as are the roles and functions of implementing agents and those affected by the activity or project.

Viewing evaluation from this perspective has major implications for the evaluation programmes of donor agencies and granting councils, where learning has been largely based within the funding agency and where the project has been the basic unit of analysis. With the focus on performance measurement and results-based management, a project should be assessed in the context of how it is contributing to the larger goal of development. This means that there has to be learning both for the funding agency and the recipient organization. The unit of analysis changes and-perhaps more importantly-it means that performance is measured against progress in a development context, not solely against achievement of the project.

From project evaluation to institutional assessment

The growing awareness within the donor community of the importance of institutional³ capacity building as a critical part of development work in the South is part of moving away from a project model of development to a more systemic model. It is recognized that institutions and organizations play vital roles in how a community evolves and what opportunities it acquires. Institutional capacity building takes a variety of forms: some argue that organizational structures need to be created and reinforced; others argue that alternate forms of support such as networks of support among researchers in different countries are a more effective mode than building organizations. But in all cases, there is recognized need for a support structure so that strong and capable individuals do not operate in isolation (Bernard, 1996; Lusthaus et al., 1995). There is a need to create a space for consultation, a space for bringing along junior researchers and a space for action and influence on the policy-making process that extends beyond the individual reach of any one person.

The establishment of strong and capable local institutions-and not only strong projects-is necessary to make decisions effectively and to implement programmes. This need is part of the recognition that development agencies don't deliver 'development' but rather deliver pieces of the development puzzle which countries, organizations, networks or individuals can choose to use or not. Many different types of programmes have been designed around this issue, both on the research side and on the development side. They include organizational support grants for research centres, the creation and strengthening of research networks, support to government agency capacity building, support to NGOs, and so on; they include specialized research area grants, core grants, and training programmes. An issue that emerges is how to evaluate progress in this area. What constitutes institutional capacity strengthening? How does it differ from individual capacity building? And what criteria should be used and who should be involved in the assessment process?

In many countries where IDRC is working, individual research capacity has grown significantly over the 25 years that the centre has been operating. We find that we are working with an increasingly sophisticated research community (Salewicz and Dwivedi, 1996). While many efforts are under way to expand research capacity both within the traditional university-related research community and outside, an increasing emphasis is on the institutional structures within which individual researchers operate. Strong researchers need institutional support structures to conduct their work and mechanisms through which to influence the policy process. This may mean the building of traditional research structures-university departments, research institutes-but it may also mean building other forms of institutional support, such as research networks. Whatever the strategy, there is a need to explore the most effective patterns for institutional support and to build a capacity to assess the organizations and institutions that are created or strengthened. As the centre moved towards this direction, several requests were directed to the Evaluation Unit at IDRC to identify some appropriate tools for assessing institutional development, to complement the existing abilities in assessing individual research capacity.

The Evaluation Unit of IDRC undertook to develop a framework for the assessment of institutional capacity with a particular focus on research institutions. This framework was developed with the Universalia Management Group (Lusthaus et al., 1995) and was the basis for development of an approach to diagnose organizational strengths and weaknesses and provide a basis on which to identify and determine potential areas for support. What is unique about this framework is that it explicitly addresses several dimensions of institutional strengthening. While most institutional assessment work focuses primarily on capacity within the organization as the critical dimension, this framework looks equally at four dimensions of an organization:

• capacity (leadership, management, human resources) remains important, but balanced with
  
  
• motivation (history, mission, culture, incentives) and
  
  
• environment (legal, social, technical, etc.). These three key elements are situated in a
  
  
• performance framework, based on effectiveness, efficiency, relevance and financial viability.

The approach is based on the premise that performance demonstrates the results of the organization's work-in efficiency, effectiveness, relevance and financial viability. Performance is then the synthesis and result of the way in which the organization uses its capacities, builds motivation, and deals with its environment. In order to assess these areas of performance, the three areas of capacity, motivation and environment are assessed.

Since each institution or organization is unique-with different capacities, environments and mission-this framework for institutional assessment is not prescriptive. Rather, this framework provides a set of guidelines around the key areas that need to be addressed. These factors are interrelated, as illustrated in Figure 13.1.

Figure 13.1: Institutional assessment framework

The framework can be used for external or internal review. It can be used for a comprehensive review of an organization, or to address a specific issue or problem. It was developed in the first instance as a tool for a funding agency to assess its partnerships. However, because of the factors noted above (i.e. the importance of ownership in the use of results, and the relevance of assessment as part of the capacity of an organization), we tested the framework as a self-assessment approach. Several case studies based on use of this model were presented at the Canadian Evaluation Society meetings in Ottawa in May 1997. At that point, the work was just coming to a close in most of the organizations that adopted the self-assessment framework. Since then, we have had the final reports which give us further insights into the areas covered, the problems encountered and the potential for this work.

What we will do here is to elaborate on the findings of testing this framework and explore their implications for applying the model and for strengthening future work in the area of participation in institutional assessment. Based on our experience, this model is not restricted to research organizations but is also useful for other types of organizations. While the cases presented here are all research oriented, they nonetheless provide useful insights more generally in the area of participatory institutional assessment. What emerges from these experiences is that a participatory monitoring and evaluation approach should form a key part of any organizational assessment, as organizations are the platform from which actions and initiatives spring.

Experiences in institutional self-assessment

We present experiences here as a synthesis of the self-assessment work undertaken in several research institutions in West Africa and South Asia, rather than highlighting any one case (see Box 13.1).⁴ First, we outline what we thought would happen; then we summarise what actually happened. We then explore some of the lessons that emerge and the potential we see for ourselves, other funding agencies, and the recipients gaining more control of the evaluation process. Finally, we will raise some issues for future research.

The plan

The institutions involved in self-assessment were approached on the basis of recommendations and suggestions from IDRC programme officers. The concept was that this would be a joint assessment, involving both IDRC and the recipient, as both had learning needs about capacity of the organizations. The process was to be facilitated by Universalia Management Group, who would assist in the identification of terms of reference with each institution, identification of tools, support for methodology for data collection and analysis, and commentary on the final report. IDRC would remain involved to some degree with the participating institutions in the expectation that the reviews could be of value to IDRC and could obviate the necessity for external review in some cases. It was also expected that IDRC would learn more about the potential of the assessment as a tool in building organizational capacity. Time frames were individually established; however, it was intended that there be considerable overlap in timing amongst the three institutions in West Africa-in part to save on travel costs for the facilitators, and in part so that there would be some opportunity for comparisons and joint work by the organizations.

In South Asia the process was slightly different, integrating a strategic planning process into the self-assessment. This entailed a workshop following the assessment in which the members of the organization met for a week to discuss how the diagnosis influenced their strategic plan.

In both settings, an initial visit by IDRC to propose the institutional self-assessment was followed by a consultation with the Universalia team to discuss 'readiness'⁵ and to begin the definition of terms of reference and a work plan; to establish a process in each organization; and to consider the resources (internal and external) that would be needed to conduct the assessment. Finally, the consultants were asked to provide a comment to IDRC on the external review, not so much in terms of the conclusions of the team, but rather in terms of the quality and reliability of the data on which the conclusions were based: did they ensure full data collection? did they ensure access to reliable data? did they identify all relevant sources? and so on. The purpose of this comment was to provide back-up to IDRC on the legitimacy and quality of the assessment so that it had the potential to be used for IDRC purposes as well.

In West Africa, IDRC has a regional evaluation officer based in Dakar. She worked closely with Universalia to provide back-up for the institutions participating in the process. Her role was to keep the process moving, either by providing assistance herself, or involving a programme officer, the consultants or the evaluation unit as needed. She was involved from the beginning of the assessments and maintained a watching brief, assisting where appropriate.

It was anticipated that the assessment would result in a report that could be used not only by the organization in its own planning but also by IDRC as part of its accountability requirements.

What happened

These case studies were all within organizations that have received funding from IDRC, in West Africa and South Asia. They are all research/development organizations, but of somewhat different types-from regional institutions, to research institutes within a university. All engage in development research and all seek to influence development policy at the national and regional levels. All are engaged in work that is intended to create an 'indigenous body of knowledge' in their respective fields of endeavour (economics, social sciences, rural development)-that is, all are seeking to create or adapt models of research for local conditions.

There was initial scepticism in most of the organizations. This was based on previous experiences with evaluation and organizational assessment (where it had been used in other contexts to down-size, reduce funding, etc.), on concerns about the links between the assessment and ongoing IDRC support, and on the perceived commitment of resources to a process advocated from outside. Not surprisingly, scepticism was least pronounced where there was no direct link between the assessment and any projects, both in terms of timing and programme officer involvement. In the process of implementation, scepticism was slowly overcome in all but one case, and the assessments proceeded effectively. Overcoming the scepticism was an incremental process; it happened as the participants perceived the relevance of the process to their own needs. In one case, scepticism persisted and is, in our view, the primary reason that the assessment has not been completed to this date. Start-up was slower where scepticism was higher.

The work was carried out by providing facilitation support to design an institutional self-assessment process around the framework. The actual development of terms of reference, data collection and analysis were carried out by the organizations themselves, with some involvement of the facilitators and some external expertise commissioned in some cases. In West Africa, the self-assessment process emerged as a result of a joint design workshop involving all three research institutes with IDRC and the facilitators. The workshop was called to outline the nature of the self-assessment, develop terms of reference for each study and begin to design data collection instruments. It was both helpful and a distraction to have the three organizations working together. To some extent they were able to learn from each other and to strengthen the development of terms of reference and data collection. At the same time they each needed a very different process and needed to address different issues. On reflection, perhaps a one-day workshop together, followed by individual organizational workshops would have been more productive. Data gathering by each organization was structured differently and teams to manage the self-assessments were set up according to the prevailing norms in the organizations. In one case, the executive director created a self-assessment team composed of several young professionals led by the head of training. The team was responsible for all aspects of the process, and their work was reviewed by the executive director. This case illustrates how both senior-level support and staff commitment were critical in successfully conducting the entire process (see Box 13.2).

In another organization, the process was led by a team of two very senior managers who subcontracted external consultants to carry out specific aspects of the process such as data collection and analysis of some issues. The team then integrated these external reports into their own synthesis outputs. In a third organization, the senior management operated as a steering committee responsible for the strategic aspects of the self-assessment and mandated various individuals inside the organization to conduct parts of the process.⁶

Different mechanisms were employed in the organizations, from placing the bulk of the work in the hands of relatively junior professionals, to actively involving senior managers throughout the process. The organizations themselves determined which mechanisms to apply. For instance, in one organization the executive director's role was intentionally minimal during the process of the self-assessment; however, his role was crucial in ensuring that important stakeholders would provide needed data. He is influential and respected in his region and he personally called stakeholders both within and outside the organization and encouraged them to respond to the questionnaire that the operational team was sending. The response rate increased significantly with his intervention. In another case, the organization involved a former executive director (the founder of the centre) as part of the evaluation team, and he was able to provide the historical perspective on many of the issues discussed. The individual became the 'wise' adviser and his role was invaluable.

In all cases where the assessment has been completed, there has been strong support from management for the initiation of this process, and there have been human and financial resources dedicated to completion of the work. In the one case where the process is not yet complete, there has not been strong support from the management of the centre: in the midst of discussions it became clear that the director would be leaving his post and from that point on he had no incentive to engage in the process. A new director may or may not make a difference to the process. Discussions have to be undertaken with the new director to determine whether or not the process could usefully proceed at this stage. What will need further clarification is how much the new director will see this as an opportunity to assess the structure and functions of the research centre, or whether he or she will see it as a compliance mechanism. To some extent, the new director's own views on his or her own mandate will be a determining factor, as will be the role and position of IDRC in the process (as is discussed further below).

A joint workshop involving the leadership of all three centres in West Africa was held to introduce the framework, discuss the nature of the process, and the intent of the assessment. However, given the different starting points of each organization, it was not possible to maintain the same time frame on each process. This meant a slightly more expensive process and a slightly more significant time commitment by all parties concerned. It also complicated the start-up of the exercise: as the parties were at different points and held different views, a collective exercise was difficult to use effectively. The experience of the joint workshop revealed the importance of recognizing the different perspectives and interests of each institution as key to securing their commitment to the process (Box 13.3).

The role of the funding agency (in most cases, only IDRC) in the self-assessment process varied. In some cases, programme officers from the donor agency were actively involved, and in others, assessments were undertaken without the involvement of the programme officers (other than awareness that the process was underway). The case work shows quite clearly that it is possible for the granting agency to be involved in supporting this process, but that there must be some clear boundaries. Where a programme of funding is coming to closure (whether a project or an institutional support grant), there are risks that partial information may be used against the organization. This happened with one of the participating centres. In the course of the self-assessment, a number of discussion documents were prepared and circulated within the research centre. These documents were part of tentative ideas raised by different staff members some of which were generally agreed to, while others were new issues coming up for the first time. Because IDRC was involved in working with the group on its self-assessment, the documents were also given to IDRC. In one instance, an IDRC staff person noted some issues in the report, and used the occasion to challenge what was being done in the research centre. This created concern about the use of information and a fear that openness could be penalized. It can be extremely difficult to draw the line between open engagement in discussions and raising issues from outside before the internal conclusions have been reached.

As has been noted in relation to other points above, the assessments generally took longer than anticipated (one is not yet complete). No one realized in advance the implications of a self-assessment process in terms of involvement of staff, members and other constituents. Overcoming some of the barriers outlined above had to be achieved with all the different constituencies. For example, in one case, a member of a self-assessment team had had a difficult experience in the past with an external consultant who was involved in conducting a self-assessment exercise; as a result, this team member raised a lot of initial resistance to the process. The consultant facilitating the self-assessment had to acknowledge and deal with the resistance before the process could actually move forward. This was achieved primarily through dialogue, negotiation and persistence. In another example, a self-assessment was undertaken officially and everyone in the organization was informed. In practice, however, the staff members responsible for the self-assessment did not have enough time to simultaneously conduct the assessment and continue their normal professional activities. Ultimately, the team brought the issue to management to resolve, and the staff member was allocated more time for this task.

In all cases, the self-assessment resulted in focusing on issues pertaining to the mission and direction of the organization-as Bajaj noted in her study (1997) it is the organization itself, not the project, that is of most interest to those being evaluated. What emerged in all cases, was that there were fundamental changes that should be considered in the mission or structure of the organization. For instance, one centre realized that in its efforts to be wellfunded and become a strong organization, it had started to compete with its members for donor-funded projects. The board and management realized that they had to change the nature of the projects supported, in such a way that they would complement and support their members' efforts, rather than take projects away from them. Instead of obtaining funding solely for project implementation, management identified a need to obtain support to provide training for their members, to explore new research areas their members could work in, and in general to find ways to enhance their members' capacities so that they could carry out the work in their own countries.

This outcome of the self-assessment process that leads to a greater organizational focus is not surprising in the sense that as the environment changes, the discordance between any organization's structure and mission with the environment increases. The institutional assessment work creates a timely mechanism for addressing this issue. Since the extent of the potential for change was not appreciated at the beginning of most of the assessments, this meant that not enough time was allocated to consider these issues: it was generally assumed that the assessment would lead to finetuning more than anything else. However, it usually resulted in revealing the potential for much more fundamental change, for which time requirements are more long term.

What we learned

Each organization we dealt with in the process was unique. They were all at different stages in development and all had different issues as a starting point. This highlighted the individual nature of the process and confirmed for us that there is no single approach that can be advocated. Each assessment needs to be defined in the context of the specific setting, and each design has to be sufficiently flexible to adapt as the layers of the organization are peeled back. The experiences to date have suggested several important lessons, both as to the design and to the process of self-assessment. The main insights are highlighted below.

Those inside are not necessarily easier on themselves than an external reviewer would be

In the cases conducted, the leadership has addressed, and in some cases adopted, recommendations that fundamentally challenge the governance structures of the organizations. Because the investigation, analysis and recommendations were drawn from inside the organization, the potential for application is much stronger. The following examples show how results from the self-assessment have been utilized directly by those involved.

• One organization learned that it needed a much stronger capacity to provide training and technical support to its members. As a result of the recommendations of the self-assessment, they have since strengthened the training unit and given it much more prominence in the work they carry out.
  
  
• Another organization continued the self-assessment process with a three-day strategic planning exercise, during which the self-assessment data was used as a basis for the development of strategies.
  
  
• One organization used its self-assessment report to develop a special Board session at their annual meeting.

However, one particular case illustrates that ownership over results may not always be achieved in the process. In this case, the organization never fully completed the exercise due to various changes in leadership. The director left just as the process was to begin. A new director was not in place for some time. There has been no follow-up, and the draft report is likely to be shelved.

There is always the possibility in a self-assessment that the self-interest of those involved will lead them to paint a rosy picture of the situation, either to maintain a view that things are going well or to present a picture to the outside that will lead to further funding. We did not find this to be the case. Difficult issues were raised and addressed in the course of the self-assessment in all the institutions. Challenges to their missions were made and recommendations have included some quite fundamental changes. There are several reasons for this:

• The nature of the self-assessment process involved a range of actors, not only one 'level' of actor in the organization. This means that there are opportunities to raise different perspectives and issues. No organization consists of only one perspective; by involving different actors in the self-assessment, these different perspectives and concerns are brought out.
  
  
• In addition, all of those involved have at one point experienced external reviews in which they had to deal with someone who failed to unravel the layers of complexity in their organization, and who therefore was unable to present relevant recommendations. Those involved in the self-assessment appreciated the opportunity to deal with the issues in depth with a group of participants aware of the complexities within the organization.
  
  
• In the end, it is the staff and membership of the organization who have to live with its successes and failures, not the external reviewers. They, therefore, have a stake in taking the opportunity presented to do everything they can to improve the organization.

The self-assessment process is most effective when it is de-linked from the project cycle

One of the first challenges in the self-assessment process was scepticism about motives: was this simply an alternative way for the funding agency to get inside the organization to decide about future funding? This concern was exacerbated in those organizations closest to the end of their current funding cycle. Since most evaluation is conducted as part of determining whether or not to continue funding of a project or an organization, this remained a problematic factor in the self-assessment cycle. Thus, while the concept of self-assessment should make it part of institutional strengthening, there was a natural tendency to consider how the assessment will affect the project cycle. In instances where project funding was coming to a close, there was a strong tendency to expect the outcome of the assessment to lead into the next (potential) project.

In the one case where the process was de-linked most explicitly from the funding cycle, implementation was much smoother. In this case, the donor agency programme officer was not actively involved with the self-assessment exercise. There was an open discussion of this issue between the donor agency programme officer and the staff of the recipient organization in the beginning of the assessment; it was clearly agreed then that the assessment would not be linked to the project, and that the programme officer from the donor agency would not be directly involved in conducting the assessment. This agreement was fully upheld during the implementation. The programme officer was kept informed of events over the course of the assessment, as well as of the outcomes of the assessment; but he was not necessarily kept informed about the details of the assessment as it took place. While it is possible to develop a collaborative approach to institutional self-assessment, and that assessment can be useful for both the organization and the funding agency, the parameters of that collaboration must be clearly spelled out at the beginning. The principles which would seem to apply are that:

• the terms of reference should be developed collaboratively
  
  
• the process documents should be shared judiciously and their receipt by the recipient organization should be treated as a demonstration of trust and collegiality; the contents should not be used against the organization nor should there be a perception of use in that way
  
  
• the purpose of the self-assessment needs to be kept clearly in focus. For the organization, it contributes specific change recommendations. For the funding agency, it is not so much the specific outputs that are at issue, but rather the identification of capacity building through effective assessment, followed by implementation of the recommendations.

Self-assessment and external review fulfil different purposes

Both external review and self-assessment are legitimate review processes. External review is often needed for accountability of funds received and also for quality control. But, without some parallel review processes internal to the organization, external review does not necessarily contribute to institutional strengthening and capacity building. Self-assessment fulfils that need, by providing the mechanism for an organization to look at its own progress and determine what changes should be made. It strengthens an institution's capacity for reflection, a key component of any learning organization and helps organizations deal on a more equal footing with external stakeholders (i.e. funding agencies). This means more capacity to negotiate with donors on the design of evaluations, resulting in a stronger focus on the progress of the organization as a whole rather than the success of the individual project.

Lessons from the process

Aside from these key areas of learning, there are a number of elements of the self-assessment process that proceeded differently in each organization. The successes and problems encountered suggest some adaptations to the process that should be considered by both implementing organizations and facilitators:

• The self-assessment needs a 'champion', but the champion needs to put a system in place to ensure full participation and continuity if the process is going to proceed clearly and smoothly.
  
  
• The self-assessment needs the support of the relevant interest groups, both within the organization (staff and members) and in the surrounding environment (those affected, government departments, other funding sources, and so on).
  
  
• The organization should be prepared to have discussions on both the mission and structure of the organization. While there was not an intent in most cases to move the assessment to this level, this is what happened in all cases.
  
  
• The process often leads to an ongoing interest in evaluation as a mechanism for learning and organization building. In that context, the establishment of an ongoing monitoring and evaluation process (or a modification of an existing evaluation role) is sometimes an outcome. The concept of a learning approach to evaluation has major implications within the organization in terms of human resources and time investment in evaluation.
  
  
• While the self-assessment process may have been a more time-consuming process than external review, the recommendations are readily understood when they are presented, and do not require the sort of review and internalizing that is required when recommendations come from an external review. Time lag from recommendation to implementation, therefore, is greatly reduced. While we have not tested this idea, it would appear, if we look at time requirements (starting from the beginning of assessment to the implementation of recommendations) that self-assessment is no more time consuming than external review and may actually be less so.
  
  
• There is a need to determine the optimum relationship in a collaborative self-assessment when external actors are involved. While we still don't know what best defines such a relationship, an open exploration of the issues and potential conflicts would certainly be an essential ingredient in the design of a collaborative self-assessment.

The research agenda and next steps

Giving evaluation away to those most affected remains a strong research agenda in building capacity for participatory evaluation within our organization and in work with our partners. The potential for learning from evaluation is much stronger in such a context, and the relevance of evaluation is more clearly demonstrated. The ongoing frustration on the part of evaluators as to whether or not anyone actually uses their results is mitigated when the conclusions are reached by those most affected. As these cases demonstrate, when it is within their power to do something, the members and stakeholders in an organization will conduct an assessment that addresses questions fundamental to the organization and their future work. Several critical questions remain unanswered.

• We don't know how sustainable the interventions for institutional self-assessment will be. Hence, follow-up with the participating organizations over the next several years will be critical.
  
  
• We are only beginning to work with these and other partners on the question of the design of relevant internal monitoring and evaluation systems that will assist them in such processes on an ongoing basis.
  
  
• We don't know if and how the process could be repeated in an organization: would there be reluctance to get so deeply into mission and structure again? Or is there potential for follow-up on a more ad hoc basis, dealing only with a few issues?
  
  
• While we hypothesize that self-assessment will be seen as relevant to the donors, we don't yet know how true that is: will it help the organizations reduce the amount of external review to which they are subjected? Will the donor community begin to see this as a relevant demonstration of built capacity?
  
  
• To date we have not distinguished clearly between institutions and organizations. One distinction may be to describe institutions as policy-making entities and organizations as the structures to implement the rules and policies. Thus, can the same conditions apply in institutions as in organizations? While it is complicated to assess an organization in a participatory manner, moving to the level of an institution (such as the educational system) significantly increases the complexity of applying a participatory-assessment approach: it will have to take into account a larger range of actors, a number of issues, and the different organizations involved.

These are some of the outstanding questions in operationalizing a participatory approach that we will be exploring over the next few years. Other remaining issues deal with methodological considerations, such as issues related to concepts of 'validity', 'rigour', and 'objectivity', which need to be addressed if participatory monitoring and evaluation is ever to be seen as legitimate and relevant, and if its results are going to be applied seriously beyond the boundaries of the community using the approaches.

We became involved in this kind of process because of our own experience in IDRC in terms of its limited use of evaluation and the centre's philosophy of collaborating with Southern partners rather than simply providing expertise that they do not have. For that collaboration to be effective, our partners need to drive their own decision making and development, and our role is to engage with them in that capacity building. In the case of evaluation, it is very much a joint search for new approaches as we are only at the beginning of understanding a more effective role for evaluation in our own setting. Our partners, who have more often than not been the subject of evaluation, bring strong direct experience to those issues that could strengthen our own use of evaluation as well as their control of the evaluation process in their own settings.

Document(s) 17 of 24