Government of Canada

Foreign Affairs, Trade and Development Canada

www.international.gc.ca

Guide to Gender Sensitive Indicators


August 1997

Catalogue No. E94-266/1997
ISBN 0-662-63145-5

Introduction

The Guide to Gender-Sensitive Indicators and the Project-Level Handbook are the final products of a 14-month research project completed with the Division for Women in Development and Gender Equity in CIDA's Policy Branch. Both the Guide and the Handbook were researched and compiled by Dr. Tony Beck of the Institute for Asian Studies at University of British Columbia, and Dr. Morton Stelcner of Concordia University's Department of Economics. The project was managed by Dr. Rajani Alexander, Senior Policy Analyst with the Women in Development and Gender Equity Division. The Division would be happy to receive any comments you have regarding these documents and their application in the field. I am very pleased to have been asked to write this Introduction as the former Director of the Division. I will continue to follow the use and adaptation of these tools with interest.

Marnie Girvan


Table of Contents

Chapter 1
Introduction to the Guide
1.1 Introduction
1.2 How to use this Guide

Chapter 2
The Purpose of the Guide and its Audience
2.1 Background
2.2 Purpose
2.3 The audience of the Guide

Chapter 3
What are gender-sensitive indicators and why are they useful?
3.1 Indicator Concepts, their History and Present Use
3.2 Categories - Quantitative and Qualitative Indicators
3.3 Indicators and development objectives
3.4Types of indicators - risk, input, process, output and outcome
3.5 Criteria for the selection of indicators

Chapter 4
Indicators at the Branch and Region/Country levels
4.1 Indicators at the Branch level
4.2 Indicators at the Region/Country level

Chapter 5
Indicators at the project level
5.1 Introduction
5.2 Indicators in the education sector
5.3 Indicators in the health sector
5.4Indicators of participation
5.5 Indicators of empowerment
5.6 Guiding ideas for indicator use at the project level

Bibliography

Annexes 1-11


Chapiter 1 - Introduction to the Guide


1.1 Introduction

This Guide explains why gender-sensitive indicators are useful tools for measuring the results of CIDA's development initiatives. It concentrates in particular on projects with an end-user focus, and shows how gender-sensitive indicators can and should be used in both gender integrated and WID-specific projects, and in combination with other evaluation techniques. After introducing concepts, the Guide reviews the techniques of choosing and using indicators at the project level, so that CIDA staff can utilize them as an instrument of results-based management. The key questions addressed here are:
  • What are gender-sensitive indicators?
  • Why should CIDA use them?
  • What are the types of such indicators?
  • What are their limitations?
  • How can they be used at the branch and region/country levels and in particular in projects with an end-user focus?
1.2 How to use this Guide

You will make the best use of this Guide by first reviewing the concepts and the background to indicator use (chapter 2). This will make it easier to match development objectives and indicator categories (chapter 3) with your own mandates and particular tasks. It should then become clearer why and how gender-sensitive indicators will be relevant to your work at all levels, from branch and country/region to project (chapters 4 and 5). To assist you in reflecting on the discussion, summaries of chapters 3, 4 and 5 have been provided.

To try out the ideas and techniques covered in this Guide, you can turn to the examples and scenarios provided in the Annexes. To assist you in further exploring the conceptual and practical aspects of indicators, a comprehensive bibliography has been included. A shorter project level Handbook on gender-sensitive indicators, for quick reference, has also been produced to accompany this Guide.

The Guide is not intended to give exact answers to all questions on indicators, or to set up a format for indicator use that should be used or replicated in all instances. Instead, it is intended as a document that makes suggestions about indicator use and how indicators might be adapted for different purposes. The authors' hope that it will be a helping hand rather than a blueprint model.


Chapter 2 - The Purpose of the Guide and its Audience


2.1 Background

Since the early 1990s most development agencies have been strengthening their evaluation capacities. CIDA's Agency-wide program on results-based management is its mechanism for achieving improved evaluation of its endeavours. A key feature of results-based management is the development and selection of indicators that can measure development effectiveness at the program as well as project level.

As Women in Development and Gender Equity (WID/GE) is a cross-cutting theme in CIDA, the WID/GE Division has been working to develop a set of gender-sensitive indicators specifically related to CIDA's WID/GE Policy. The purpose of this is to enable Agency staff to measure WID/GE results more effectively, which in turn will lead to a greater understanding of how results can be achieved, and feed back into better program delivery. The 1995-2000 WID/GE policy states that "gender sensitive performance indicators be contained in Branch Plans, program objectives and priorities, R/CDPFs, and in program/project documentation" (CIDA 1994k: Annex 1, p5). However, indicator use by donors is not yet well developed, and to the best of the present authors' knowledge, this is the first guide that deals with gender-sensitive indicators at all the different levels at which CIDA operates (Branch to project).1

In October 1994, CIDA's WID/GE Division organized a full-day workshop for an initial discussion on gender-sensitive indicators, where it heard from five specialists in the field. One decision to emerge from the workshop was to produce a more comprehensive Guide. There followed a year of extensive consultation and review of this Guide and its accompanying Handbook with CIDA staff in Gatineau and in the field, other practitioners and academics. In all about two hundred people were consulted during the Guide and Handbook's preparation, and many read the Guide and Handbook in full and provided extensive and useful comments, for which the authors' are grateful.

2.2 Purpose

The aim of this Guide is to:
  1. Promote conceptual and methodological understanding of indicators, with special emphasis on gender-sensitive indicators.
  2. Offer suggestions and guidance for use of gender-sensitive indicators, with a particular focus on projects with an end-user focus.
Readers should note that this is a Guide toindicators and not a Guide to monitoring and evaluation, which covers a much wider field. Indicators make up only one part, albeit an important part, of monitoring and evaluation work. A detailed and very useful discussion of monitoring and evaluation at the project level can be found in the handbook for policy makers and project staff by Bamberger and Valadez (1994).

2.2.1 A focus on projects

The Guide focuses on results at theproject level, rather than internal WID/GE implementation within CIDA. This is not to suggest that either of these two areas is more important than the other. However, as noted by the DAC, most of the focus of institutional strengthening within donors has been on internal institutional processes (CIDA 1994i). Less attention has been paid to developing indicators related to program delivery. For this reason the WID/GE Division, in consultation with a number of Agency staff, decided to produce a Guide with a primary concentration at the project level, and in particular on projects with an end-user focus.

The Guide does not, therefore, cover important areas of CIDA's work such as institutional strengthening, capacity development or policy dialogue. While it is crucial to improve monitoring and evaluation of these initiatives, it was felt that to cover these areas was not feasible within a single guide.2

2.3 The audience of the Guide

The Guide will be useful to anyone in the Agency working with indicators. The discussion of concepts and methods of indicator use explains why indicators are useful and what their limitations might be. Project level staff in particular will find both the rationale and the strategy for using indicators in carrying out their project responsibilities. In the Annexes you will find specific methodological information on the advantages and disadvantages of the use of particular indicators.

Because it focuses on the project level, the Guide will be of particular use to any Agency staff who wish to strengthen the WID/GE evaluation component of their project work.


Chapter 3 - What are gender-sensitive indicators and why are they useful?


3.1 Indicator Concepts, their History and present use

3.1.1 What is an indicator?

An indicator is a pointer. It can be a measurement, a number, a fact, an opinion or a perception that points at a specific condition or situation, and measures changes in that condition or situation over time. In other words, indicators provide a close look at the results of initiatives and actions. For this reason, they are front-line instruments in monitoring and evaluating development work.3

Bauer (1966) described social indicators as "statistical series, and all other forms of evidence....that enable us to assess where we stand and where we are going with respect to values and goals, and to evaluate specific programs and determine their impact." This definition is useful because it recognises the normative nature of indicators, in that a change in a particular direction can be interpreted as "good" or "bad". For example, a rising birth rate may be interpreted as good in one country but bad in another. It also recognises that indicators can come from "all... forms of evidence", both quantitative and qualitative; and that indicators must measure changes over time. Because of their normative nature, care must be taken in defining the norm or benchmark implicit in any indicator and against which change is measured. For example, in examining the status of women, is the norm the situation of men in a particular country, or is it women in other countries?

Gender-sensitive indicators have the special function of pointing out gender-related changes in society over time. Their usefulness lies in their ability to point to changes in the status and roles of women and men over time, and therefore to measure whether gender equity is being achieved. Because use of indicators and other relevant evaluation techniques will lead to a better understanding of how results can be achieved, using gender-sensitive indicators will also feed into more effective future planning and program delivery.

3.1.2 Political bias and indicator use

As tools for measuring social change, indicators are subject to political forces. It is important to recognize before using indicators that all indicators have their own political heritage and bias. This is important because certain types of indicators, particularly so-called quantitative indicators, receive legitimacy as they are considered 'objective'.

However, gender specialists have pointed out that the use of economic indicators, for example in relation to employment, has until quite recently been used in a way which has excluded women's work from analysis (Waring 1988). This serious omission has been seen as a consequence of the fact that most censuses and large scale surveys, particularly in developing countries, are organised and carried out by men and for male policy-makers.

There have been two responses to this gender-bias. Firstly, gender specialists have promoted the collection of sex-disaggregated data which has been used to challenge gender bias in indicator use and to advocate for policy changes. Perhaps the best known example of this is the UN publication The World's Women 1970-1990, the preface of which states that its central concern is to (UN 1991: xi): "provide(s) concerned men and women with information they can use to inform people everywhere about how much women contribute to economic life, political life and family life and to support appeals to persuade public and private decision-makers to change policies that are unfair to women." The 1995 UNDP Human Development Report makes similar statements about the need to bring gender-sensitive indicators to the attention of policy-makers as a first step towards changing policies biased against women. Here, gender-sensitive indicators are not ends in themselves but a political tool to be used to challenge the status quo.

Other gender specialists, particularly from the disciplines of sociology and anthropology, have argued that indicators by themselves are insufficient to capture women's experience, for example in areas such as women's empowerment or participation (Reinharz 1992). They argue that policy-makers need to pay more attention to women's experience, towards which indicators can be a pointer.

This Guide takes a stance that includes both of these positions, looking at how a wide range of indicators can measure changes in gender inequity and women's status over time.

The political nature of indicator use must be kept in mind particularly in relation to qualitative indicators, because it is often claimed that such indicators are 'subjective' or unreliable and therefore of little worth. The truth is that such indicators are essential for promoting stakeholder participation in projects, as discussed in section 3.2.3.

The evolution of indicator use can be summarized as follows:

The evolution of indicator use

Before the 1970s most attention was paid to economic indicators, particularly those relating to GNP and national accounts systems. This was due to the development focus of the time on economic growth and infrastructure development.
  1. As aid philosophies moved towards human-centred development and basic needs during the 1970s and early 1980s, an extensive literature developed on what were termed "social indicators", that is, indicators related to health, education, employment and population. Social indicators were developed to supplement economic indicators, with a specific purpose related to advocacy around social issues. However, little of this early literature was gender-sensitive.
  2. From the mid-1980s there has been a stronger focus on indicators of empowerment and participation, and on gender-sensitive indicators, again mirroring a shift in development priorities. The redefinition of priorities and the recognition of women as stakeholders in development have established gender-sensitivity as a necessary condition of development efforts. Advocacy by women's organizations and the insights of WID specialists in development agencies have underscored the value of sex-disaggregated data. However, the development of gender-sensitive indicators is still at an experimental level (Beck 1994; Oakley 1991).

3.1.3 Indicator use and WID/GE at the donor level

Most agencies recognize that their monitoring and evaluation capability, including their ability to develop and use indicators, is weak (CIDA 1994b; World Bank 1994a). A review of donor literature also shows that the terminology employed in indicator use is inconsistent. How agencies have dealt with indicators so far can be summed up in the following four general points:
  1. Indicator systems have usually been developed by donors in a non-participatory fashion, using expertise and knowledge from developed countries. As such, the indicator systems designed may not be relevant to social scientists or women in developing countries.
  2. Most donor work has dealt with indicator questions such as, "Were women included in project planning?" rather than indicators themselves. Such questions are usually less precise and more difficult to use than indicators.
  3. Most agency documents have put their energies into developing indicators of the implementation of WID policies within the agency, focussing, for example, on funds devoted to WID, WID training and the hiring of WID specialists. Less work has been done on the development of indicators to evaluate the effectiveness of development programming.
  4. Much more work has been done on quantitative than qualitative indicators. Some recent work has experimented with the use of qualitative indicators (see chapter 5), but the development of such indicators in agency literature is still in its infancy (CIDA 1994i). However, even the production of quantitative sex-disaggregated data in project evaluations is not widespread within DAC members. CIDA staff should be aware of these facts when attempting to institutionalize the use of quantitative and qualitative gender-sensitive indicators within their work.
There are no gender-sensitive indicator systems or models currently in use at the project level that have proven themselves effective. However, a number of agencies have carried out research in the last few years, and some of the on-going work on indicators relevant to CIDA's requirements is highlighted below.

Recent agency work on gender-sensitive indicators

The World Bank has recently explored the extension of the use of Key Performance Indicators at the project and sectoral levels as a part of Monitoring and Evaluation. It has, as part of its "Next Steps" indicator program, developed a series of sectoral indicators, including indicators of population, education, agriculture, poverty and housing, some of which are gender-sensitive (World Bank 1995). The section on indicators of poverty (also published separately as Carvalho and White 1994) is a good source for a general discussion of indicators.

USAID, under its Program Performance Information for Strategic Management (PRISM), has developed a methodology for gender-sensitive indicator use, giving some examples (USAID 1994b), but the approach is preliminary, there is some confusion about typologising indicators, and qualitative indicators are largely ignored.

British ODA (1993a, b) has been developing a gendered Policy Information Marker System (PIMS) to evaluate project objectives, using a simple scoring system based on DAC requirements to evaluate how far the project has met WID objectives.

GTZ has developed the ZOPP (German acronym for "Objectives-oriented Project Planning"), which has involved some participatory development of indicators (World Bank 1994a).

UNDP has extended its work on the Human Development Index to include a country level index of gender development, and a country level index called the gender empowerment measure (UNDP 1995). The gender development index compares women's and men's life expectancy, educational attainment and income. The gender empowerment measure concentrates on gender differences in income, access to jobs classified as professional and technical and administrative and managerial, and the percentage of parliamentary seats held by women and men.


Most of this on-going work is experimental, and apart from GTZ's work, none is participatory. However, these and other recent explorations of indicators by development agencies have provided useful information for indicator development in this Guide. Readers should also note that CIDA's Performance Review Division is compiling an extensive Handbook on indicators, organized by country, sector and project, with useful information about each indicator.

Sections 3.2, 3.3 and 3.4 now outline some of the practical elements involved in selecting and using gender-sensitive indicators, focusing on qualitative and quantitative indicators, the setting of objectives, and different stages of indicator use.

3.2 Categories - Quantitative and Qualitative Indicators


The basic strategy with gender-sensitive indicators is one of using quantitative and qualitative methods in combination to measure gender-related changes in society over time.

Almost all of the agency literature on indicator methodology, including CIDA policy, stresses the need to use both quantitative and qualitative indicators for the measurement of gender equity (e.g. CIDA 1994c). However, there are sometimes disagreements about what constitutes "quantitative" and "qualitative" indicators.

In this Guide, we define quantitative and
qualitative indicators in the following way:


quantitative indicators can be defined as measures of quantity, such as the number of people who own sewing machines in a village. qualitative indicators can be defined as people's judgements and perceptions about a subject, such as the confidence those people have in sewing machines as instruments of financial independence.

While this definition is relatively simple, it hides some of the more complex ways in which these two types of indicators are used. For example, "quality of life" indicators, such as those that measure changes in a population's health, education or employment, are often confused with qualitative indicators, because both appear to refer to "quality". In fact, health, education or any other subject can be measured by either qualitative or quantitative indicators. Also, there is sometimes considerable overlap between quantitative and qualitative approaches, making it difficult to tell one from the other. Two ways of distinguishing between these two types of indicators is by their source of information and the way in which this information is interpreted and used.

3.2.1 Source of information

One useful way to distinguish between quantitative and qualitative indicators in the development field is by the different sources of information they use. Quantitative indicators focus on areas that are easy to quantify, such as wage rates or education levels. Because of this quantitative indicators are usually drawn from censuses, enumerations, and administrative records. That is, quantitative indicators are extracted from more formal surveys.

On the other hand, because they are people's perceptions and viewpoints, qualitative indicators are typically obtained from sources such as public hearings, attitude surveys, interviews, participatory rural appraisal, participant observation, and sociological or anthropological field work. That is, qualitative indicators are extracted from less formal surveys.

3.2.2 Interpretation and use

A further means of differentiating between these two types of indicators is by considering how they are interpreted and the use to which each is put. Because of their focus on formal surveys, quantitative indicators are usually interpreted using formal methods such as statistical tests, and the results of these tests are then used to suggest changes in policy. Quantitative indicators are often presented in a way that is quite distanced from the events they are describing. For example, examining increases in literacy rates may tell you very little about the incredible effort women have made to become literate. Because they are distanced in this way and are expressed in terms of numbers, quantitative indicators are often also called "hard" or "objective".

On the other hand, in development studies most qualitative indicators are generated by informal studies and the results of these studies are often presented in a descriptive fashion rather than analysed by statistical or other formal techniques, although they are subject to quantification.4 Because they describe people's viewpoints, these types of indicators are also known as "subjective" or "soft" indicators. The importance of these indicators is discussed further in the next section.

The two types of indicators are really complementary, and both are important for effective monitoring and evaluation. This is because they can cross-validate and point out problems with each other (see example 1 below).

Whatever indicator is chosen, it must meet two tests, that is the tests of "reliability" and "validity". Validity means that the information that indicators provide must be close to the reality they are measuring. Ways of ensuring an indicator is valid are: 1) common sense; 2) whether the indicator reflects similar findings in different situations; and 3) whether different survey instruments yield or uncover the same indicators. In general, the validity of an indicator can be enhanced by triangulation, or use of multiple sources of information and data. It is in this context that quantitative and qualitative approaches can be fruitfully mixed. Reliability means that indicators used must be accurate and consistent; an indicator is reliable if multiple uses of the same instrument (an interview, a survey, etc.) yield the same or similar results. Whether quantitative or qualitative indicators are chosen, they must meet these two tests to be useful.

3.2.3 Why qualitative indicators?

The importance of quantitative indicators - for example mortality or employment rates - is clear, although as noted in section 3.1.2, such indicators have been criticized by gender specialists for their gender bias. The importance of people's views or perspectives - qualitative indicators - is less clear. Users of qualitative indicators often stress that these indicators are important because they focus on people's own experience. For this reason qualitative indicators are particularly useful in understanding local people's views and priorities related to development and development projects. Proponents of qualitative indicators are often interested in establishing such indicators as part of a framework for participatory development which includes local people's views.

It is also often argued that the use of qualitative indicators is problematic because their reliability and validity is suspect. However, reliability and validity of qualitative indicators can be ensured by use of careful survey techniques. Properly developed and interpreted, qualitative indicators can play a significant role in identifying constraints to implementation and obstacles to success, which would otherwise not be readily apparent. It is also often claimed that qualitative indicators, particularly those of poor women, are of little use because such women are not educated or their views are less important than those of men. However, the importance of different kinds of local knowledge, including poor women's knowledge, is increasingly understood as it is realized that ignoring such knowledge is a major reason for the failure of development projects and programs.

The use of qualitative indicators can therefore play an important role in the promotion and understanding of stakeholder perspectives, particularly those relating to women, and in fostering participation.Developing gender-sensitive indicators in a participatory fashion requires a focus on including people's own indicators of development.

3.2.4 Qualitative analysis and the limits of qualitative and quantitative indicators

Qualitative and quantitative indicators provide a methodological gateway, but can only achieve a certain level of analysis. Functioning on that level they indicate where future questions and problems might lie. When working with indicators, there is usually a need for further qualitative analysis as to why a certain situation exists.

Qualitative analysis is used to understand social processes, why and how a particular situation that indicators measure came into being, and how this situation can be changed in the future. Qualitative analysis can and should be used at all stages of the project cycle, and should be used alongside quantitative and qualitative indicators.

An example of qualitative analysis is the examination of gender roles, that is the different roles that women and men have in society, how these came into place and how they can be changed. Qualitative and quantitative indicators can reveal certain aspects of different gender roles, but they can only go a certain way in revealing why these roles have come into place and how they can be changed. There are two useful Guides to qualitative analysis, Coady International Institute (1991), and CCIC et al (1991). These are widely available and the reader should turn to them for further information on this important subject.

Two examples will clarify the indicators-analysis continuum:

Example1

Indicators of change in rural India

An innovative study on the incidence of poverty in two Indian villages was carried out over twenty years by Jodha (1989), between 1964 and 1984. Jodha gathered quantitative indicators on household income and used these indicators to chart the fortunes of the different households; the findings from these quantitative indicators were that 38 percent of sample households had become poorer during this twenty-year period.

Unusually, Jodha also collected qualitative indicators from farmers who were deemed to have become poorer. Poor farmers' qualitative indicators or perceptions of change over the twenty-year period were phrased not in terms of declining income but rather in terms of: reduced reliance on patrons, reduced dependence on low-paid jobs, improved mobility and improved assets. Even though these farmers had become poorer in monetary terms, they considered that their quality of life had improved.

Jodha then uses qualitative analysis to examine why, because of social changes in their villages, poor people felt that they were better off, the main reasons being because they had greater independence, more mobility and were no longer at the mercy of the village elite. Jodha's is one of the best studies to show how quantitative and qualitative indicators can cross-validate each other and why there might be differences between the two kinds of indicators.

Example 2

Quantitative and qualitative indicators and qualitative analysis: women and political representation

An often used quantitative indicator is the number of women parliamentarians. This indicator can be tracked over time, and is used as a proxy for women's political participation and empowerment (e.g. UNICEF 1995).

While this quantitative indicator is useful for assessing gender inequality, if used on its own it misses key points about political participation. Qualitative indicators might be used to show the women parliamentarians' own perspective on their political representation, for example whether they felt they were marginalised or were having an increasing impact on decision-making.

Qualitative analysis would concentrate on the relationship between the number of women parliamentarians and decision-making processes, for example how many times issues related to gender equity are raised in parliament, what legislation related to gender equity is passed and enforced, whether women's speeches are making an impact, and how women could be supported in their pursuit of gender equity or other policy initiatives. It would also concentrate on the impact of women's political representation on broader social issues and the functioning of government.

3.3 Indicators and development objectives

Since indicators are tools to elicit results, they are tied to the objectives with which a development initiative begins.

When using indicators, objectives must be clearly articulated. Objectives should be determined in relation to base-line studies, against which results can be measured.

A base-line study, which should include data disaggregated by sex, socio-economic grouping and ethnicity, will reveal the situation in the project area before donor intervention, and this study can be used for comparison when using indicators through the project cycle.

3.3.1 Clear objectives as critical anchors

For various political and institutional reasons donors have faced difficulties in setting clear objectives in their programming. However, the setting of clear objectives is crucial to the functioning of results-based management because objectives have considerable influence on the way policy is implemented. Like indicators, objectives should be developed with input from all stakeholders in an iterative fashion.

Objectives also serve as "anchors" through the project cycle in two important ways:
  1. They provide a statement of the principal contributions that a program or project will make in affecting a particular condition, and,
  2. They act as a set of "information handles" to assess progress during implementation and evaluation.
Objectives may be organized in various categories and given different labels (goals, targets, purposes, etc.). However, regardless of how they are grouped or termed, the following five critical tests of a clear statement of objectives can be applied:

Critical tests for setting clear objectives

  1. Explicitness: Objectives should explicitly state intended results, and not just refer to resources to be used and activities to be performed.
  2. Clarity and precision: Terms and concepts that are used should be clearly defined, and the statement of objectives should be understood by anyone who reads or hears it, and its meaning agreed upon by all stakeholders.
  3. Feasibility or realism: Realistic objectives should be established, which take account of available resources, the external environment, as well as actual and potential constraints, obstacles, risks and enabling factors.
  4. Measurability and verifiability: Objectives must be crafted in a way that includes operational criteria that would allow verification of the achievements during implementation and upon completion.
  5. e-frames. The statement of objectives should include realistic deadlines, time-frames (or even specific dates) punctuated by intermediate targets.


In sum, objectives must be realistic, operational, and measurable. They must also be tied to a credible implementation plan that links courses of action and intermediate targets to the expected final outcomes.

However, we must recognize that donors will in some cases continue to set unclear objectives, partly because the ultimate aim of development projects and programs cannot always be clear, particularly where the development of objectives is iterative and in areas related to human development. A review of a variety of projects with an end-user focus has shown that two types of objectives are generally found, which we have typologised as follows:

3.3.2 Types of objectives


Type 1: the results are relatively easily quantifiable. Type 2: the results of which are less easily quantifiable, concern social processes, and will require emphasis on qualitative analysis to measure them.

When setting objectives, projects will often use a mix of Type 1 and 2, but will lean more towards one or the other. While your project objectives may not fit exactly into either type, thinking in terms of Type 1 and Type 2 objectives will help with the development of indicators.

An example of a Type 1 objective would be:

"Increase gross enrolment rates by 50 percent at the primary school level in the project area over a five year period, ensuring gender and socio-economic equality."

An example of a Type 2 objective would be:

"Empower poor rural dwellers in the project area so that they are able to make a contribution to decision-making in their villages."

It is relatively easy to measure changes in enrolment rates in a given area (see Annex 5). But it is more difficult to determine whether poor people have been empowered and how they contribute more to decision-making. If your project objectives are closer to Type 1, you will probably be using standard quantitative indicators of education, employment or health, and will concentrate on outcome indicators. If they are closer to Type 2, you will probably concentrate more on indicators of empowerment or participation, and will focus more, at least initially, on process indicators. Type 1 objectives also relate closely to changes in women's practical needs, and Type 2 objectives relate closely to changes in women's strategic interests. For a definition of women's practical needs and strategic interests, see CCIC (1991).

In practice, most objective statements fall somewhere in between these two examples, but the typology will be a useful one when thinking about setting indicators, and should be used as food for thought. A number of examples of the two different types of objectives are given through the Guide.

3.3.3 Objectives, indicators and lines of business

Indicators keyed to development objectives must also fit the seven lines of business (LOBs) related to the bilateral project cycle set out in CIDA's Geographic Programs Road Map. It is likely that the most often used LOBs will be the Blueprint and Iterative Models.

The Blueprint Model requires significant study, analysis and planning before approval, and will for the most part involve projects related to infrastructure, institutional strengthening and human resource development.

The Iterative Model also requires significant study, analysis and planning, although the level will be less than the Blueprint Model. It follows a philosophy of learning by doing; objectives may be set for the short term, but these are challenged by a strong management review process which shapes longer term objectives and activities. This allows for lessons learned to be incorporated into project design.

It is likely that objectives in the Blueprint Model will be closer to Type 1 objectives, and objectives in the iterative model will be closer to Type 2 objectives. In the case of the Iterative Model, objectives may change over time, but at some point in the project cycle final objectives will be set, and then indicators can be developed for them.

The following Table gives examples of objectives, indicators and models.

Model Objective Indicator Example of objective
Blueprint Type 1 Relatively straightforward Raising literacy by 50% in 'x' region over 5 years.
Iterative Type 2 More complex Increasing poor people's decision-making by 50% in 'x' region over 5 years


3.4 Types of indicators - risk, input, process, output and outcome

The rationale behind indicator use is to feel the pulse of a project as it moves towards meetings its objectives. To do so, every significant part of the project cycle should be covered by appropriate indicators. Types of indicators have been conceptualized in different ways, and after reviewing several systems of indicator organization in donor and academic documents, this Guide recommends flexible use of the typology set out below, which is the most common indicator typology found. In this typology, it is useful to think of using indicators in a chain, starting from input through to outcome. A short definition is given for each type, followed by a more detailed discussion and example.5

Definitions of indicators

Risk/enabling:
measure the influence of external factors on the project of program
Input: concern resources devoted to the project or program
Process: measure delivery activities of the resources devoted to a program or project.They monitor achievement during implementation, serving primarily to track progress towards the intended results.
Output: identify intermediate results, for example at a point when donor involvement is close to complete.
Outcome: relate directly to the longer-term results of the project, and after donor involvement is complete.
Numbers of indicators can be quite small, and a rule of thumb is that up to six indicators should be chosen for each of the above types.


3.4.1 Risk/Enabling Indicators

Experience shows that at all stages of its cycle a project may be affected by a variety of risks or enabling features. So by risk/enabling indicators we mean those factors external to a project that contribute to the project's success or failure.

Risk/enabling indicators include socio-economic and environmental factors, as well as the operation and functioning of institutions, the legal system, and socio-cultural practices. For example, in an income-generating project indicators of risk would be the attitude of local institutions or the local elite, or the potential for marketing of goods produced.Like all indicators, they should be developed with stakeholder participation, as end-users in particular are likely to know most about potential project risks/enabling factors.

It would be worthwhile to explore the possibilities of scoring risk/enabling factors at the project level to isolate CIDA's responsibility for success. For example, risk and enabling factors could be scored on a scale of 1-5. In an income generating project in a particular region, a major political or environmental disruption in that region that negated the effects of the project could be scored '5'. Less devastating disruptions, for example floods that made a road impassable for three months, could be scored '1' or '2'. Some experimentation would be needed at the project level before it was possible to see if such a system is feasible.

Related questions are to determine which results in the project area are caused by factors outside of the project's control, and what the effects of the project are on the area outside the project. For example, how can the intervention of another donor in a similar sector to the one in which CIDA is working be factored in to evaluation. This short Handbook cannot deal with these important questions, and the reader should turn to specialist advice related to them which can be found in Bamberger and Valadez (1994).

3.4.2 Input indicators

Also called "resource" indicators, they relate to the resources devoted to the project or program, for example funding, human and non-human resources, infrastructure, institution-building, and other means by which a program or project is put into effect. They play an important role in flagging potential problems and identifying their causes, but input indicators alone will not reveal whether or not the project or program will be a success.

In an education project, input indicators might include: credit disbursed; materials purchased or adapted; views of the community as to the feasibility of the project; or trainees identified.

When project funding begins and input indicators are used it is also crucial to collect base-line data from which results can be measured. An example of base-line data in an education project might be literacy and enrolment rates, disaggregated by sex, ethnic and socio-economic grouping.

3.4.3 Process indicators

Also called "throughput" or "activity" indicators, they reflect delivery of resources devoted to a program or project on an on-going basis. As such, they are the best indicators of implementation and are used for project monitoring. However, while they reflect achievement of results, they should not displace measures of distal outcomes. A process may be successful at the same time as the outcome is a failure, as noted in this piece of wry folk wisdom, "the operation was a success, but the patient died," or, "the schools were built on time, teachers were hired, parent involvement was high, as were enrolment and graduation rates. But the graduates were unemployable because they lacked relevant skills."

Some projects, for example projects with an empowerment focus or projects which concentrate on capacity development and institutional strengthening, will rely more on process indicators, because they involve long-term change over many years. But even in these projects the focus for evaluation once donor funding is complete should be on outcome.

In an education project, process indicators include: views of the community as to facilities being built; number of facilities in operating condition; enrolment rates of girls and boys; or amount of stipend disbursements.

3.4.4 Output indicators

Output indicators are often used in project evaluations, but are less useful than outcome indicators as they do not track distal results. This is because output indicators measure intermediate results concerning products and services that are delivered when a program or project is completed, but not longer-term results.

One of the most important tasks in use of indicators is to carry out evaluation at the outcome as well as the output level.

In an education project output indicators might include: number of girls trained; opinions of teachers on training facilities provided; or number of facilities in operating condition.

3.4.5 Outcome indicators

Outcome indicators concern the effectiveness, often long-term, of a program or project as judged by the measurable change achieved in improving the quality of life of beneficiaries. They are also known as "impact" indicators.

In most cases, the primary emphasis in using indicators should be on outcome, because this best measures distal results.

Examples of outcome indicators in an education project are: views of parents on the benefits of schooling; number of girls and boys employed from project schools; type of employment; or impact of employment on women's empowerment.

Many program and project evaluations use input or process indicators rather than outcome indicators. Reasons for this include lack of resources devoted to evaluation and a lack of institutional capacity for evaluation. Use of outcome indicators will often involve long-term tracking of participants and in-depth qualitative analysis. But these should not be regarded as strong arguments against using outcome indicators.

The cost of using outcome indicators will not normally be prohibitive, and will be repaid at a later date if the intensive lessons learned from the use of these indicators can be applied in other programs or projects.

As is apparent in the examples given above, indicator types are sometimes difficult to separate, in particular process and output indicators. A number of other indicator chains are also available for use. For example, CIDA's Performance Review Division (CIDA 1995b) has suggested using a similar four part chain to the one recommended here, based on the Logical Framework Analysis and its terminology of "input, output, outcome and impact". In practice it is clear that the label, e.g. process or output, attached to indicators is less important than a clear understanding of what the indicator is measuring. The other key point to remember is that indicators should cover all stages of monitoring and evaluation, through the project cycle.

3.4.6 Indicators, time-frames and sustainability

Time-frames related to indicators should be as clear as is feasible. Setting clear time-frames will be easier in projects with Type 1 objectives, as these objectives set out clear project features to be accomplished. This task will be more difficult with more iterative projects with Type 2 objectives, where objectives and subsequently time-frames may change as the project progresses. But even in projects with Type 2 objectives, a final set of objectives should be formulated at some stage during the project cycle, after which approximate time-frames can be set.

A related question is which indicators to use at which stage of the project cycle. Ideally, input indicators should be used at or close to the start of the project, at which point base-line data is collected. Process indicators should be used while the project is proceeding, for purposes of monitoring, and until near the end of donor involvement. Output indicators should be used near to the end of donor involvement, and outcome indicators should be used after donor involvement is complete. Process, output and outcome indicators can then be compared against each other and against base-line data in order to determine how far objectives have been met. In some cases, the same indicator will be used to measure process, output and outcome (e.g. disaggregated enrolment figures, literacy rates or the local community's level of satisfaction with the project), the difference coming in that indicators are used at different points in time.

A rule of thumb is that outcome should be determined three to five years after donor funding has ended, using outcome indicators. Waiting for three to five years after donor funding has ended will allow an assessment of whether or not the project is likely to achieve sustainable results. In the case of larger, more complex projects, it may also be necessary to revisit the project area more than five years after funding has ended in order to assess sustainability. But remember that indicator use over time cannot always be organised in such a clear way because projects often do not have clear beginnings and ends.

3.5. Criteria for the selection of indicators

There is obviously no such thing as a set of universal indicators. Users must design and adapt indicators for their own purposes. The most important criteria to bear in mind are:

Criteria for the selection of indicators
  • Indicators should be developed in a participatory fashion, including all stakeholders wherever possible.
  • Indicators must be relevant to the needs of the user, and at a level that the user can understand.
  • All indicators should be sex-disaggregated.
  • Both qualitative and quantitative indicators should be used.
  • Indicators should be easy to use and understand.
  • Indicators must be clearly defined.
  • The number chosen should be small. A rule of thumb is that up to six indicators can be chosen for each type of indicator (input - outcome).
  • Indicators should be technically sound.
  • Indicators should measure trends over time.
  • The ultimate focus should be on outcome indicators.

More detailed information on each of the points listed above is given in Annex 2.

Chapter Replay

Indicators are not neutral tools; like all methodologies, they are influenced by political values and contexts.
  • Gender-sensitive indicators measure gender-related changes in society over time.
  • Qualitative and quantitative indicators should be used together. In recent thinking, qualitative methods have assumed greater importance, as they focus on empowerment and participation.
  • Objectives must be as clearly set as possible, and indicators must be closely related to objectives.
  • Indicator types correspond to each part of the project cycle: risk/enabling, input, process, output, outcome. The terms used for
  • indicators are less important than how the indicators are defined.
  • Base-line studies should be conducted, using data disagreggated by sex, socio-economic and ethnic grouping and age, against
  • which progress and results can be measured.
  • The most important indicators for project evaluation are outcome indicators.
  • The time-frame within which objectives are to be met and indicators to be used should be set out as clearly as possible.


Chapter 4 - Indicators at the Branch and Region/Country levels


When developing gender-sensitive indicators at the Branch and Region/Country (R/C) level, CIDA staff will need to decide whether:
  1. Branch or R/C results can simply be "rolled up" from programs and projects; or
  2. Whether the sum of results at Branch or R/C level is greater than the constituent parts at the program and project level.
When considering these points, it will be useful to differentiate between type 1 and type 2 objectives (see section 3.3.2 and below). Setting indicators at the Branch and R/C level also requires attention to the specificity and time-scale of objectives. This is illustrated in the next section.

4.1 Indicators at the Branch level

The Guide reviewed Branch level plans setting out WID objectives and indicators from Asia, Africa and Middle-East, and Multilateral Branches. The example given below in Table 4.1 is a generic example adapted from the Branch level plans, to illustrate how objectives can be set and indicators can be used.

4.1.1 Objectives and indicators

To illustrate whether Branch results can be rolled up from project and program results we will use examples of type 1 and type 2 objectives from the Table.

The type 1 objective, Increased number of girl children attending primary school, can be measured by adding up enrolment rates of girl children in CIDA project areas throughout the region, and the result will be the sum of the individual projects CIDA is supporting.

The type 2 objective, also from the Table,The promotion of full participation of women in political and social decision making, is more difficult to measure quantitatively. Because the objective is broad, rather than adding up the effects of individual projects, it would be necessary to trace the effects of overall programming through a careful qualitative analysis of the part women play in decision making.6

Table 4.1 Objectives and indicators at the Branch level

Objective Type of objective Indicator Type of indicator
More women gainfully employed Type 1 Data reflecting increased employment and increased income for women from projects. Process / outcome
Increased number of girl children attending primary school. Type 1 Newly registered girl children in primary schools. Process
Improved advisory and credit services for small and medium entrepreneurs, including women entrepreneurs. Type 1 No. of small and medium entrepreneurs seeking and receiving advice. Process
No. of new loans. Input
No. of businesses promoted Input
Improved quality of primary education for girls. Type 2 No. of classes prepared and of teachers trained for girls' education.. Input/process
Promotion of full participation of women in political and social decision making and the improvement of their socio-economic condition Type 2 Increased number of women's groups promoting fundamental change and more equitable development. Process


The indicators in Table 4.1, taken from Branch plans, are mainly indicators of input and process, which limits their usefulness, because it is outcome indicators that are likeliest to tell us whether objectives have been met and results achieved. It would have been relatively easy to set outcome indicators for type 1 objectives. For example, the process indicator Number of classes prepared and of teachers trained for girls' education is limited, because it is possible for there to be an increase in teachers' trained at the same time as a decline in the quality of education. One outcome indicator for the objective Improved quality of primary education for girls could have been Thirty per cent increase in literacy among girls in primary education, which would reveal the outcome of increased teacher training and classes prepared.

Table 4.1 shows that indicators are likely to be closely tied to objectives. For example, the phrasing of the objective Increased number of girl children attending primary school means that it is likely to require a process rather than an outcome indicator to measure it, in this case increased enrolment.

4.1.2 Indicators and time-scales

Table 4.1 also illustrates that thetime-scale of objectives is important in indicator choice. In the case of the quality of girls' education, CIDA's input to training teachers for girls' education might not lead to increased literacy for several years, whereas it is possible to measure the number of teachers trained over a given year. So if objectives are being set for one year, a process rather than an outcome indicator may be more relevant.

4.1.3 Indicators and non-specific objectives

When measuring WID results at the Branch level, CIDA will need to decide about the level of specificity it requires (see section 3.2.3). For example, the objectives More women gainfully employed and Increased number of girl children attending primary school do not specify how many more women or girls are required to participate before the objective is met. Similarly the objective Improved quality of primary education for girls does not specify how far the quality of education is required to improve for the objective to be met.

4.2 Indicators at the Region/Country level

The Guide reviewed present or past Region/Country (R/C) level gender strategies and objectives for Bangladesh, Indonesia, the Philippines, Jamaica, Vietnam and Zimbabwe, as well as a consultant's paper on a gender equity strategy for CIDA in Central America. Gender-related objectives were consistent across countries, and focused on four main areas:
  1. implementation of WID & GE in CIDA;
  2. ensuring that WID & GE are included throughout the project cycle;
  3. istrengthening of institutions in the developing region/country;
  4. policy dialogue.
Objectives 1 and 2 relate to the implementation of WID/GE, already well covered in agency literature (see section 3.1). For these objectives it is relatively easy to develop indicators (e.g. of resources or of training). Indicators of objective ii can be rolled up from the project to the R/C level.

Objectives 3 and 4 present greater difficulty in developing indicators. It will be easier to develop input and process indicators than outcome indicators for these objectives. For example, it will be possible to measure the resources devoted to institutions, but it may be a number of years before those resources have any effect. There has been little work done on indicators for the measurement of institutional strengthening; CIDA's Pakistan Desk is developing such indicators, although not from a gendered perspective (see CIDA 1995).

Indicators of policy dialogue are difficult to develop because of questions of causality, that is whether a particular intervention related to women caused a change in policy. Such indicators have been developed successfully for economic reform (Mosley et al 1991), but because many policy areas related to gender are less tangible than economic reform, gender-sensitive indicators will be more difficult to develop at this level.7

Again, at the R/C level, dividing objectives into Type 1 and Type 2 will be a useful method to determine whether indicators can be easily developed or not.

4.2.1 Setting baselines

One of the most important tasks in using indicators is to set baselines from which progress or regress can be measured. CIDA's Asia Branch has produced a Resource Guide for the development of generic gender-sensitive base-line indicators for use at the R/C Program level (CIDA 1995a). Chapter 3 of the Asia Branch Resource Guide provides a comprehensive set of indicators from the political, institutional, legal, economic, social, demographic and environmental spheres. While this indicator set is specific to the Asian region and the priorities of women there, it can be adapted for use by other Branches for the R/C Program level.7

Chapter Replay

In choosing gender-sensitive indicators, a vital question to be answered is whether Branch and region/country results can be rolled up from program/ project results. The time-scale and specificity of objectives should determine indicator choice. Gender-related objectives were consistent in CIDA initiatives across geographical Branches, focussing on: i) implementation of WID/GE ; ii) ensuring that WID/GE are included throughout the project cycle; iii) strengthening of institutions; iv) policy dialogue. The results of the latter two objectives are difficult to measure. Branch level plans to date have focused on indicators of input and process rather than output and outcome. Indicators have to be anchored in baselines from which progress or regress can be measured.



Chapter 5 - Indicators at the project level


5.1 Introduction

This chapter and its Annexes illustrate how indicators can be used at the project level in the priority sectors covered in CIDA's WID/GE policy. Sections 5.2 and 5.3 outline indicators for the education and health sectors, with examples of generic projects (constructed for this Guide), and indicators to accompany these projects. These sectors were chosen both because of their importance to women and because of CIDA's current commitment to make 25 percent of disbursements to basic human needs, of which health and education are two main elements.

Sections 5.4 and 5.5 cover empowerment and participation, two other areas that CIDA considers crucial in working towards gender equity. As the production of this Guide progressed there was considerable interest expressed in indicators of empowerment and participation, so they are covered in some detail.

You will make the best use of this chapter if you read it in tandem with Annexes 5, 6, 10 and 11, which provide further details on indicators of education, health, empowerment and participation along with more detailed methodological aspects related to gender-sensitive indicators. Annexes 7, 8 and 9 note generic indicators for employment, and the water and agriculture sectors. While we have focused on several sectors, this Guide is not comprehensive; indicators can be used in any sector in which CIDA works, and in both gender integrated and WID-specific projects.8

Since indicators related to education, health and water have been used and tested for some time, they are familiar enough to dispense with a definition here. Indicators of empowerment and participation are more experimental, so for the sake of clarity we have defined and described them before suggesting the actual indicators.

5.1.1. The need for ensuring participation

Ideally all project level indicators should be developed in close co-ordination with project stakeholders, including intended beneficiaries.

Agency literature, including CIDA policy documents, increasingly points to the need for the participatory development of indicators. In practice this is rarely done in a systematic way, partly because of cost and time constraints, partly because of a mistrust of stakeholders, and partly because of a lack of methodological knowledge as to how to formulate indicators in a participatory fashion. However, developing indicators in a participatory fashion need not be expensive or complex, and in the long term should lead to more efficient and effective projects that relate closely to the needs of local communities and reflect their priorities, and therefore to sustainable development.

Annex 4 gives details of current methods of participatory evaluation and the potential for developing indicators from this work. A review of experience and the potential for the use of participatory methods in CIDA is given in CIDA (1994l).

As work on indicators within CIDA continues, there is an opportunity for CIDA staff to collect, analyse and learn from material from past and on-going projects that have included the participatory formation of gender-sensitive indicators. As participation has been a key element in development projects for several years, there is already extensive knowledge among CIDA staff and CEAs related to participatory development of indicators, but this knowledge may be unrecorded, named in project documents as something other than "indicators," or uncollected.

5.2 Indicators in the Education Sector

The following is an example of indicator use in a generic gender integrated education project.

Example:an education project

Background

Consider a region in which agriculture is the principal source of livelihood, supplemented by home-based activities such as weaving and sewing, carried out mainly by women. Women's main production activities include growing vegetables, hoeing, weeding, harvesting, threshing, and small-animal husbandry (e.g. poultry, milking). Because of poverty, socio-economic inequality and lack of gender-equity in the region, there is a bias in the local educational system towards the education of boys from wealthier households.

All primary and secondary education is provided in village-based public schools, and teachers are mainly educated local people, with 75 percent of the teachers being men. The number of teachers is insufficient for the potential number of students. Although the law requires compulsory primary schooling from age 6 to 13 years, there is only sporadic enforcement of this.

Enrolment rates show substantial differences across different socio-economic groupings and between women and men. Gender bias is found in the enrolment rate at the primary level (grades 1-6): 50 percent for girls compared to 80 percent for boys. Bias towards wealthier socio-economic groups is also found in the enrolment rate at primary level. When socio-economic groups are classified from 1 to 5 dependent on total income, group 1 (wealthiest) displays 90 percent enrolment rate and group 5, 15 percent.

Project objective

Objective Type of Objective
To ensure socio-economic and gender equality in access to primary education, - without reducing enrolment rates - within five years. Type 1

Project components
  1. A system of incentives aimed at parents and teachers. Parents from poorer socio-economic groupings will be given monthly in-kind or cash payments if their children satisfy pre-determined enrolment and attendance rates. Parents from all socio-economic groupings will be given monthly in-kind or cash payments if their daughters satisfy pre-determined enrolment and attendance rates. The second part of this component attempts to ensure "buy-in" from households from all socio-economic groups.
  2. To maximize community involvement village-based advisory schooling councils (ASCs) with equal representation of male and female parents and teachers, and equal representation from across socio-economic groups, will be formed. The size of the payments to parents will be determined in consultation with the ASC, which is also responsible and accountable for their distribution. This component attempts to facilitate community participation and ensure on-going community involvement and therefore sustainability.
  3. Improvements in quality of school equipment: mid-day meals, and classroom equipment, such as desks, blackboards, and textbooks.
  4. Improvements in the gender content of the curriculum.
Base-line data should be collected on the number of students admitted by socio-economic grouping, sex, grade, age; public expenditure per student, e.g. teachers' salaries, meals, instructional material, equipment, etc.; number of teachers and school administrators, by sex; state of school equipment, e.g. textbook-student ratio; nature of the curriculum, e.g. time devoted to different subjects.

After collection of base-line data, the following are quantitative and qualitative indicators that can be used to measure the results of this project, using the risk and input-to-outcome typology. For further discussion of education, health, employment and water supply indicators, see Annexes 5 to 9 in the Guide.

Example

Risk/enabling indicators
  • Government support gauged by analysis of official attitudes.
  • Popular community support gauged by attendance and analysis of comments at meetings.
  • Elite support, gauged by focus group meetings and comments at meetings and interviews.
Input indicators
  • Amount of project funding.
  • Community perceptions of the feasibility of the project.
  • Degree of community input to project planning, by socio-economic grouping and sex.
Process indicators
  • Receipt of cash or in-kind payments by socio-economic grouping of household.
  • Regular monthly meetings of ASCs.
  • Equal participation across socio-economic groups and by women and men in ASCs.
  • Parental views of benefits of schooling.
  • Net and gross enrolment rates by socio-economic grouping and sex.
  • Drop-out rates by socio-economic grouping and sex.
Output indicators
  • More equitable employment of women and men teachers.
  • Equitable school enrolment rates by socio-economic grouping and sex at end of donor involvement.
  • Equitable school completion rates by socio-economic grouping and sex at end of donor involvement.
  • Equitable literacy rates by sex and across socio-economic grouping at end of primary level.
  • Improved perceived gender content in the curriculum (e.g. improved representation of women).
Outcome indicators
  • Equitable school enrolment rates by socio-economic grouping and sex three years after end of donor involvement.
  • Equitable school completion rates by socio-economic grouping and sex three years after end of donor involvement.
  • Improvements in status of boys and girls from poorer groups, in terms of health and employment.
  • Changes in community and parental perceptions of the desirability of having children from poor households and girls educated.
This would involve identifying and isolating the factors that caused specific indicators to be in place and why the project succeeded or failed. For example, it would involve analysing how, why and when members of the community participated, any constraints placed on the project by wealthier socio-economic groups and how these were overcome, and showing the links between increased education and higher status.

5.3 Indicators in the health sector

Project example 2

Background

Making sure that primary health care is available is a central task in health interventions and specially important for women and girls. As an example of how such an initiative can be evaluated, we shall consider a WID-specific health project whose principal focus is "safe motherhood," i.e., prenatal care and delivery, postpartum care, and family-planning services. The stakeholder population lives in an isolated area where child and female mortality and morbidity rates are about 50 percent higher than in the rest of the country. For generic health related indicators and discussion of some of the methodological issues with these indicators, see Annex 6.

Project objective

To promote women's access to essential health services, so that infant, child, and maternal mortality and morbidity rates are reduced to the national average, within five years.

Again, as in the education example, this objective can be defined as a Type 1 objective; it has clear aims that are relatively easy to measure over a set time.

Project components

To train, and upgrade the skills of 300 birth attendants and primary health care providers. These would be drawn mainly from women in the community, and their roles include:
  • providing advice and counselling on basic health care;
  • diagnosing reproductive-related and other health or injury problems, and prescribing limited treatment;
  • providing essential medicines and immunizations;
  • refering serious complications to regional hospitals;
  • hearing and reporting complaints about domestic violence and abuse, and;
  • collecting information and data on health in the community.
Information, education and communication are required to generate a demand for health services and family planning for women and men. Fifty per cent of the stakeholder population, of whom at least 75 percent will be women, will receive information about local health services. This should ensure sustained access to local services.

Women will be provided with greater access to more advanced medical technologies, and more skilled health care personnel. To do this, a mobile health clinic will be set up, which will visit the community according to a schedule convenient for women.

Following is a list of risk/enabling, quantitative and qualitative indicators that would be used through the project cycle.

Risk/enabling indicators
  • Women from the community have an interest in the project and are willing to participate.
  • Women are interested in being treated by more advanced medical technologies.
  • Local men support the project.
  • Local elites support poor people's access to the resources provided.
Input indicators
  • Number of health workers trained.
  • Number of buildings used as clinics, and their condition.
  • Instructional material and its gender make-up
  • Amount of medical supplies and equipment provided.
  • Provision of mobile clinic.
Process indicators
  • Regular meetings of women and health care worker.
  • Degree of education, counselling, and information provided. Feedback on this information by women.
  • Number of visits to mobile clinic, by sex of mother and child.
  • Views of activities of health workers and clinic of parents.
  • On-going data on health status.
Output indicators
  • Number of immunizations and vaccinations.
  • Satisfaction with services by mothers, and training by health personnel.
Outcome indicators
  • Reduced infant, child, and maternal mortality and morbidity rates as compared to the national average within five years.
  • Improved status of women through better female health.

5.4 Indicators of participation

Memo:

There is no agreed-upon method for the measurement of participation. Most work on indicators of participation has focussed on participation in groups. Little attention has been paid to qualitative indicators of participation.


5.4.1 What is participation and how is it evaluated?

Participation is used in two main senses in relation to development projects (Oakley 1991a):
  1. Involvement in an externally formulated project, usually providing some form of labor, for example, to a cash-for-work or income generating scheme.
  2. iInvolvement in decision-making and control over the project through the project cycle, probably also involving the provision of labor.
The use of gender-sensitive indicators to evaluate the first type of participation is relatively easy as this type of participation is close to labor-force participation. So, for example, indicators could be used to measure on a sex-disaggregated basis the amount of work done and labor provided, the quantity and quality of output, and the views of those involved.

Measurement of the second kind of participation, which is closer to what CIDA and other donors mean when they refer to participation, is much more complex. After an extensive literature review, Oakley noted: "the truth of the matter is that both conceptually and methodologically the evaluation of [the second type of] participation is still in its relative infancy" (1991: 239). Oakley also notes that there is a lack of development of indicators or analysis to measure the qualitative aspects of participation. This means that development and use of gender-sensitive indicators to measure participation will be experimental.

5.4.2 Participation Indicators

Most of the development projects related to participation have involved the formation of groups. For this reason most work on indicators of participation has focussed on group formation. However, much of the literature on indicators and participation is not gendered, in that it does not consider separately the participation of women and men. The Handbook for Social/Gender Analysis produced by Coady International Institute (1991) has suggested a gendered approach to participation in projects, focusing on gender equity in terms of differential input to the project, receipt of project benefits, and project control and decision-making. This is the approach followed here.

There has also been a substantial literature on the requirements for group formation from the environment and water sectors, much of it from developing countries, which has paralleled an increased focus on operation and maintenance of projects by local groups.9 The literatures of both these sectors as well as other studies of participation (ODA 1994; Oakley 1991) overlap in identifying a number of indicators and indicator questions relevant to WID/GE concerns regarding participation. We have listed these below as a guide to the most frequently used indicators or indicator questions.

The specific indicators selected below are fairly easy and inexpensive to collect and can be adapted for use in projects whether or not the focus of the project specifically promotes participation. Indicators of participation should be gathered in a participatory fashion, but in practice, as with most other indicators, this is rarely done. Time-frames should be made clear for each indicator.

5.4.2.1 Examples of quantitative indicators of participation

Quantitative indicators of participation can be organised through the project cycle as follows:

Identification and planning level
  • Input and risk/enabling indicators
Implementation level
  • Input and process indicators (of sustainability)
  • Input and process indicators (of control)
  • Input and process indicators (of activities)
  • Process indicators (of scale and make-up)
Evaluation
  • Output indicators (of benefit and returns)
  • Outcome indicators (of evaluation)
Examples of specific quantitative indicators follow. While disaggregation by gender is used in most of the examples, where possible each indicator should also be disaggregated by socio-economic and ethnic grouping, and age:

Identification and planning level

Risk indicators
  • Level of government support for local participation.
  • Level of support by different sectors of the local population to participation (e.g. men, women, local elites).
  • Project dominated by different sectors of the population.
  • Lack of long-term commitment by donor.
Input indicators
  • Levels of input of women/men at different levels (government departments, NGOs, local stakeholders) to identification and planning.
  • Numbers of identification and planning meetings held with local stakeholders.
  • Attendance by local stakeholders at identification and planning meetings by sex, socio-economic background, age and ethnicity.
  • Levels of contribution/participation by local stakeholders at identification and planning meetings.
  • Levels of participation by local stakeholders to base-line study.
Implementation level indicators

Input and process indicators (of sustainability)
  • Audit of resources or funds held regularly and openly.
  • Existence of a set of rules that were developed in a participatory fashion, and the extent of involvement of women and men in this.
  • Reduced reliance on external funds.
Input and process indicators (of control)
  • Frequency of attendance by women and men.
  • Number of women and men in key decision-making positions. Rotation of people in leadership positions.
Input and process indicators (of activities)
  • Project input take-up rates. These would be specific to the type of project and need to be monitored for gender sensitivity e.g. amount of loan taken, level of school enrolment, number of visits to the clinic and their increase or decrease since group formation started.
  • Levels of women's and men's inputs, in terms of labor, tools, money, etc.
  • Maintenance of physical installations by women/men.
Process indicators (of scale and make-up)
  • Number of local women's and men's groups established.
  • Membership of groups by sex.
  • Rate of growth or drop-out of membership by sex.
  • Socio-economic, age and ethnic make-up of women and men attending meetings.
Evaluation indicators

Output indicators (of benefit and returns)
  • Benefits going to men and women, by socio-economic class, ethnicity and age (e.g. increased employment, greater crop yields).
  • Benefits to the "community" (e.g. community assets such as a school created to which all have access).
Outcome indicators (of evaluation)
  • Use of benefits to men and women, by socio-economic class, ethnicity and age.
  • Uses made of community benefits, by sex, class, ethnicity and age.
  • Levels of participation by different stakeholders in evaluation.
  • Degree to which lessons of evaluation are acted upon by different stakeholders.
5.4.2.2 Examples of qualitative indicators of participation and qualitative analysis

Indicators for qualitative evaluations of participation have to be framed mainly in terms of interactions within relevant groups and organizations.

Memo:
Qualitative indicators and analysis of participation relate to three main areas: Organisational growth. Group behaviour. Group self-reliance.

Looking for indicators in these areas helps in selecting indicators that lead to evaluations of:
  • Stakeholder and end-user perceptions of levels of participation through different stages of the project cycle. This could be on a scale of one to five, or through the use of participatory rural appraisal techniques such as ranking (see Annex 4).
  • The degree of solidarity and mutual support among the group and between women and men, to be measured through responses of stakeholders and qualitative analysis of changes in group behaviour.
  • The ability of the group to moderate conflict resolution and to prevent conflict, to be measured through stakeholder responses and qualitative analysis.
  • The participation of an adequate number of women in important decision making (adequacy to be mutually agreed by all stakeholders), to be measured through stakeholder responses and by qualitative analysis of the impact of different decisions
An example of the potential use of quantitative and qualitative indicators, as well as qualitative analysis, in two participatory projects, is given in Annex 11.

5.5 Indicators of empowerment

Empowerment and participation are closely linked and pose similar problems of understanding and evaluation. Some guide-lines to keep in view are:

Memo:

Empowerment is difficult to measure, and there is no agreed-upon method for measuring it. Empowerment needs to be clearly defined if indicators of empowerment are to be developed. Indicators of empowerment should encompass both personal, and socio-economic and political change.


5.5.1 What is empowerment and how is it evaluated?

A clear definition of empowerment is crucial for determining indicators to measure empowerment. Empowerment has been defined in various ways; the following definitions cover most of the components usually considered as contributing to empowerment:

Definitions of empowerment

"Empowerment is about people - both women and men - taking control over their lives: setting their own agenda, gaining skills, increasing self-confidence, solving problems, and developing self-reliance. It is both a process and an outcome" (CIDA 1994k: Annex 3, p.3). "A term generally used to describe a process by which powerless people become conscious of their own situation and organise collectively to gain greater access to public services or to the benefits of economic growth" (ODA 1994: 32).


These definitions of empowerment can be sharpened by breaking it down to its component elements:

Components of empowerment

women's and men's sense of internal strength and confidence to face life; the right to make choices; the power to control their own lives within and outside the home; "the ability to influence the direction of social change towards the creation of a more just social and economic order nationally and internationally" (Heyzer 1994: 24)1. 1Other definitions for Latin America can be found in Radcliffe and Westwood (1993), and for South Asia in Basu (1993).


These definitions stress two main areas of empowerment:
  1. A personal change in consciousness involving a movement towards control, self-confidence and the right to make decisions and determine choices.
  2. Organisation aimed at social and political change.
The indicators chosen should fit these areas, but they will differ depending on whether the area of empowerment covered is personal growth or social and political change, or both. They will also differ depending on the objective of empowerment being measured (e.g. greater self-respect, or access to the benefits of economic growth, or the creation of a just social and economic order). Because of these differences, when a project objective is the "empowerment of women" or "people's empowerment" it is important to specify:
  • the type of empowerment;
  • rough time-scales within which project objectives will be accomplished;
  • the numbers of people to be affected; and
  • the degree to which they will be affected.
In short, choosing indicators of empowerment will depend on the way in which empowerment is defined.

Even when empowerment is clearly defined, it remains a complex problem to measure it at the project level, for the following reasons:
  • It is difficult to measure changes in states of mind (from disempowered to empowered).
  • Measuring different elements of empowerment, e.g. who is making a decision, particularly at the household level, can be difficult and time-consuming, requiring in-depth study and detailed qualitative analysis.
  • Participation is a key element in empowerment, but the measurement of participation is itself complex (see section 5.4).
  • Definitions of 'knowledge', 'self-respect' or other elements of empowerment may be culturally specific, and therefore vary between localities and by socio-economic grouping, ethnicity and age.
While these problems exist, a number of usable indicators to measure empowerment do exist. These are listed below, and a selection from the following indicators could be made for measuring the effectiveness of a range of empowerment projects. Some of these indicators have already been tested in a study of empowerment and fertility in Bangladesh using an innovative scoring system, details of which are given in Annex 10. The following is divided into quantitative and qualitative indicators. In most cases the level of empowerment or disempowerment of women can be compared to that of men.10

5.5.2 Examples of quantitative indicators of empowerment

Legal empowerment
  • Enforcement of legislation related to the protection of human rights.
  • Number of cases related to women's rights heard in local courts, and their results.
  • Number of cases related to the legal rights of divorced and widowed women heard in local courts, and the results.
  • The effect of the enforcement of legislation in terms of treatment of offenders against women.
  • Increase/decrease in violence against women/men.
  • Rate at which the number of local justices/ prosecutors/ lawyers who are women/men is increasing/decreasing.
  • Rate at which the number of women/men in the local police force, by rank, is increasing or decreasing
Political empowerment (indicators should reflect changes over time)
  • % of seats held by women in local councils/decision-making bodies.
  • % of women in decision-making positions in local government.
  • % of women in the local civil service.
  • % of women/men registered as voters/% of eligible women/men who vote.
  • % of women in senior/junior decision making positions within unions.
  • % of union members who are women.
  • Number of women who participate in public protests and political campaigning, as compared to the number of men.
Economic empowerment
  • Changes in employment/unemployment rates of women and men.
  • Changes in time-use in selected activities, particularly greater sharing by household members of unpaid housework and child-care.
  • Salary/wage differentials between women and men.
  • Changes in % of property owned and controlled by women and men (land, houses, livestock), across socio-economic and ethnic groups.
  • Average household expenditure of female/male headed households on education/health.
  • Ability to make small or large purchases independently.
  • % of available credit, financial and technical support services going to women/men from government/non-government sources.
Social empowerment11
  • Numbers of women in local institutions (e.g. women's associations, consciousness raising or income generating groups, local churches, ethnic and kinship associations) relative to project area population, and numbers of women in positions of power in local organisations.
  • Extent of training or networking among local women, as compared to men.
  • Control of women/men over fertility decisions (e.g. number of children, number of abortions).
  • Mobility of women/men within and outside their residential locality, as compared to men.
5.5.3 Examples of qualitative indicators of empowerment

Qualitative indicators of empowerment are particularly hard to agree upon, in part at least because empowerment itself is a concept that awaits a consensus around its definition.12 At the present stage of methodological debate perceptions of empowerment are more likely to be elicited by indicator questions of the following types rather than by the indicators implicit in the questions. These questions have to be reinforced by others that relate to qualitative analysis.

Indicator questions to assess empowerment
  • To what degree are women aware of local politics, and their legal rights? Are women more or less aware then men? Does this differ by socio-economic grouping, age or ethnicity? Is this changing over time?
  • Do women and men perceive that women are becoming more empowered? Why?
  • Do women perceive that they now have greater self-respect? Why? How does this relate to men's perceptions?
  • Do women/men perceive that they now have greater economic autonomy? Why?
  • Are changes taking place in the way in which decisions are made in the household, and what is the perceived impact of this?
  • Do women make decisions independently of men in their household? What sort of decisions are made independently?
Key questions for qualitative analysis
  • How have changes in national/local legislation empowered or disempowered women as opposed to men (e.g. concerning control over resources such as land)?
  • What is the role of local institutions (including women's institutions) in empowering/disempowering women/men?
  • Is the part women, as compared to men, are playing in major decisions in their locality/household increasing or decreasing?
  • Is there more acknowledgement of the importance of tasks customarily carried out by women, e.g. child care?
  • How are women organising to increase their empowerment, for example against violence?
  • If employment and education for women are increasing, is this leading to greater empowerment?
5.6 Guiding ideas for indicator use at the project level

This section and its associated Annexes have been designed after examining gender-sensitive project level indicators in several of the sectors in which CIDA works. This examination has led to the suggestions here as to how different levels of indicators (input to outcome, qualitative and quantitative) can be used throughout the project cycle.

Guiding Ideas

In most cases, indicators can only point the way to further analysis by raising important questions but not necessarily providing answers to these questions. This is particularly true in projects related to participation and empowerment, where indicator use should be supplemented by qualitative analysis.
  • All data should be disaggregated by sex, and also wherever possible by age, ethnicity and socio-economic grouping.
  • Project level indicators should be developed in a participatory fashion, with input from stakeholders wherever possible.
  • There is a need to consider indicators on a time spectrum or chain from input, through process and output, to outcome. The central focus of measuring for results should be on analysis of outcomes and outcome indicators. At present, the focus of measuring for results in CIDA and most donors is on inputs and process.
  • Indicator systems should be developed that are relevant to the social science expertise and political environment of the country in which they are to be used.
  • Because measuring for WID/GE results at the project level will mean measuring outcomes, CIDA needs to make a conceptual and financial commitment to investment in evaluation.


Chapter Replay

Indicators can be used in all sectors, and in gender integrated and WID-specific projects. Indicators must correspond to each of the project cycle stages: input, process, output and outcome. For each stage of this chain, efforts should be made to choose both quantitative and qualitative indicators. Participation and empowerment are complex and little work has been done on indicators to measure them. Their complexity demands that quantitative and qualitative indicators be underpinned by qualitative analysis. Indicators of participation have mainly reflected group/organisational dynamics. Indicators of empowerment have mainly reflected changes in personal growth of participants, organisational and political change.



Bibliography



Agarwal, B. (1995) A Field of One's Own: Gender and Land Rights in South Asia. Cambridge: Cambridge University Press.

Agarwal, B. (1995a) 'Gender and Legal Rights in Agricultural Land in India'.Economic and Political Weekly 30 (12), A39-56.

Agarwal, B. (1988) 'Who Sows? Who Reaps? Women and Land Rights in India'. Journal of Peasant Studies, 15 (4), 531-81.

Anderson, V. (1991)Alternative Economic Indicators. London: Routledge.

Asian Development Bank. (1993)Gender Indicators of Developing Asian and Pacific Countries. Asian Development Bank, mimeo.

Bamberger, M. and J. Valadez. (1994) Monitoring and Evaluating Social Programs in Developing Countries. A Handbook for Policymakers, Managers and Researchers. Washington D.C.: World Bank, Economic Development Institute.

Bardhan, P. (ed). (1989) Conversations Between Economists and Anthropologists: Methdological Issues in Measuring Change in Rural India. Delhi: Oxford University Press.

Basu, A. 'The Status of Women and the Quality of Life among the Poor.' Cambridge Journal of Economics 16, 249-67.

Basu, A. (1993) Two Faces of Protest. Contrasting Modes of Women's Activism in India. Delhi: Oxford University Press.

Bauer, R. (ed)(1966)Social Indicators. Cambridge: MIT Press.

Bazinet, L. (1994) 'Towards a Gender Equity Strategy for CIDA in Central America'. Mimeo.

Beck, T. (1994) 'What Has Worked and Could Work for CIDA'. Presentation to a CIDA workshop on 'Gender sensitive indicators: what works and what could work for CIDA', Gatineau, October, mimeo.

Beck, T. (1994a) The Experience of Poverty: Fighting for Respect and Resources in Village India. London: Intermediate Technology Publications.

Beneria, L. (1992) 'Accounting for Women's Work: The Progress of Two Decades'. World Development, 20 (11), 1547-60.

Beneria, L. (1993) 'The Measurement of Women's Economic Activities: Assessing the Theoretical and Practical Work of Two Decades.' In Westendorff and Ghai, 263-283.

Berkes, F. (ed) (1989): Common Property Resources: Ecology and Community-Based Sustainable Development. London: Bellhaven.

Bromley, D. (ed) (1992)Making the Commons Work. San Fransisco: ICS Publications.

Brown, B. (1995) Results-Based Management: Danger or Opportunity? Lessons from the CIDA Experience. Gatineau: CIDA, mimeo.

Bulmer, M. (ed). (1978) Social Policy and Research. London: Macmillan.

Buvinic, M. et al. (1989)Women and Development: Indicators of Their Changing Roles. Paris: UNESCO, Socio-economic studies 3.

Carley, D. (1981) Social Measurement and Social Indicators: Issues of Policy and Theory. London: Allen and Unwin.

Carley, D. and K. Kumar (1987) Project Monitoring and Evaluation in Agriculture. Baltimore: Johns Hopkins.

Carvalho, S. and White, H. (1994). Indicators for Monitoring Poverty Reduction. World Bank Discussion Paper No. 254, Washington D.C.: World Bank.

CCIC et al. (1991) Two Halves make a Whole. Balancing Gender Relations in Development. Ottawa: CCIC, mimeo.

CDP. (1994) 'Termination Report, CDP Training Coordinator'. Mimeo. (prepared by Roger Clayton).

Chambers, R. (1994a) 'The Origins and Practice of Participatory Rural Appraisal.' World Development, 22 (7), 953-969.

Chambers, R. (1994b) 'Participatory Rural Appraisal (PRA): Analysis of Experience.' World Development, 22 (9), 1253-1268.

CIDA. (1992a) Women in Development. A Policy Statement. Gatineau: CIDA.

CIDA. (1992b) The annual report of the WID steering committee to the President's Committee. Gatineau: CIDA, mimeo.

CIDA. (1992c) Working with Women. CIDA's Gender Strategy in Bangladesh. Gatineau: CIDA, mimeo.

CIDA. (1992d) WID Country Case Study: Indonesia. Working Paper 2E, Gender as a Cross-Cutting Theme in CIDA's Development Assistance. Gatineau: CIDA, Audit and Evaluation Division, mimeo.

CIDA. (1992e) WID Country Case Study: Phillipines. Working Paper 2F, Gender as a Cross-Cutting Theme in CIDA's Development Assistance. Gatineau: CIDA, Audit and Evaluation Division, mimeo.

CIDA. (1992f) WID Country Case Study: Zimbabwe. Working Paper 2D, Gender as a Cross-Cutting Theme in CIDA's Development Assistance. Gatineau: CIDA, Audit and Evaluation Division, mimeo.

CIDA. (1993) Bangladesh Crop Diversification Programme. Mid-Term Review. Gatineau: CIDA, mimeo.

CIDA. (1993a) Gender as a Cross-Cutting Theme in Development Assistance - An Evaluation of CIDA's WID Policy and Activities, 1984-1992. Final Report. Gatineau: CIDA, Audit and Evaluation Division.

CIDA. (1993b) 'Pakistan Household Survey Capability Programme: End of Project Notes'. Gatineau: CIDA, mimeo.

CIDA. (1993c) 'ASEAN Workshop on Network of Indicators and Statistics on Women'. Jakarta: CIDA ASEAN Regional Programme, mimeo.

CIDA. (1993d) Human Development Profile for SouthEast Asia. Gatineau: CIDA, Social Dimensions Sector, Professional Services Branch, May, mimeo.

CIDA. (1993e) Human Development: Guide for Thematic Planning. Gatineau: CIDA, Social and Human Resources Development Division, May, mimeo.

CIDA. (1993f) WID DAC Evaluation. Theme III, phase 1: WID as a Cross-cutting Issue in Development Aid Evaluation (Synthesis Report). Gatineau: CIDA, Audit and Evaluation Division, June, mimeo.

CIDA. (1993g) 'WID Terminology'. Gatineau, CIDA, mimeo.

CIDA. (1994a) 'Background Study for Viet Nam Country Development Policy Framework'. Gatineau: CIDA, Indo-China Program, June, mimeo (author Saniye Gulser Corat).

CIDA. (1994b) 'Literature Review of Social-Gender Indicators'. Gatineau: Asia Branch (author Tony Beck).

CIDA. (1994c) 'CIDA's Policy for Performance Review. Part A: Guide to the Policy. Part B: Draft Implementation Strategy'. Gatineau: CIDA, mimeo, July.

CIDA. (1994d) 'NGO Division's Allocation Criteria and Process'. Gatineau: CIDA, NGO Division Draft, September, mimeo.

CIDA. (1994e) 'Results Based Programming'. Gatineau: CIDA, NGO Division Draft, September, mimeo.

CIDA. (1994f) 'Managing for Results'. Gatineau: CIDA, YHR Division, Policy Branch, November, mimeo (author Joseph Stern).

CIDA. (1994g) 'Asia: Trends in Economic and Human Development'. Gatineau: CIDA, Asia Branch, mimeo, July.
CIDA. (1994h) 'A Hitch-hiker's Guide to Managing for Results'. Gatineau: CIDA, mimeo, June.

CIDA. (1994i) 'Assessment of DAC members WID Policies and Programmes'. Gatineau: CIDA, mimeo, December.

CIDA. (1994j) Corporate Planning Exercise, 1995-1998. Gatineau: CIDA, Multilateral Programmes Branch, mimeo.

CIDA. (1994k) Corporate Implementation Strategy for the Women in Development Gender Equity Policy. Gatineau: CIDA, Draft, mimeo.

CIDA. (1994l) Participatory Development: A Brief Review of CIDA 's Experience and Potential. Gatineau: CIDA, Social Dimensions, Policy Branch (Authors Rémy-Claude Beaulieu and Violeta Manoukian).

CIDA. (1995) Performance Indicator Tracking System. Gatineau: CIDA, Asia Branch, mimeo., January (author John Jackson).

CIDA. (1995a) Developing Baseline Gender Indicators and Analysis for Country Program Planning - A Resource Guide. Gatineau: CIDA, Asia Branch, mimeo (author Tony Beck).

CIDA. (1995b) 'The Bilateral Performance Review Model. A Draft Model and Status Report'. Gatineau: CIDA, The Model Group, mimeo.

CIDA. (1995c) Institutional and Capacity Development, Results-Based Management and Organizational Performance. Report prepared for Heather Baser (authors P. Morgan and A. Qualman), mimeo.

CIDA. (1995d) CIDA's Policy on Women in Development and Gender Equity. Gatineau: CIDA.

CIDA. (nd)Women, Human Rights and Democracy: Making the Connection. Gatineau: CIDA.

Coady International Institute (1991) A Handbook for Social/Gender Analysis. Ottawa: Coady International Institute, mimeo.

Dasgupta, P. (1990) 'Well-Being in Poor Countries'. Economic and Political Weekly 25 (31), 1713-20.

De Neufville, J. (1975) Social Indicators and Public Policy: Interactive Processes of Design and Application. Amsterdam: Elsevier.

De Neufville, J. (1982) 'Social Indicators of Basic Needs: Quantitative Data for Human Rights Policy.' Social Indicators Research, (11), 383-403.

Diener, E. (1994) 'Assessing Subjective Well-being: Progress and Opportunities.' Social Indicators Research, 31, 103-57.

Dyson, T. (1994) 'On the Demography of the 1991 Census.' Economic and Political Weekly, 29 (51+2), 3235-9.

ESCAP. (1993) 'Proceedings of the Asian and Pacific Symposium of Non-governmental Organization (NGOs) on Women in Development: Action for Equality, Development and Peace'. Manila: United Nations Economic and Social Commission for Asia and the Pacific, mimeo.

Franke, R. (1993)Life is a Little Better. Redistribution as a Development Strategy in Nadur Village, Kerala. Boulder: Westview Press.

Fraser, E. (1994) 'Reconciling Conceptual and Measurement Problems in the Comparative Study of Human Rights.' International Journal of Contemporary Sociology, 1-2, 1-18.

Galnoor, I. (1974) 'Social Indicators for Social Planning: The Case of Israel.' Social Indicators Research, 1, 27-57.

Government of Canada (1995) Canada in the World: Government Statement. Ottawa: Foreign Affairs and International Trade.
Griffin, L. and C. Ragin (1994) 'Some Observations on Formal Methods of Qualitative Analysis.' Sociological Methods and Research, 23 (1), 4-21.

Groenfeldt, D. (1988) 'The Potential for Farmer Participation in Irrigation System Management' Irrigation and Drainage Systems, 2(3), 241-57.

Heyink, J. and T. Tymstra. (1993) 'The Function of Qualitative Research.' Social Indicators Research, 29, 1-27.

Heyzer, N. (1994) Market, State and Gender Equity. mimeo.

Hirschmann, D. (1991) 'Women and Political Participation in Africa: Broadening the Scope of Research.' World Development 19 (12), 1679-94.

Hunt, R. (1989) 'Appropriate Social Organization? Water User Associations in Bureaucratic Canal Irrigation Systems.' Human Organization, 48 (1), 79-90.

IDRC. (1994) Institutional Evaluation: A Framework for Building Organizational Capacity for IDRC's Research Partners. Mimeo (authors Charles Lusthaus et al).

International Institute for Sustainable Development. (1995)Performance Measurement for Sustainable Development: Compendium of Experts, Initiatives and Publications. Winnipeg: International Institute for Sustainable Development.

Illo, J. (1994)Counting Gender In: A Monitoring System for Community Projects. Manila, Institute of Philippine Culture, mimeo.

IWRAW (1988) Assessing the Status of Women. New York: International Women's Rights Action Watch.

Jahangir, A. (1993) 'Human Rights for the Disadvantaged in South Asia.' In Democracy, the Environment and Human Rights in the Developing World. Montreal: Centre for International Academic Co-operation, Concordia University, 21-30.

Jain, P. (1994) 'Managing for Success: Lessons from Asian Development Programs'.World Development, 22 (9), 1363-1377.

Jeffres, L. and Dobos, J. (1995) 'Separating People's Satisfaction with Life and Public Perceptions of the Quality of Life in the Environment.'Social Indicators Research, 34, 181-211.

Jodha, N.S. (1989) 'Social Science Research on Rural Change: Some Gaps'. In Bardhan, P.K. (ed.),Conversations Between Economists and Anthropologists. Methodological Issues in Measuring Economic Change in Rural India. Delhi: Oxford University Press, 174-99.

Joekes, S. et al (1994) 'Gender, Environment and Population.'Development and Change, 25, 135-67.

Johnson, D. (1985) 'The Development of Social Statistics and Indicators on the Status of Women'. Social Indicators Research, 16, 233-61.

Johnston, D. (1988) 'Toward a Comprehensive 'Quality-of-Life' Index'. Social Indicators Research, 20, 473-96.

Khaleque, A. (1992) 'Work Values, Attitudes and Performance of Industrial Workers in Bangladesh.'Social Indicators Research, 27, 187-95.

Khan, H. (1991) 'Measurement and Determinants of Socioeconomic Development: A Critical Conspectus'. Social Indicators Research, 24, 153-75.

Kumar, K. (ed). (1993) Rapid Appraisal Methods. Washington D.C.: The World Bank.

Leelakulthanit, O. and Day, R. (1992) 'Quality of Life in Thailand'. Social Indicators Research, 27, 41-57.

Lusthaus, C. et al. (1994) 'Institutional Evaluation: A Framework for Building Organizational Capacity for IDRC's Research Partners'. Ottawa: IDRC, mimeo.

Mangahas, M. (1977) 'The Philippine Social Indicators Project'. Social Indicators Research, 4, 67-96.

Mbilinyi, M. (1988): "I'd have been a man": Politics and the labour process in producing personal narratives. Paper given at the Institute of Commonwealth Studies, London: University of London, mimeo

McGillivray, M. (1991) 'The Human Development Index: Yet Another Redundant Composite Development Indicator'. World Development, 19 (10), 1461-8.

McGranahan, D. et al (1985) Measurement and Analysis of Socioeconomic Development. An Enquiry into International Indicators of Development and Quantitative Interrelations of Social and Economic Components of Development. Geneva: UNRISD.

McGranahan, D. et al. (1993) 'Qualitative Indicators of Development'. In Westendorff and Ghai, 65-94.

Morris, M.D. and M. McAlpin. (1979) Measuring the Condition of World's Poor. The Physical Quality of Life Index. New York: Pergamon Press.

Moser, C. (1994) 'Evaluating Gender Impacts'. Conference on Evaluation and Development, Washington D.C.: World Bank, December, mimeo.

Moser, C. (1989) 'Gender Planning in the Third World.' World Development, 17 (11), 1799-1825.

Moser, C. (1993) Gender Planning and Development. Theory, Practice and Training. London: Routledge.

Mosley, P., J. Harrigan, and J. Toye. (1991) Aid and Power: The World Bank and Policy-based Lending. London: Routledge.

Mosse, D. (1995) 'Authority, Gender and Knowledge: Theoretical Reflections on the Practice of Participatory Rural Appraisal.' Economic and Political Weekly, 30 (11), 569-78.

Munansinghe, M. and W. Shearer (eds). (1995) Defining and Measuring Sustainability: The Biogeophysical Foundations. Washington D.C.: World Bank and UNU.

Murray, C. (1993) 'Development Data Constraints and the Human Development Index'. In Westendorff and Ghai, 40-64.

Nanda, A. (1994) 'Census 1991. Challenges and Innovations'. Economic and Political Weekly, 29 (51-2), 3227-31.

National Commission. (1992) Statistics on the Filipino Women. Manila: National Statistics Office and National Commission on the Role of Filipino Women, mimeo.

National Commission. (1993) 'Guidelines for Developing and Implementing Gender-Responsive Projects and Programs'. Manila: National Commission on the Role of Filipino Women and National Economic and Development Authority, mimeo.

National Commission. (n.d.) Gender-Based Indicators: A Preliminary List. Manila: National Commission on the Role of Filipino Women, mimeo.

Nussbaum, M. and A. Sen. (eds)(1993) The Quality of Life. Oxford: Clarendon.

Oakley, P. (1991a) 'The Concept of Participation in Development'. Landscape and Urban Planning, 115-22.

Oakley, P. (1988) 'The Monitoring and Evaluation of Participation in Rural Development'. Rome: FAO, mimeo.

Oakley, P. (1991) Projects with People: The Practice of Participation in Rural Development. Geneva: International Labour Office.

ODA. (1993) 'Report on Progress in Implementing ODA's Policy on Women in Development'. London: Overseas Development Administration, Social Development Department, February, mimeo.

ODA. (1993a) 'Policy Information Marker System'. London: ODA Statistics Department, September, mimeo.

ODA. (1993b) 'Policy Information Marker System: Training Notes, First Draft'. London: ODA, September, mimeo.

ODA. (1993c) 'Social Development Handbook. A Guide to Social Issues in ODA Projects and Programmes'. London: ODA, Social Development Department.

ODA. (1994) 'Stakeholder Participation in Aid Activities'. London: ODA, Draft Technical Note no. 12.

OECD. (1977) Basic Disaggregations of Main Social Indicators. Paris: OECD Social Indicator Development Programme, Special Studies no. 4 (author Denis Johnston).

Ostergaard, L. (ed). (1992) Gender and Development: A Practical Guide. London: Routledge.

Ostrom, E. (1990) Governing the Commons: The Evaluation of Institutions for Collective Action. Cambridge: Cambridge University Press.

Overholt, C. et al. (1991)Gender Roles in Development Projects. West Hartford: Kumarian Press.

PACT. (1987) 'Monitoring and Evaluating Small Business Projects. A Step by Step Guide for Private Development Organisations'. New York: mimeo.

Radcliffe, S. and S. Westwood. (1993) 'Viva': Women and Popular Protest in Latin America. London: Routledge.

Reinharz, S. (1992) Feminist Methods in Social Research. Oxford: Oxford University Press.

Sachs, I. (1995) 'Introduction: The Quantitative and Qualitative Measurement of Development - its Implications and Limitations'.International Social Science Journal, 143, 1-10.

Salmen, L. (1987)Listen to the People. Observer Evaluation of Development Projects. New York: Oxford University Press.

Schuler, S. and Hashemi, S. (1993) 'Credit Programs, Women's Empowerment and Contraceptive Use in Rural Bangladesh.' Studies in Family Planning, 25 (2), March/April.

Sen, A.K. (1985): Commodities and Capabilities. Amsterdam: North Holland.

Smillie, I. (1994) A Results-Based Approach to NGO Activities. A Pakistan Case Study. Mimeo.

Srinivasan, K. (1994) 'Sex Ratios: What they Hide and what they Reveal.' Economic and Political Weekly, 29 (51+2), 3233-34.

Steifel, M. and W. Marshall. (1994) A Voice for the Excluded: Popular Participation in Development, Utopia or Necessity? New Jersey: Zed.

Stelcner, M. (1994) 'Gender Sensitive Indicators'. Presentation to a CIDA workshop on 'Gender sensitive indicators: what works and what could work for CIDA, mimeo.

UN. (1984)Compiling Social Indicators on the Situation of Women. New York: UN, Studies in Methods, Series F, no. 32.

UN. (1984a) Improving Concepts and Methods for Statistics and Indicators on the Situation of Women. New York: UN, Studies in Methods, Series F, no. 33.

UN. (1985) National Household Survey Capability Programme. New York: UN.

UN. (1986) Concepts and Methods for Integrating Social and Economic Statistics on Health, Education and Housing. New York: UN, Studies in Methods, Series F, No. 40.

UN. (1987) Training Users and Producers in Compiling Statistics and Indicators on Women in Development. New York: UN, Studies in Methods, Series F., no. 45.

UN. (1988)Improving Statistics and Indicators on Women Using Household Surveys. New York: UN, Studies in Methods, Series F No. 48.

UN. (1988a) Compendium of International Conventions Concerning the Status of Women. New York: UN.

UN. (1989) Handbook on Social Indicators. New York: UN, Studies in Methods, Series F, no. 49.

UN. (1989a) Compendium of Statistics and Indicators on the Situation of Women. New York: UN, Series K., no. 5.

UN. (1990) Handbook for National Statistical Data Bases on Women and Development. New York, UN, Social Statistics and Indicators, Series K , no.6.

UN. (1990a) Methods of Measuring Women's Participation and Production in the Informal Sector. Studies in Methods, Series F, no. 46. New York: UN.

UN. (1991) The World's Women,1970-1990: Trends and Statistics. Social Statistics and Indicators, Series K, No. 8, New York: UN.

UN. (1991a) Compendium of Social Statistics and Indicators, 1988. New York: UN, Series K, no.ˆ9.

UN. (1992) WISTAT: Women's Indicators and Statistics. New York, UN Department of Economic and Social Development Statistical Divsion, Series K No. 10.

UN. (1993) Methods of Measuring Women's Economic Activity. Technical Report. Studies in Methods, Series F, no. 59. New York: UN.

UN. (1994) Statistical Yearbook, 1992. New York: UN, 39th issue.

UN. (1994a) Demographic Yearbook, 1992. New York: UN, 44th issue.

UNDP. (1995/3/2/1) Human Development Report. New York: Oxford University Press.

UNESCO. (1977)Indicators of Social and Economic Change and Their Applications. Paris, UNESCO.

Uphoff, N., et al . (1990) '"Optimum" Participation in Irrigation Management: Issues and Evidence from Sri Lanka.' Human Organization, 49 (1), 26-40.

USAID. (1992a) 'A Working Paper on: the Use and Analysis of Gender Disaggregated Data and Indicators'. USAID, mimeo, (Authors Mohini Malhotra et al.).

USAID. (1992b) Engendering Development in Asia and the Near East: A Sourcebook. USAID: Washington.

USAID. (1992c) 'Gender and Generation in the World's Labor Force'. Washington: US Department of Commerce, Economics and Statistics Administration, wall chart.

USAID. (1992d) 'Gender and Generation in the World's Labour Force. Module One: International and National Trends'. USAID: Office of Women in Development, Bureau of Research and Development, mimeo.

USAID. (1993a)Women in Development. Report, FYs 1991 and 1992. Washington: USAID.

USAID. (1993b) 'Democracy and Gender: A Practical Guide to USAID Programs'. GENESYS Special Studies no. 9, Washington, (Author David Hirschmann).

USAID. (1993c) 'The Gender Dimensions of Program Effectiveness'. GENESYS and PRISM, November, mimeo.

USAID. (1994a) 'Country Gender Profiles: A Tool for Summarizing Policy Implications from Sex-Disaggregated Data'. Washington: USAID.

USAID. (1994b) 'Gender in Monitoring and Evaluation: A Tool for Developing Project M&E plans'. Washington: USAID.

USAID. (1994c) 'Gender and Sustainable Development: A Training Manual'. Washington: USAID.

Vemuri, M. (1994) 'Data Collection in Census. A Survey of Census Enumerators.' Economic and Political Weekly 29 (51+2), 3240-48.

Vermillion, D. (1990) 'Potential Farmer Contributions to the Design Process: Indications from Indonesia.' Irrigation and Drainage Systems, 4 (2), 133-1500.

Vlassof, C. (1994) 'From Rags to Riches: The Impact of Rural Development on Women's Status in an Indian Village'. World Development, 22 (5), 707-19.

Waring, M. (1988)If Women Counted: a New Feminist Economics. San Francisco: Harper Row.

Westendorff, D. and D. Ghai (1993) Monitoring Social Progress in the 1990s. Data Constraints, Concerns and Priorities. Aldershot: Avebury.

Wieringa, S. 'Women's Interests and Empowerment: Gender Planning Reconsidered.' Development and Change, 25, 829-48.

World Bank. (1992) Gender and Poverty in India. Washington D.C: World Bank.

World Bank. (1993)World Development Report. Washington D.C.: World Bank.

World Bank. (1993a) Turkey: Women in Development. Washington D.C.: World Bank.

World Bank. (1994)Gender issues in Bank Lending: An Overview. Washington D.C.: World Bank, Operations Evaluation Department.

World Bank. (1994a)An Overview of Monitoring and Evaluation in the World Bank. Washington D.C.: World Bank.

World Bank. (1994b) Social Indicators of Development. Washington D.C.: World Bank.

World Bank. (1995)Sectoral Indicators. Washington D.C.: World Bank.


Annexes 1 - 11


Annex 1 - Definitions of key terms in this Guide

(a.k.a. means 'also known as')

Indicator:

An indicator is normally defined as summarising a large amount of information in a single figure in such a way as to give an indication of change over time. Bauer (1966) described social indicators as "statistical series, and all other forms of evidence....that enable us to assess where we stand and are going with respect to values and goals, and to evaluate specific programs and determine their impact." This characterization of indicators is useful because it highlights:
  1. the inclusion of different forms of information, i.e. statistical data, as well as qualitative descriptions based on attitudinal information;
  2. the link to objectives;
  3. the diagnostic role (during the implementation and monitoring phases of projects) and the evaluative role (upon completion) of indicators; and
  4. the link to measurement with respect to values and goals, i.e the normative nature of indicators.
Indicators, as the name suggests, are best interpreted as indicative or suggestive, that is they are not prescriptive in nature in terms of providing diagnosis and remedies.

A GENDER INDICATOR provides "direct evidence of the status of women, relative to some agreed normative standard or explicit reference group" (Johnston 1985). In other words,a statistic becomes an indicator when it has a reference point against which value judgements can be made. A GENDER INDICATOR can be defined as using quantitative and qualitative measures to capture gender-related changes in society over time.

Quantitative indicator: (a.k.a. hard, objective)

Quantitative indicators are based on information gleaned from censuses, surveys, enumerations and administrative records and are measures of economic and non-economic aspects of living standards and the quality of life.

Qualitative indicator: (a.k.a. soft, subjective)

Qualitative indicators deal with living standards and the quality of life using information about perceived levels of satisfaction or dissatisfaction with personal and socio-economic conditions. Qualitative indicators are subject to some quantification, for example the numbers of people expressing satisfaction with a development project would be a qualitative indicator of the projects success. Scales using qualitative indicators, such as Lickert and Thurstone attitude scales, are frequently used in large qualitative surveys, but are too complex for evaluation of most development projects. For more details see Bamberger and Valadez (1994).

Qualitative analysis:

Qualitative analysis involves examination of quantitative and qualitative social process through use of a number of analytical techniques. Consider changes in gender roles over time as an example, where it would be necessary to use qualitative analysis along with quantitative and qualitative indicators. Quantitative indicators would be changes in time of women and men involved in child-care, house-work, or more employment for women in managerial positions. Qualitative indicators would be men's and women's views of these changes and why they had taken place. Qualitative analysis would examine the social forces at work that had caused changes, for example the role of the media, or elite groups, or of the school curriculum. In a development project, qualitative analysis would examine the social factors involved in project results, for example female and male networks and their importance, or the influence of culture. Qualitative analysis therefore explains why certain indicators of phenomena are in place and how they can be changed over time.

Type 1 objectives:

These are objectives the aims of which are relatively easily quantifiable.

Type 2 objectives:

These are objectives the results of which are less easily quantifiable, concern social processes, and will require emphasis on qualitative analysis to measure them.

Risk/enabling indicator: (a.k.a contextual, situational, external, environmental, surrounding)

These indicators serve to identify the largely exogenous or larger forces which may impede or facilitate the accomplishment of the project's objectives. Typical examples of these indicators include general economic conditions, the legal system, socio-cultural practices and traditions, institutional structures, community characteristics, internal and external political events, and environmental events.

Input indicator: (a.k.a. means, resources)

These are concerned with resources devoted to project or program: funding, human and non-human resources, infrastructure and institutions.

Process indicator: (a.k.a. throughput, activity, delivery, conversion, action)

These indicators refer to what is done with the inputs during the delivery process, i.e. the succession of tasks or activities in carrying out a project. These indicators, which are the main focus during monitoring, serve primarily to gauge or track progress towards the intended results. Care should be taken that they do not displace measures for evaluating later stages of project results.

Output and outcome indicators: (a.k.a. achievements, end, distal, effects, impacts, results, product, performance)

These indicators should be the central consideration in evaluating a project in terms of its stated objectives. A distinction can be made between "output" or "intermediate" objectives, and "long-term" or distal objectives. The former type (output) concern the mix of products and services that are delivered shortly after a donor funding of a project is deemed to have been completed, while the latter type (outcome) evaluate the effectiveness of the project over a longer time frame, preferably three to five years after funding has ended. The distinction between "outputs" and "outcomes" often hinges upon practical considerations related to the type of evaluation one wishes to perform, that is a "terminal" evaluation or a comprehensive "in-depth" retrospective evaluation.


Annex 2 - Criteria for the selection of indicators (see section 3.5)

  1. Participation. Indicators should wherever possible be collected in a participatory fashion, with input from all stakeholders. At the project level in particular, indicators should be developed in unison with intended beneficiaries.
  2. Relevance. Indicators should be relevant to the needs of the user; they should fit closely with what the user is aiming to achieve.
  3. Disaggregation. All data should be disaggregated by sex. This is a key point for the development of gender-sensitive indicators. Wherever possible data should also be disaggregated by age, ethnicity and socio-economic grouping.
  4. Comprehensibility. Indicators should be easy to use and understand. Indicators should be developed at a level relevant to the institutional capabilities of the user; there is little point developing a complex system where there is limited social science expertise.
  5. Clarity of definition. A vaguely defined indicator will be open to several interpretations, and may be measured in different ways at different times and places. For example, does the indicator adoption of a new technique by a farmer mean that he or she buys some fertilizer only once (if so, how much?), or must the purchase be repeated (if so, how often and for how long?), and should the application of the new coverage be monitored (if so, over what period?) (Carvalho and White 1994). In addition, care must be taken in defining the norm or bench-mark implicit in any indicator. For example, in examining the status of women, is the norm the situation of men is a particular country, or is it women in other countries?
  6. The number chosen should be small. There are no hard and fast rules to determine an appropriate number, but a rule of thumb is that the user should avoid two temptations:

    'information overload';
    'over-aggregation', that is, designing a composite index (such as UNDP's Human Development Index) based on aggregation and weighting schemes which may conceal important information and value judgements. In general, the number of indicators should be as small and disaggregated as possible, especially for new policy endeavours. Over time, after some experimentation, a larger set may be developed. Where composite indexes are devised, the value assumptions of selection and weighting should be made explicit, and the disaggregated components should be readily available.
  7. Specificity. The selection of indicators should stress the specific concerns of the project or program, and should reflect those things that the project intends to change.
  8. Technical soundness. Data should be reliable. The user or reader should be informed about how the indicators were constructed. In particular, a short discussion should be provided about their meaning, interpretation, and, most importantly, their limitations. Indicators must be available on a timely basis, especially if they are to provide 'feed-back' on problems during implementation. At the outset, it is important to recognize the trade-off between the cost of providing timely information on the one hand, and its quality, reliability, and precision, on the other.
  9. Forward-looking. A well-designed system of indicators must not be restricted to conveying information about only current policy concerns. Indicators must also measure trends over time. They should also act as 'policy triggers' by generating alternative policy options and by highlighting concerns that may not appear to be immediately important or even apparent, but which may become significant in the future.
  10. Adaptability. Indicators should be readily adaptable to use in different countries and circumstances. Their components should be amenable to some 'fine tuning', i.e. specific to project, sector, or regional needs, but the essence of their meaning must be retained in use in different circumstances.


Annex 3 - Examples of Region/Country Level Objectives


N.B.: Some of the following may now be out of date and are used as examples only.

Bangladesh - Gender Strategy (CIDA 1992c: 5):

'The objectives of the strategy are to:
  1. ensure that gender equity is an integral part of CIDA activities in Bangladesh;
  2. promote the participation of women in development;
  3. contribute to the strengthening of the national institutional infrastructure, both public and private, responsible for promoting the social, legal and economic status of women.'
Central American Region (Bazinet 1994: 26)
  1. 'to ensure that gender equity issues are included in policy dialogue with Central American governments, regional institutions and local and Canadian partners;
  2. to ensure that gender equity is an integral part of all CIDA CENAM programming activities;
  3. to strengthen accountability for gender equity in program and project design, implementation and evaluation;
  4. to promote the development of institutional capacity for programming for gender equity within CIDA and with partner organisations;
  5. to communicate the CENAM Program's commitment to gender equity to Canadian and local partners'.
Indonesia (CIDA 1992d: 8-9)
  1. 'continuation of the WID-integrated approach to project planning, delivery and evaluation, and the enhancement of this approach through provision of a WID support program;
  2. provision of institutional support to the Ministry for the Role of Women, as part of the support program, which will provide resources to implement integration of WID issues throughout all projects and within all executing agencies (Canadian and Indonesian) involved in CIDA projects;
  3. continuation of CIDA's lead role in Indonesia and Canada in communicating goals, activities and information on WID, specifically related to the Indonesia program.'
Philippines (CIDA 1992e: 8)
  1. 'to strengthen the Philippine institutions concerned with the status and role of women;to strengthen the capacity of women's groups and NGOs to undertake WID programming;
  2. to integrate WID in the identification, planning, implementation and evaluation of every project/mechanism.'
Zimbabwe (CIDA 1992f 13-14)
  1. 'to ensure that all on-going and planned CIDA funded projects address WID issues and involve women both as agents and beneficiaries of development cooperation, with particular consideration for their economic roles;
  2. to increase knowledge, support and commitment for WID principles by ensuring that all CEAs and co-operants as well as country partners are fully briefed and aware of CIDA's WID goals and have access to the information and mechanisms to increase WID integration in CIDA projects;
  3. to strengthen Government of Zimbabwe's (GOZ) commitment and efforts to improve both women's status and access to productive resources;
  4. to increase awareness of CIDA's WID policy in annual consultation with GOZ and promote visible communication strategies to increase awareness of the importance of WID for development effectiveness in Zimbabwe.'
  5. The authors also reviewed objectives at the country level from the section on Canadian Development Assistance to Asia provided in country background documents for 1994 for twelve countries supported by Asia Branch. In the background to objectives, five of the documents mentioned WID or gender equity as an objective (Bangladesh, Cambodia, India, Nepal and Thailand); gender was usually included in relation to overall country level programming and not as a specific country level objective for which indicators would be separately set.


Annex 4 - Participatory evaluation and qualitative indicators


It is now understood that because of a bias towards 'top-down' development, the views of stakeholders in development projects have often not been counted. In addition, the views of participants, and in particular those of rural poor women, have not generally been valued because of a bias in development projects towards the expertise of urban-based, male professionals. It has also proven difficult to find ways in which to gather and interpret the views of participants.

But, more recently, methodologies for participatory evaluation have been developed by a number of agencies and are increasingly accepted. Descriptions of two of the main methodologies follow. For more information on CIDA and participatory development, see CIDA (1994l).

Participatory Rural Appraisal (PRA)

PRA is a survey method that has been developed since the late 1980s by numerous institutions, and particularly NGOs, in about forty developing countries. Conceptually,central to PRA has been a willingness to learn from local communities, and an explicit belief that local communities, including women, have an in-depth understanding of their local environment which can be used and adapted to make development projects more relevant to these communities. Methodologically, the origins of PRA have been the dissatisfaction felt with traditional approaches to rural surveys, which tended to be extractive of information and non-participatory. PRA also grew out of Rapid Rural Appraisal, which attempted to find a middle ground between very brief 'rural development tourism' and its urban and anti-poor biases, and long-term anthropological studies which might take several years to yield results. PRA uses anthropological type methods but aims as well to let local people take the lead in evaluation and to use methods that are empowering of local people rather than extractive of their knowledge (Chambers 1994a).

PRA is a method that is relevant to all, and not just participatory, projects. It uses a number of techniques to elicit the sharing of information, in particular group discussion, which include (Chambers 1994b):

Box A4.1 Techniques of participatory rural appraisal

Semi-structured interviews. Participatory analysis of secondary sources, for example maps or aerial photographs. Participatory mapping and modeling, 'in which local people use the ground, floor or paper to make social demographic, health, natural resource... service and opportunity, or farm maps' (ibid: 960). Transect walks, involving walking with local people, and observing and listening to comments from them about the area. Time lines and trend and change analysis, involving listings of chronologies of events, people's accounts of the past, changes in land use etc. Oral histories, including people's own histories or histories of their environment. Seasonal calendars. Livelihood analysis, examining stability, crises, coping, income, expenditure, debt, etc Well-being and wealth grouping and ranking, involving the identification of groups of households according to well being or wealth, including women and those considered poorest.

These techniques have been used to foster participatory planning, budgeting, implementation and monitoring, in which local people prepare their own plans, budgets and schedules, take action and monitor and evaluate progress. While not specifically related to indicators, PRA is a useful method for the development of qualitative indicators, which often relate to questions of self-respect and economic advancement (Beck 1994a), as well as being an important tool for developing quantitative indicators.

Despite their increasingly widespread use, PRAs are not a panacea. Problems with the use of stakeholder indicators are similar to problems with participation. These revolve around whose indicators are to be counted and whose indicators excluded (e.g. who in the 'community' defines what are important indicators; are the techniques or technologies used to elicit people's indicators excluding women), and the interpretation of local people's knowledge (Mosse 1995). PRA methods may also only work where there has been substantial past contact between the project stakeholders and project staff.13

Participatory Self Evaluation

The FAO's People's Participation Programme included an element termed Participatory Self Evaluation (PSE) which provided project staff and planners with access to people's perceptions of the quality of participation achieved. In PSE, project groups and staff agree on objectives and qualitative indicators of participation. Each group's self-evaluation session involves discussion about whether these criteria are being achieved, using a simple scoring system as to whether the criteria are wholly, mostly, or partly achieved, or not achieved at all (ODA 1994: 44). Project groups can be homogeneous or heterogeneous (e.g. all female, all poor), and comparison between group responses can reveal how the project is functioning in the eyes of stakeholders.


Annex 5 - Education indicators and methodological problems with their use


A5.1 Commonly used indicators of education and literacy
  • Spending per student.
  • Literacy.
  • Numeracy.
  • Current attendance.
  • Repetition, retention and completion rates.
  • Promotion/ transition rates.
  • Drop out and absenteeism - movement out of and back into school.
  • Enrolments.
  • Level attained - years of schooling.
  • Non-formal education.
  • Age enroled and ceased.
  • Quality of schools - books, desks, teachers/student ratio, building.
  • Curriculum content.
  • Transportation.
  • Women teachers.
  • Parental involvement in school or at home - levels of satisfaction.
  • Parental costs: books, supplies, uniforms, food, transportation.
  • Private/public schools.
  • Govt. funding - secondary and post-secondary.
A5.2 Problems with educational indicators

Education indicators are among the most important for measuring the status of women. As in all indicator choice, there is a trade-off, when choosing indicators of education, between data availability, precision of definition of the indicator, and the relevance of the indicator to women. Enrolment rates, which, along with literacy, are among the education indicators most commonly used to measure the status of women, are a good example of this.

The standard enrolment indicator, the number of children enroled in primary or secondary school as a % of total number of children in the relevant age group for that level, otherwise known as gross enrolment, is problematic because it assumes an orderly and simple relationship between age group and level of education. 'In many countries, the figures for primary school enrolment in fact reach more than 100 percent, because many children of secondary school age attend primary school'. (Anderson 1991: 56). Anderson suggests instead as an indicator net enrolment ratios showing the total number of children enroled in a schooling level who belong to the relevant age group, expressed as a percentage of the total number of children in that age group, otherwise known as net enrolment. Anderson also suggests (ibid.) that the net enrolment ratio for primary schools is the most suitable educational indicator, as secondary school net enrolment ratios may reflect whether or not a country has a compulsory stage of secondary education.

One problem with net enrolment ratios is that data for them may be less readily accessible than for gross enrolment rates. In addition, a UN document on compiling indicators on the status of women (1984: 35) states that: 'Although enrolment rates may be associated with literacy levels among girls and may be used as current bench-marks so that future progress may be measured, enrolment at the elementary level is not the most significant figure. In societies where parents feel it is important to invest in the education of boys but not girls, it is likely that few girls will attain secondary levels of education, let alone a university education. Thus, it is important to obtain not only enrolment statistics, but to obtain enrolment statistics by level, and, at the higher levels, by the field of study, which may indicate the different types of educational pyramids which exist for girls and boys and also the changes in the shape of the pyramids over time.' The trade-off involved in choosing one or the other of the indicators above is illustrated in the following Table, which typologises the indicators for purposes of simplicity into high or low categories. It is apparent from the Table that there is no perfect indicator to measure gendered enrolment.

Table A5.1 Strengths and weakness of educational indicators from a gender perspective

Indicator Relevance
to women
Availability
of data
Precision of
measurement
Gross enrolment at primary level Low High Low
Gross enrolment at secondary level High High Low
Net enrolment at primary level Low Low High
Net enrolment at secondary level High Low High

There are three further points to make about enrolment ratios:
  • Enrolment ratios reveal how many people enrol but not how many attend. As the UN document states: 'because censuses do not provide data which permit an examination of absences and drop-outs, except by inference over long periods, it is important to supplement census data with other material which will provide information on this pattern of attendance.' (UN 1984: 35). However, data on drop-out rates for developing countries is not readily accessible.
  • Enrolment ratios do not cover the qualitative area of the make-up of the curriculum. 'Even when girls are attending school, they may be experiencing a very different type of educational training than are boys. In many societies the curriculum deemed appropriate for girls may be totally unrelated to potential later employment....' (UN 1984: 36). In addition, the curriculum may contain stereotypes about women which will impact on the nature of children's understanding of gender.
  • Enrolment ratios are indicators of process rather than of outcome. Outcome indicators are literacy rates, and literacy rates are usually considered more significant than enrolment ratios because they represent the outcome of schooling. However, there is a trade off in choice of indicators of literacy.Basic literacy rates, as defined by UNESCO, measure the ability to both read and write a short simple statement on one's everyday life. Some documents (e.g. UN 1989) suggest that functional literacy (e.g. the ability to read a newspaper) is a better outcome indicator; however, functional literacy rates are not usually collected for developing countries.
There are also difficulties with use ofliteracy as an indicator. Literacy reflects the functioning of the education system over a number of years. 'The average age of the developing world population over 15 is 36, therefore adult education measures reflect the average social effort for education about 20 to 35 years ago... Such prevalence measures are relatively insensitive to the current social investment in educating youth.' (Murray 1993: 36). Enrolment rates do reflect the current situation, but are also problematic, as already discussed.


Annex 6 - Health indicators and methodological problems with their use


A6.1 Commonly used health indicators
  • Prenatal & infant health, for which there are 3 key indicators:
    1. infant mortality;
    2. low birth weight;
    3. receipt of pre-natal care.
  • Prevalence e.g. low weight for height, low pregnancy weight, nutritional anaemia.
  • Distribution of food and breast-feeding, male vs female - substance abuse - drugs, alcohol.
  • Morbidity burdens, i.e. combinations of kinds of illness that require medical care.
  • Nutrition and malnutrition.
  • Chronic diseases.
  • Communicable diseases.
  • Illness duration.
  • Breast-feeding practices.
  • Safety and violence; rates of child and wife maltreatment; facilities to report and seek counselling.
  • Completeness of birth and death certificates.
  • Reported causes of death vs. actual causes (use qualitative indicators).
  • Qualitative indicators: parents' evaluations of child's health and own health.
  • Vaccinations.
  • Inoculations.
  • Emotional impairments.
  • Mental health.
  • Life expectancy rates.
  • Mortality rates - child, by gender, maternal, age-specific.
  • Accessibility to health care e.g. female doctors - female patients.
  • Access and travel time to hospitals.
  • Type of health workers.
  • Contraception, abortion rates.
  • Doctors, pediatricians /100,0000 population.
A6.2 Problems with health indicators

Indicators of health are particularly important to the determination of changes in women's status over time. Health indicators should be carefully selected. Life expectancy and infant mortality are two of the indicators most commonly used for the measurement of the status of women.

Crude death rate is defined as the number of deaths occurring in a year per thousand people in the population in the middle of the year. This indicator can be highly misleading because the higher the average age in a particular country, the higher the death rate is likely to be, so a fall in the death rate may reflect a fall in average age, not an improvement, for example, in public health (UN 1989: 48).

Life expectancy at birth may also be a problematic indicator. Whether or not a person dies in a given year depends on factors over their whole life and may not reflect present circumstances. 'Whether someone who is 70  years old will survive to 71 or not depends on a large set of factors over the whole of their lives. There is therefore a built-in time-lag, of the order of 35 years, between cause and effect. ....The only way round this problem.... is to take the shortest gap between death and birth, which must be the youngest age group. ....This information is provided by the infant mortality rate, which is defined asthe number of deaths under one year of age during a year, per thousand live births during that year.' (Anderson 1991: 62).

The infant mortality rate reflects infant and mother's health, environmental health, and general socio-economic development, and is closely related to literacy. However, the infant mortality rate, like all indicators, presents problems. Accurate infant mortality data may not be available; for example nearly every African county has data on the number of hospital beds, 'hardly any have any complete and reliable infant mortality rates.' (McGranahan et al 1985: 9). Also, the infant mortality rate may be a poor predictor of life expectancy: 'With the widespread application in developing countries of health technologies targeted to infants and children... the link between child mortality and mortality at other ages has been further weakened.' (Murray 1993: 42).

In addition, international mortality estimates disaggregated by sex often do not reflect real patterns in data, but are estimates assumed from data models because data is not available (ibid: 55).


Annex 7 - Indicators related to women, work and labor force participation, and problems with their use. Indicator use in a sample project from Egypt


A7.1 Indicators and their problems14

A UN report on social indicators has noted (1989: 53):'It is particularly important that [measures of labor force participation] be based on good data on men and women's participation. There are often particular difficulties in identifying economically active women, and the stereotype that women are usually confined to home-making activities can result in a serious loss of data on women's economic activity. The problem seems to be most pronounced in rural areas...'.

The under-representation of women's work has been one of the key areas that gender specialists have sought to redress, through pressure on public institutions in order to re-define women's work, and through more general advocacy. As a result of this advocacy, the definition of labor force participation now includes subsistence labor. The ILO now defines labor force participation as 'All persons of either sex who furnish the supply of labour for the production of economic goods and services as defined by the United Nations Systems of Accounts and Balances.' (UN 1988: 69).

The inclusion of subsistence production has made the definition more relevant to developing countries, but also means that information on labour force participation is more difficult to collect. In theory, the UN System of National Accounts requires the inclusion of all primary production together with the 'processing of primary commodities by the producers of these items in order to make such goods as butter, cheese, flour, wine, oil, cloth or furniture for their own use though they may not sell any of these manufactures.' (ibid.: 69). Beneria (1993), in a detailed discussion of the practical implementations of the resolution, makes the point that while the resolution included subsistence production in national accounts, what was to be included in subsistence production was not always clear. A survey of 70 countries in 1975 showed that most countries now attempt to incorporate subsistence production in agricultural statistics, but their treatment of other subsistence activities varies greatly (ibid.). For example, differences were found among countries regarding the inclusion in national accounts of activities such as home gardening, water carrying and food processing.

Apart from subsistence production, conceptual and methodological problems remain with estimating women's other contributions in the domestic and voluntary spheres, e.g. housework and child-rearing. The survey quoted above of the 70 developing countries in 1975 showed that 'whereas 69 included crop production and 53 covered fishing, only 6 included water carrying and none included housework [in their national accounts].' (UN 1988: 80). Beneria (1993) provides background to the conceptual and methodological problems involved in including housework and volunteer activities within the national accounts. Many authorities now recommend the production of separate or 'satellite' accounts to deal with domestic work, which would permit the generation of 'augmented' estimates of gross domestic production. Consensus regarding the inclusion of volunteer work has yet to be reached (ibid.). While conceptually there has been considerable progress concerning what women's work involves, methodologically and practically there has been little progress.

Indicators of work and employment should be disaggregated by sex and age. 'Tabulations of detailed occupation categories by sex are needed in order to monitor the extent to which sex segregation in jobs and industries exists and is increasing or decreasing.' (UN 1984: 51). Certain standard employment indicators may not capture women's work, e.g. average expected years of working life may not be a useful indicator because women enter and leave the work force many times (UN 1984).

A 7.2 Indicator use in a sample project from Egypt - Women's Initiatives Fund project

Background

The original goal of the Women's Initiatives Fund (WIF) was to improve the socio-economic situation of low-income women and to stimulate the local economy in the Qena and Aswan governorates, in Upper Egypt. This goal was to be achieved by providing credit services, training and technical assistance to female entrepreneurs by working with existing institutions in the selected governorates.

The project was located in Upper Egypt because of the high incidence of poverty and female-headed households in the area, with women often remaining in the household when men migrate seeking work. The governorate of Qena is a 'traditional' area in terms of gender relations. Many young women marry as soon as they leave school, and have little freedom to improve their economic situation. Illiteracy is high, with a marked gender bias - 79 percent for females, compared with 63 percent for all adults, and a national average of 34 percent.

CIDA's contribution was expected to be $5 million from 1990-95. After approval, the project was limited to Qena governorate because of the Gulf War.

Project Components
  1. A credit component, involving the provision of small business loans, each being up to a few hundred Egyptian pounds. This component is handled by the National Bank of Development, which expects to have issued 2,000 loans, to both men and women, by the end of the project.
  2. A New Economic Activity component, to oversee the provision of larger loans to start up small- and medium-sized businesses, or expand existing businesses.
  3. Institutional strengthening, to improve the capabilities of government, non-government and private sector organisations to support economic development activities for women. This component was to receive the largest proportion of the project funding.
The project was also to fund the execution of a baseline study to provide 'information on project-related decisions and evaluations'.

Objectives

Some objectives are stated in numerical, easily measurable, terms, e.g. 'training will be provided to fifteen CDAs'. Other objectives are stated in a less precise fashion, e.g. 'to enlarge the scope of economic activities that can provide a significant income for women'. A division into type 1 and 2 indicators (see section 3.3.2) would have facilitated the development of indicators.

The mid-term evaluation noted that the project goal of improving the economic conditions of low-income women in Upper Egypt was just starting to be realised through the credit component of the project. However, many of the loans were made to men and many of the new economic activities were male-oriented and run by men.

Indicators and examples of how WIF could have used gender-sensitive indicators.

Input indicators

Input indicators in the project documentation are the number of women who could expand or start a small business. Originally, a target of 1,000 loans was set, but this could have been clarified further by an indication of the proportion of the local population that this represents.

Other input indicators which could have been used include:
  • A listing of the institutions which could provide services to entrepreneurs.
  • Quantitative indicators could have established the number of women who enquired about the programme, and the proportion who went on to apply for funds.
  • Qualitative indicators could have registered the attitudes of women towards their economic ambitions and perceived abilities, as well as the attitudes of other members of their household towards the women's employment.
Process indicators could include:
  • Amount lent, by sex.
  • Number of applicants who receive loans, conduct business, seek advice and repay the loans, by sex.
Quantitative output indicators could include:
  • New, or expanded, successful businesses, with ownership by sex.
  • Number of new jobs created by these businesses, by sex, age and education.
  • Changes in male behaviour regarding job search (do men still migrate for work or are they able to stay with their family because of the increased work-opportunities and is there a concomitant change in number of female-headed households?).
  • Repayment rate for loans, and levels of savings of clients (by sex), from bank records.
Qualitative output indicators could include:
  • Changes in women's confidence.
  • Changes in levels of women's satisfaction.
  • Changes in women's decision-making in the household.
  • Changes in the well-being of the household - housing, clothing, food, health, schooling, etc.
  • Changes in the attitudes of other family members.
  • Future business plans of borrowers.
Outcome indicators could include:
  • Changes in the socio-economic status of women involved.
  • Increased empowerment of women, for example in decision making.
  • Uses made of extra income by women.
  • Changes in the institutions receiving assistance, e.g. have more clients been helped, has that help been more useful than that provided in the past, is the aid provided to these agencies relevant to the work they are expected to accomplish?

Annex 8 - Indicators related to water supply and sanitation - a sample project from Honduras


Background

The Rural Water Supply and Sanitation project (RWSSP), undertaken in partnership with CARE, was a gender integrated project which addressed the problem of poor health status attributable to inadequate water supplies and sanitation facilities in 78 low-income rural communities, made up of 3,236 households and some 22,500 inhabitants. The duration of the project was July 1990 to June 1995.

Objectives and project components

The main objective was to reduce the incidence of water-related diseases among the targeted rural population by providing safe water and sanitation facilities for 90% of that population within a five year period. This was to be accomplished by:
  1. the provision of potable water systems and latrine construction;
  2. micro-watershed protection and management; and
  3. community health education and organization.
Objectives and goals of the RWSSP were clearly articulated, as were the project requirements, activity targets, financial plans, and schedule of implementation.

WID/GE in RWSSP

The RWSSP recognized that women are not only the primary users of the facilities but also strongly influence household sanitary habits and overall use of the facilities. WID/GE were stressed as fundamental to the success of the project. Gender-sensitive indicators were used during the monitoring and (interim) evaluations.

In particular, the project emphasized the need for women's involvement beyond that of participation in terms of time saved and health improvements gained. Women were actively encouraged to participate in all phases, aspects and activities of the project: planning, execution implementation, and management. This was accomplished in a number of ways and was carefully monitored (starting with Report 8, Feb, 1993) during the implementation of the project.

Use of indicators

Project documents suggested using the infant mortality rate (IMR) as an indicator by which to judge the success in achieving the main objective. The CEA did not undertake the measurement of improvement in health status; this task was carried out by specialized institutions and the Ministry of Health.

In project documentation, the overall objective of the RWSSP is related to intermediate goals. Indicators are suggested to measure these goals, and it is relatively easy to divide these indicators along the recommended input to process framework as noted below. Many of the process indicators are indicators of participation (see section 5.4).

Risk/enabling indicators

Potential risks were recognized in the project, including conflicts over agricultural land-use, impacts on the environment, and community-level characteristics such as the reluctance of women to assume new roles in the project implementation and management, and increased time demands that would be placed on women's time. However, risk/enabling indicators as such were not included.

Input indicators
  • Numbers of technical personnel by sex (engineers, administrators and support staff); managers and co-ordinators (construction, education, watershed); educators and promoters (6 men and 6 women); co-ordinator for women's groups; building and office facilities; vehicles; office equipment; education material and facilities; construction equipment and materials.
  • Amount of funding.
Process indicators
  • Improvement in water quality measured by changes in coliform levels.
  • Provision of potable water measured in project documentation by the number of communities affected (e.g. gallons/person/day).
  • Percentage of beneficiaries who store drinking water in covered containers, by sex.
  • Percentage of beneficiaries who use and maintain the constructed latrines, by sex.
  • Provision of materials (e.g. wood, bricks, stones); skilled and unskilled labour; financial support; advice on appropriate location and technologies of facilities (e.g. pit and pour, pour flush latrines) and environmental/agricultural land-use conflicts.
  • Percentage (or number) of women involved in supervisory, construction, data collection, and maintenance and operation activities.
  • Representation of women on water management and village development committees.
  • The percentage of local leaders who are women.
  • Percentage of women attending training sessions.
  • Women's involvement in non-traditional activities such as collection of construction materials, digging and filling trenches, public relations (fund raising, advertising) and local leadership, attendance at training.
  • Role of watersheds and inter-relationships among watersheds, water supply, latrines, and health and hygiene; emphasis placed on women and children.
  • Proper use and cleaning of latrines, house cleaning, and personal hygiene.
Output indicators
  • Facilities (micro-watersheds, water systems, latrines) completed.
  • Continuing levels of community and women's participation in local organizations.
Outcome indicators
  • As the project was only just completed at the time of writing this Guide, outcome indicators have not yet been covered by the CEA.
The evaluation of the longer-term impacts could be measured in the following ways:
  1. Quantitative indicators should be used to determine the extent to which the incidence of water-related diseases has decreased. What was the distribution across communities and by sex, age and ethnicity, and if there are differences, what are the reasons?
  2. Qualitative indicators should be used to determine the degree of satisfaction of women and their households with their new roles and with new services provided. What changes did this bring about in their households and in the community related to the status of women? Is the role of women in local committees sustainable, or a temporary phenomenon? Did the project create a community sense of ownership in the water and sanitation facilities?
  3. Indicators could be set for unintended effects of the project: e.g.
    1. increased leisure time for women and children, which could be translated into more education or income generation.
    2. improvement in soil quality as soil erosion is reduced by the protected watersheds.
    3. increased productivity in agriculture, and hence household incomes.


Annex 9 - Indicators of training in agriculture. Indicator use in a sample project from Bangladesh


Background

The overall project goal of the Crop Diversification Project (CDP), a gender integrated project, was: to improve the dietary diversity and level of food in Bangladesh through increasing the production and consumption of pulses, tubers and oilseeds. CIDA's contribution was approximately $30 million between 1991 and 1995.

A general overview of the project objectives is given in the Logical Framework. The Canadian component of the project addressed research, extension, market development, and institutional support; a WID component was also included. The project mid-term review made the point that the development of indicators to measure the performance of CDP was poor (CIDA 1993: 89), and a commitment was made on the part of the CEA to develop relevant indicators. The mid-term review also noted that (CIDA 1993: vii): 'aspects such as....qualitative aspects of the crop demonstrations; clear identification of farmer type and target groups; farmer adoption response; and the real impact that CDP is having on female staff in DAE [Department of Agriculture Extension] and within the farm family, are all areas that still need considerable clarification.' (ibid.: 3).

The project was chosen for discussion in this Guide as it is representative of a CIDA agricultural project in Asia Branch, and is also representative of the present state of indicator use in Asia Branch.

WID/GE in CDP

WID policies were not initially integrated in the project design, but the Inception Report developed a framework for the integration of WID into the various components of CDP. The mid-term evaluation of the project notes that the WID Implementation Plan was never formalized (CIDA 1993), and this was partly because of institutional and political constraints. Despite this, the project has achieved some success, particularly in the areas of gender awareness training, and training of women farmers.

Use of indicators in relation to WID/GE in CDP

Indicators as such are not suggested for the measurement of WID/GE. The following two points set out some of the problems with the lack of indicators, after which the potential for indicator use is outlined.
  1. The objectives, approach and outputs given in the Workplan are for the most part quite broad. No numerical indicators are set out for measuring the effects of the project on women, a point common to many other projects. A similar point is made by the mid-term review, which notes for the project as a whole: 'The expected outputs, as presented in the Inception Report are not specifically quantified, but are formulated in general terms such as.... "greater access for women to extension services" .....which have limited value as a management tool, or as yardsticks against which project progress can be measured '(ibid.: 3). However, in its recommendations for remaining inputs and outputs the mid-term review itself only attempts to quantify one output out of seven (i.e. that 20 percent of crop demonstrations should be conducted by women by the end of the project, ibid.: 4.22).
  2. Qualitative indicators are missing. While it would be relatively easy to specify the numbers involved in or affected by the project, it would have been much more difficult to specify the effect that the project had on those involved and their opinions of this.
The attendance of training workshops is a good case in point. The project did provide sex-disaggregated details of attendance at CDP training events (ibid.: 4.12). But the effects of training on participants is not noted. As is well understood, it is possible to attend training sessions and not understand them, or understand them and not act on their recommendations. A key question to answer about training is: 'How did the training change participants' views and activities?'

How the CDP might have used gender-sensitive indicators

Clear indicators could have been developed for input, process, output and outcome. The example of extension training in the field will be taken again.
  1. Related to input, targets for the numbers of women attendants could have been set, and the qualitative aims of this input could have been clearly defined (e.g. was the intention to increase networking among women, or to change perceptions about women's role in agriculture, or to change the way women carried out certain farming related activities, or was there an attempt to combine these or other qualitative elements?).
  2. Indicators of process would have been, quantitatively, the number of women attending workshops and, qualitatively what occured at the workshops, for example whether women networked, participated or refused to participate.
  3. Indicators ofoutput andoutcome would be quantitative changes in farming (e.g. leading to higher crop yields, better nutrition, better environmental practices) and qualitative changes in women's lives (e.g. greater control over resources, more say in agricultural decisions).
The final report of the CDP training Co-ordinator acknowledges that evaluation of training was carried out mainly at the level of input, and states that resources limited the ability of the CDP to do systematic evaluation of process and outcome (CDP 1994). The report notes that measuring the impact of training is difficult, costly and time-consuming. However, if CIDA is to commit itself to results-based management it must also commit itself to greater input to evaluation.

An original setting of clear objectives concerning extension training would have made the use development of indicators much easier through the project cycle.


Annex 10 - Indicators of women's empowerment from Bangladesh


Schuler and Hashemi (1994: 74-5) in a study of empowerment and fertility in rural Bangladesh developed eight indicators of empowerment as follows:
  1. Mobility: The respondent was presented with a list of places (the market, a medical facility, the movies, outside the village) and asked if she had ever gone there. She was given one point for each place she had visited and an additional point if she had ever gone there alone. A respondent with a score of 3 or better was classified as empowered.
  2. Economic security: One point was given if the respondent owned her house or homestead land, one point for any productive asset, one point for having cash savings, and an additional point if the savings were ever used for business or money-lending. A respondent with a score of 2 or better was classified as empowered.
  3. Ability to make small purchases: One point was given for purchasing small items used daily in food preparation for the family (kerosene, cooking oil, spices), one point was given for purchasing small items for oneself (hair oil, soap, glass bangles), and one point for purchasing ice cream or sweets for the children. For each of these types of purchases one additional point was given if the purchases normally were made without asking for her husband's permission, and another additional point if the purchases were made at least in part with money earned by the respondent herself. A respondent with a score of 7 or better was considered empowered.
  4. Ability to make larger purchases: One point was given for purchasing pots and pans, two points for children's clothing, three points for saris for oneself, and four points for buying the family's daily food. an additional point was given for each category if the purchase was made, at least in part, with money earned by the respondent herself. A respondent with a score of 5 or better was considered empowered.
  5. Involvement in major decisions: One point was given for making a decision (individually or jointly with her husband) within the past few years about house repair or renovation, one point for a decision to take in a goat to raise for profit, three points for deciding to lease land, and four points for deciding to buy land, a boat, or a bicycle rickshaw. An additional point was given for each category if money earned by the respondent was used. A respondent with a score of 2 or better was classified as empowered.
  6. Relative freedom from domination and violence within the family. The respondent was asked if, within the past year, she had been beaten by her husband; money had been taken from her against her will; land, jewelery, or livestock had been taken from her against her will; she had been prevented from visiting her natal home; or she had been prevented from working outside the home. A respondent was classified as empowered if she said that none of the things had happened to her.
  7. Political and legal awareness: One point each was given for knowing the name of a local government official, a member of parliament, or the prime minister, and one point each for knowing the law governing inheritance, and one point for ever having campaigned for a political candidate. A respondent was classified as empowered if she had a score of 4 or better.
  8. Participation in public protests and political campaigning: The respondent was classified as empowered if she had campaigned for a political candidate or had come together with others to protest a man beating his wife, a man divorcing or abandoning his wife, unfair wages, unfair prices, misappropriation of relief goods, or "high-handedness" of police or government officials.
To combine the eight indicators into a single score, a woman was classified as empowered if she had a positive score on five or more of the eight indicators.

This is an innovative approach that could be adapted to a variety of empowerment projects. Like any such system, the scoring system is arbitrary, but it is internally consistent and relevant to women's lives in Bangladesh.


Annex 11 - Sample project on participation


The following Table gives two examples of (constructed) projects on empowerment/participation. The examples illustrate how indicators can be used for Type 1 and 2 objective statements, and how indicators can be used through the project cycle. Base-line studies would need to be carried out for the two projects to examine the benchmarks (e.g. number of women political representatives at the start of the project) against which success could be measured.

Sample project on participation

The following Table gives two examples of (constructed) projects on empowerment/participation. The examples illustrate how indicators can be used for Type 1 and 2 objective statements, and how indicators can be used through the project cycle. Base-line studies would need to be carried out for the two projects to examine the benchmarks (e.g. number of women political representatives at the start of the project) against which success could be measured



Objective Risk/enabling
indicators
Input
indicators
Process
indicators
Output
indicators
Outcome
indicators
a. Improve the status of women in project area through greater access of women to political power, over a five year period (type 2 objective)
  • Local population acceptance
  • Government acceptance
  • Women willing to stand for power
  • Women's political representation will lead to an improvement in the status of women
  • Leadership training for women of all socio-economic classes
  • Education programs for women of all socio-economic classes
  • Funds devoted to project
  • Number of women trained
  • Number of women attending education programs
  • Women's views on training
  • Short-term effects of training (women able to take greater role in decision-making locally, public speaking, increased number of women involved in politics)
  • Increase in political representation of women by 200 percent at end of 5 year period
  • Improvement in women's status (literacy, employment and health) as a result of greater political participation of women (as a direct result of certain legislation being passed because of greater involvement of women in politics)
  • 50% less violence against women
  • Women's views on changes in status
b. Increase in political representation of women in 300 rural councils in project area by 200 percent over ten year period

(type 1 objective)
  • Local population acceptance of women politicians
  • Government acceptance
  • Women willing to stand for power
  • Stable political situation in country
  • Political system will remain pluralist
  • Local community views of feasibility of the project
  • Funds devoted to project
  • Increase in political representation of women by 100 percent over five year period
  • Degrees of training, education, etc. provided for women
  • Women and men's attitudes to training and change, by socio-economic group
  • Increase in political representation of women by 200 percent at end of ten year period
  • Views of women and men of likely effects of increase
  • Number of key decision-making positions held by women and men
  • Increase in political representation of women by 200 percent, maintained for one election after end of donor involvement in project
  • Socio-economic background of women political representatives

In the first example, the objectives are relatively broad and the indicators have to 'interpret' what the objectives mean. Because the objectives are broad a large number of indicators could have been used to measure whether the objectives have been met, and those given in the Table are sample indicators.

In the case of the second example, the objectives are relatively narrow and well defined and indicators are easy to set. However, the outcome indicators in this case will not tell us very much about empowerment of women. To learn about this we would need to look at factors such as:
  • how many women were in decision-making positions and the effect of this, and
  • the effect of women gaining political power on the status of women.
To do this it would be necessary to use qualitative analysis, which would examine the links between women's political representation and social and political change.



1 The Guide has been produced by Tony Beck and Mortong Stelcner. Dr. Beck is a development consultant and a Research Associate at the Institute of Asian Research, University of British Columbia. Dr. Stelcner is a full Professor in the Economics Department, Concordia University.

2 There have been a number of studies recently related to measuring capacity development and institutional strengthening. See for example CIDA 1995c; IDRC 1994; International Institute for Sustainable Development 1995; and Munansinghe and Shearer 1995.

3 See Annex 1 for the difference between an indicator and a statistic.

4 There has been extensive discussion and experimentation of quantitative methods for the analysis of qualitative indicators, but this has not spread in general to development studies.

5 See Annex I for more detailed definitions. The example is adapted from Carvalho and White (1994).

6 Another alternative would be to determine if a project was successful on its own terms, and then count the number of successful projects in each Branch.

7An illustrative listing of R/C level objectives is given in Annex 3. See foot note 2 for other references to work on indicators of sustainability in institutions.

8Projects were selected to be broadly representative of CIDA's WID/GE work at the project level. The original intention was to take a representative project from each of the geographical branches and one project each from Partnership and Multilateral Branches. However, the latter two Branches were unable to locate suitable projects. The authors reviewed details from corporate memory of WID-specific and gender integrated projects over $1 million for the geographical branches. The following criteria were set for the selection of projects: a) representative of CIDA's work as a whole, and of the work of that Branch; b) had a large WID/GE component; c) had a strong monitoring and evaluation component.

The following comments on projects do not involve a criticism of either CIDA or CEAs. As noted above work on indicators is complex and projects that were running before the current focus in CIDA on indicators should not have been expected to have made significant advances in the use of indicators.

9 E.g. on the environment Bromley 1992, Ostrom 1990, Berkes 1989; and from the water sector Hunt 1990; Uphoff 1990; Vermillion 1990; Groenfeldt 1988.
10 The following list is taken from CIDA 1995a.

11 Improvements in women's health and education are key features of social empowerment, and are considered in Section 5.2. Social indicators of empowerment overlap considerably with indicators of participation.

12 Because of the biases against particularly poor rural women (who are in may cases in the most need of "empowerment"), little is known about women's indicators of empowerment. Some preliminary work has been done on this topic in Asia (Beck 1994a; Vlassof 1994), Latin America (Radcliffe and Westwood 1993) and Africa (Mbilinyi 1988). This work points to two areas through which women themselves feel empowered - greater economic stability (for themselves and their households), and greater self-respect (defined in various ways).

13 Useful sources for PRA material are:
The PRA/PALM Series, available from MYRADA, 2 Service Road, Domlur Layout, Bangalore 560 071, India. RRA Notes, available from International Institute for Environment and Development, 3 Endsleigh Street, London WC1H ODD, U.K.

14 Annex A7.1 is an abridged version of Annex 1 in CIDA (1995a). See this document for further details


Alternate Format

Note: If you cannot access the documents that are provided in an alternate format, refer to the Help page.

Guide to Gender Sensitive Indicators (91 pages, 662 KB)