Review of the Afghanistan Program - Review Objectives, Approach and Limitations

Table of Contents

• 3.0 Review Objectives, Approach and Limitations
  • 3.1 Objectives of the Review
  • 3.2 Review Approach
  • 3.3 Coverage and Sample of projects selected for Review
  • 3.4 Review Criteria
  • 3.5 Limitations of the Review

3. Review Objectives, Approach and Limitations

3.1 Objectives of the Review

The objectives of the Review are to:

1. assess performance in terms of progress made towards the achievement of the stated results of the Afghanistan Program; determine if the results are contributing to the development of a legitimate and stable state, which is the aim of CIDA's support to Afghanistan;
2. assess the adequacy of the accountability framework to ensure that the funds are properly managed and spent for intended purposes; and
3. provide lessons and recommendations to improve the Program.

3.2 Review Approach

The information for the evaluation was derived from both primary (interviews) and secondary sources (documents). Documents used include existing evaluation, audit and monitoring reports already produced by CIDA, CIDA partners (e.g. World Bank, UN organizations, Donors, GoA, etc), and other independent international research institutions, after assessing their reliability and credibility. The credibility of evaluations, for example, was assessed on the basis of usual quality criteria such as balanced reporting, methodological rigour, valid and reliable data, use of multiple lines of evidence, and conclusions supported by the evidence.

Interviews were conducted with a wide range of relevant persons: a) existing and former CIDA HQ and the Field staff; b) staff from other Government departments in Canada (DND and DFAIT); c) CIDA external monitoring team and evaluation consultants; and d) officials from the GoA.

3.3 Coverage and Sample of projects selected for Review

The Review period encompasses fiscal years (FY) 2004-05, 2005-06 and 10 months of FY 2006-07 i.e. until the end of January 2007. It should be noted that the Review, initially planned for fiscal year 2005-06, was postponed to fiscal year 2006-07 for two reasons: firstly, most of the initiatives were too recent to assess development results at the outcome level; and secondly, the security constraints were a major issue at that time. The Agency Audit Committee also approved a multi-year audit plan to 2007 to 2010 in April 2007. This plan includes an examination of the Afghanistan Program, encompassing disbursements on the Kandahar component.

With regard to this Review, the selection of projects (see Table 1) examined in depth in the Review was guided by considerations such as:

• coverage of a significant portion of money spent; and
• reflection of the variety and complexity of challenges: different delivery mechanisms, size of investments, and sectoral /thematic areas of focus.

Twelve (12) of the 27 projects in the Afghanistan Program nation-wide were selected for in depth examination (see Table 1). The remaining projects have only been cursorily examined by the Review. CIDA's investments in Kandahar, most of which started in 2006, were not covered.

1. The 12 projects account for 83 percent in terms of the Program budget and represent $205.6 million or 73 percent of CIDA's total disbursements for the nation wide Program in Afghanistan as of January 2007 (see Table 1). These projects are: Afghanistan Reconstruction Fund (ARTF) Operating Budget Support;
2. National Solidarity Program (NSP);
3. Microfinance Investment Support Facility for Afghanistan (MISFA);
4. National Area Based Development Program (NABDP);
5. Mine Action Program for Afghanistan (MAPA);
6. Support Parliamentary Elections, and Provincial Councils;
7. Support to the Establishment of the Afghanistan Legislature (SEAL);
8. Strengthening the Rule of law in Afghanistan (SRLA);
9. Disarmament, Demobilization and Reintegration (DDR);
10. Afghanistan Stabilization Program (ASP);
11. Support to the Development of Free Media in Afghanistan; and
12. Western Basin Water Resources.

These initiatives were chosen from the primary list shown in Table 1. They cover the four components of the Afghanistan Program i.e.: Operating Budget Support for the GoA within the ARTF initiative; Rural Livelihoods and Social Protection; Security and Rule of Law, and Natural Resources.

In addition to the above, the cross cutting theme ''Equality between Women and Men" was also looked at.

3.4 Review Criteria

Review criteria were established to assess a) the adequacy of reporting with regard to the financial and project results (e.g. pertinence, credibility, independence and follow-up action on recommendations); and b) the adequacy of the accountability regime in place to ensure that funds were used for intended purposes and that intended results of the Program have been achieved (see Annex 6).

3.5 Limitations of the Review

The Review team did not proceed to a field visit due to the fact that there were numerous independent evaluations, audits and monitoring mission reports from credible sources, which were already available to the team for their exercise. In addition, given the security situation in Afghanistan coupled with significant restrictions on movement of personnel, it is doubtful whether a field visit would provide additional information or value. It should be noted that the Review Team discussed this limitation with Treasury Board Secretariat representatives and it was agreed to do a desk review coupled with interviews with a range of relevant persons knowledgeable of the CIDA initiatives in Afghanistan. At the planning stage of our Review, it was also decided to limit our coverage and exclude the initiatives related to Kandahar since these projects had only started in 2006 and it was too early to assess results.

Table 1 Projects Sampled for in depth Analysis in the Review (see projects shaded)

Project No	Project Title	Funding Channel or Partner	Total disbursement last 3 years	Relative % of total investment
Sub-Total			252.29	N/A
Other small projects < $1million			32.71	11.48%
Total for all Projects			285.00	100.00%
Total value and percentage of the sample			$205.70	72.18%
Budgetary Support
A-032445	ARTF Operating Budget Support	World Bank	60.00	21.05%
Rural Livelihoods and Social Protection
A-32660/32209	National Solidarity Program 2005/09	World Bank	30.00	10.53%
A-032234	Microfinance Investment Support Facility for Afghanistan (MISFA)	World Bank	37.00	12.98%
A-32609/32002	National Area Based Develop. (NABDP)	UNDP	15.10	5.30%
A-032440	Afgha. Community Renewal Program (ACRP)	AKF-C	1.75	0.61%
A-32630	Integr. Alternative Livelihoods Prog. Kan. (IALP)	UNDP/MRRD	5.00	1.75%
A-32194	Women's Rights Fund (WRFA)	R & D	1.52	0.53%
A-32600	Refugee Return & Integration	HCR	5.00	1.75%
Natural Resources
A-32170	ADB Hari Rud River Basin Study	ADB	1.00	0.35%
A-033143	Western Basins Water Resources	ADB	10.00	3.51%
Security and Rule of Law
A-32446	Afghanistan Stabilization Program (ASP)	World Bank	12.00	4.21%
A-032168	UN Mine Action National Dev. Budget	UNMAS	15.00	5.26%
A-32597	Disarmament/Demobilization/ Reintegration (DDR)	UNDP/UNAMA	6.50	2.28%
A-32671	Ammunition Survey & Destruction (ASD)	UNDP/UNAMA	1.20	0.42%
A-32431	Heavy Weapons Cantonment (HWC)	UNDP/UNAMA	1.20	0.42%
A-032956	Disarming Illegal Armed Groups (DIAG)	UNDP	4.00	1.40%
A-032995	A/P Mine & Ammo Stockpile Destruction	UNDP	4.00	1.40%
A-32169	Law & Order Trust Fund (LOTFA)	UNDP	2.00	0.70%
A-32484	Strengthening rule of Law, AF	IDLO	4.30	1.51%
A-32531	Making Budgets Work	UNDP	3.00	1.05%
A-32806	Support to the Estab. Afghan Legislature (SEAL)	UNDP	1.20	0.42%
A-032503	Counter-Narcotics Capacity Building	UNODC	1.00	0.35%
A-32165	Election Support - Presidential	UNDP	8.00	2.81%
A-32623	Parliamentary Elections Support	UNDP/JEMB	13.00	4.56%
A-32189	Media Support	IMPACS	1.60	0.56%
Multilateral Branch
M-011660	Kabul Widows Humanitarian Project	CARE Int.	5.70	2.00%
Common to all sectors managed by Asia Branch
A-31648	Program Support Unit - Afghanistan	CIDA/PSU	2.22	0.78%