Procedural Effectiveness of International Humanitarian Assistance

Procedural effectiveness relates to the impact assessment; so, the objective of this study is to identify the procedural effectiveness indicators and develop a conceptual mapping of existing impact indicators in the field of International Humanitarian Assistance (IHA). We conducted a literature review and searched the websites of humanitarian assistance organizations. Documents were included if they focused on IHA procedural effectiveness or impact evaluations. Reviewers identified the eligible studies and extracted data. A conceptual framework was used to categorize the indicators of secured (protection, good health and food security) and avoided (illness, death and malnutrition) issues. 10 documents were included and categorized based on the framework. It was found that there is no agreed and unique methodology and indicators to assess the impact of humanitarian assistance projects. This study could be the step toward understanding of IHA procedural effectiveness indicators and also the findings can make a base line to start more research.


Introduction
Office for the Coordination of Humanitarian Affairs (OCHA) defines the humanitarian assistance (HA) as: "Aid that seeks, to save lives and alleviate suffering of a crisis-affected population" (OCHA, 2013).The aim of HA is to protect life of the affected people during and after disasters.International Humanitarian Assistance (IHA) will be provided when the governments of the affected countries are unable to provide aid in an effective way.The assistance is provided in emergency situations to save and protect human life in both natural and man-made disasters (CIDA, 1996).This assistance should be allocated in an effective way in order to maximize its impact.
Effectiveness is a term which seems to have many different definitions.Etzioni in 1964 was one of the first researchers who defined the goal attainment approach of effectiveness (Etzioni, 1964;Spar & Dail, 2002).Georgopoulos and Tannenbaum in 1957 and Yuchtman and Seashore in 1967 emphasized a financial approach.(Georgopoulos & Tannenbaum, 1957;Yuchtman & Seashore, 1967).The approach of reputation was evolved from Georgopoulos and Mann's project in the hospital system (Georgopoulos & Mann, 1962).This approach relies on subjective performance measures reported by informants or organizational stakeholders (Jobson & Schneck, 1982).After that the literature focused on the definition of more complex methods of effectiveness, such as multi-dimension models (Cameron & Whetten, 1983;Foster & Lock, 1990;Zammuto, 1984), competing values models (Quinn & Rohrbaugh, 1983), contingency models (Lewin & Minton, 1986), and balanced scorecard (Kaplan & Norton, 1996) approaches.These methods tried to unify the aspects of goal attainment, resource control and reputation approaches (Herman & Renz, 1999).
Substantive effectiveness relates to the goal achievement (Sadler, 1996).When the practice is completed, substantive effectiveness is used to answer if the objectives are obtained or not (Baker & McLelland, 2003).In situations when there are some changes in the policy, plan or program, substantive effectiveness could be assessed (Theophilou et al., 2010).
Transactive effectiveness considers the achievement of outcomes by focusing on the cost and time (Sadler, 1996).Using fewer resources to achieve the objectives or outcomes could be assessed by the transactive effectiveness (Baker & McLelland, 2003).Considering the skills and the roles of practitioners in addition to the time and cost to define the transactive effectiveness was mentioned by Theophilou (2010).
Normative effectiveness considers the achievement of normative objectives (Baker & McLelland, 2003).In situations when there are some changes in the organizations, mission and culture, the decision making could be affected (Cashmore et al., 2004).Normative changes could be observed based on the perceptions of those who were involved as stakeholders to the process or in the implementation of the tool.
Procedural effectiveness relates to the impact assessment (Sadler, 1996).It should also reflect what and how the procedures of the projects are implemented (Baker & McLelland, 2003).Bina (2007) added that the methodological approach could affect procedural effectiveness.
Measuring the procedural effectiveness involves the need first to define the impact.The definition of impact used by Oxfam is: "Significant or lasting change brought about by a given action or series of actions" (Roche, 1999).The most commonly-used definition of impact is provided by the OECD/DAC: "The positive and negative, primary and secondary, long-term effects produced by a development intervention, directly or indirectly, intended or unintended" (OECD/DAC, 2002).
Measuring impact in emergency situations and contexts is complicated.Many international humanitarian organizations have an approach to assessing the impact of their work.They have their evaluation guidelines to assess the impact of humanitarian aids (ALNAP 2001, 2005, World Food Program 2008, The Sphere Project 2004).In spite of these commitments to assessing the impact of humanitarian assistance, the lack of effective impact assessment is more pronounced.
The objective of this study is to present the conceptual mapping of indicators which are used for measuring the procedural effectiveness and the impact assessment of international humanitarian assistance.

Methods
Procedural effectiveness relates to the impact assessment (Sadler, 1996).So, literature review was used for summarizing current thinking on measuring the procedural effectiveness.This review presents a summary of the literature on impact assessment indicators, focusing on impact assessment of international humanitarian assistance.This study is conducted in two phases.First, the impact indicators of humanitarian assistance were searched from humanitarian organizations' websites.Second, a conceptual framework was used to categorize the impact indicators.

Search Strategy
The search keywords were selected after consulting the experts.Databases were searched with the following keywords: humanitarian* AND assistance* OR relief* OR aid* AND effectiveness*.This search was conducted following organizational reports and documents: Oxfam, OCHA, Save the Children, Swedish International Development Cooperation Agency (SIDA), Canadian International Development Agency (CIDA), Active Learning Network for Accountability and Performance (ALNAP), Department for International Development (DFID) and Humanitarian Practice Network (HPN) and Google as a general search engine.

Inclusion Criteria
There are different definitions for effectiveness.But we focused on the procedural effectiveness which relates to the impact assessment in this manuscript.So, we include the studies if they were focused on the impact indicators.
Eligible studies included English-language articles with no limitation in time of studies.No other filters were applied.Articles were included if they focused on international humanitarian assistance and procedural effectiveness or impact indicators.Both quantitative and qualitative studies were included.Also, organizational reports and documents were included.

Exclusion Criteria
Non English studies, the irrelevant studies to IHA effectiveness and articles with similar keywords but covering unrelated topics were excluded.

Selection of Studies
After excluding duplicated studies, references found through the search were evaluated for inclusion criteria by reading their titles and abstracts.After an initial screening of abstracts of those that remained, the first author (SM) read and analyzed their full texts to determine their eligibility.This was done under the supervision of second author (AA).The first author had passed IHA course at the Harvard Humanitarian Initiative.Quality assessment was performed using study design appropriate Critical Appraisal Skills Program (CASP) tools.Disagreements were resolved by consensus or by consulting other authors.
Source: Hofmann, C.A., Roberts, L., Shoham, J. & Harvey, P. (2004).Measuring the impact of humanitarian aid: A review of current practice, The Humanitarian Policy Group at the Overseas Development Institute, p.9.
Figure 1.Framework and dimensions that need to be considered for evaluating the impact of humanitarian assistance

Data Extraction and Analysis
We extracted the following data from each study report: organization; indicator; and the reference.Data extraction forms were designed using Microsoft Excel.Extracted data were presented in a descriptive table.

2-1-Conceptual framework:
Charles-Antoine Hofmann, Les Roberts, Jeremy Shoham and Paul Harvey ( 2004) defined a conceptual framework for evaluating the impact of humanitarian assistance.Figure 1 shows the dimensions that need to be considered for evaluating the impact of humanitarian assistance.This conceptual framework shows the complexity of the humanitarian aid system, and the impact on population.The impact on population has been divided to two categories of secured (protection, good health and food security) and avoided (illness, death and malnutrition) issues.

Results
Evaluation process often tends to focus on outputs and outcomes rather than impacts.10 studies were included in this manuscript.Table 1 shows the indicators suggested by various studies on impact assessment of international humanitarian assistance in two categories of secured protection, good health and food security and avoid illness, death and malnutrition.There are several indicators (n=22) categorized in Secured category while there are 3 main indicators of mortality, morbidity and malnutrition categorized in Avoid category.60% of documents were focused on Secured indicators and 40% of documents were focused on the Avoid indicators.All the indicators, directly or indirectly, were related to the health subjects.

Discussion
Focusing on impact indicators and evaluation methodologies is needed for the measurement of the procedural effectiveness of international humanitarian assistance.How the impact of humanitarian assistance projects can best be measured, is a technical question that should be answered before starting to assess the impact of such projects.
We found several indicators for assessing the procedural effectiveness.The most interesting result was that all the indicators, which were related to the impact if IHA, were related to the health issues.
There are a range of methodologies to assess the impact indicators which were found in the result section.Impact assessment of humanitarian assistance in an emergency situation is a relatively young area of research (Oakley, et al., 1998).Lack of standardized tools and rigorous methodologies for impact assessment is a global concern (Hofmann et al., 2004;Global Center for Development, 2006).
For assessing the procedural effectiveness we need to use an appropriate methodology.There are different methodologies which were used by different organizations to assess the impact of humanitarian assistance.Randomized Controlled Trial (RCT) and randomized impact evaluation was used in Liberia, Colombia, Chad and some other African countries to conduct 'before' and 'after' household surveys.International Rescue Committee (IRC) used this methodology to avoid selection bias, which reduced the chances of evaluators attributing differences between program and non-program communities to the interventions.WFP & IFPRI conducted the RCT study in Colombia.Food consumption, social capital and anemia were the indicators of their study (Watson, 2008;Puri, 2015).
Participatory impact monitoring and Participatory Impact Assessment (PIA) are other methodologies used in Niger and Liberia to assess the impact.In these studies PIA focused on the changes in household food security and income resulting from the project's re-stocking.Semi-structured household interviews, focus-group discussions, 'before and after' ranking and scoring were used to assess the impact (Puri, 2015).
Recipient perceptions review is another methodology used by FAO andFritz Institute (Fritz Institute, 2004-2006).They used qualitative (story telling) and quantitative (questionnaires) methods to assess the impact of humanitarian assistance.UNHCR & WFP (2012) used quasi experimental studies in Bangladesh and used food security, refugee movement, global acute malnutrition, economic activity and earnings, copying strategy index, household dietary diversity score and protection as the study indicators (Watson, 2008;Puri, 2015).
Despite existing different methodologies (Watson, 2008;Puri, 2015), this paper has shown that there is no best method to assess the impact of humanitarian assistance.Also, lack of epidemiological skills of NGOs' staff is an obstacle in choosing an appropriate methodology for assessing the impact of humanitarian assistance (Roberts & Hofmann, 2004).
Identifying and using appropriate indicators is an important part of assessing the impact of an intervention.Most of humanitarian organizations tend to assess process/output indicators, rather than impact indicators (Roberts, 2004;Shoham, 2004).Also, the Sphere project focused on process indicators and most of the indicators are not defined to assess the impact of humanitarian intervention.One reason may be that the collection of impact indicators is sometimes seen to be more difficult than process/output indicators (Hofmann, 2004).
While ALNAP annual review, coping strategies index (emergency nutrition network), impact indicators of Tsunami Recovery, Assessment and Monitoring System (TRIAMS), Roche and Feinstein International Center define impact indicators in 'secured protection, good health and food security' category, other studies focused on 'Avoid Illness, Death, Malnutrition' category.These indicators could be analyzed at different levels from the individual project, at the level of a sector, at a country level or at the level of an organization and its global impact (Hofmann, 2004).
Adequately trained, experienced, and motivated staff should be hired to design and evaluate the impact of humanitarian assistance projects.Also, surveillance systems should be used to monitor the impact indicators.Surveillance is the systematic collection of information over time.Lack of systematic monitoring and surveillance in the humanitarian assistance projects is a serious obstacle to assessing the impact of humanitarian aid (Roberts & Hofmann, 2004).

Conclusion
As recognized at the Consultation Workshop for Humanitarian Effectiveness in 2013, identifying ways of defining, measuring and improving the effectiveness of humanitarian assistance was one of the thematic areas of focus for the World Humanitarian Summit (WHS) (OCHA, 21 March 2013).This study has reviewed current knowledge about the procedural effectiveness assessment of humanitarian assistance.The findings of this study indicate that there is no agreed and unique methodology and indicators to assess the impact of humanitarian assistance projects, although there is a global need for evaluating the effectiveness of international humanitarian assistance projects.Also we found that all the impact indicators of IHA were related to the health issues.We developed a conceptual mapping of existing indicators through the conceptual framework.We believe that the findings of this study could help the researches to use an appropriate methodology and indicators to assess the procedural effectiveness of international humanitarian assistance.

Limitation
The findings might not be representative of all IHA organizations, since we had analyzed the documents of major IHA organizations.

Table 1 .
Indicators for measuring the impact of international humanitarian assistance