Evaluation Consultant (International)
Introduction
Established in 1951, IOM is a Related Organization of the United Nations, and as the leading UN agency in the field of migration, works closely with governmental, intergovernmental and non-governmental partners. IOM is dedicated to promoting humane and orderly migration for the benefit of all. It does so by providing services and advice to governments and migrants.
Project Context and Scope
The project “Improving the Control Capacity of DG Customs Enforcement”, funded by the European Union (EU) and implemented by IOM Türkiye in cooperation with the Ministry of Interior, the Department of European Union Affairs and Foreign Relations, aims to improve the surveillance and control capacity of the customs in the movement of goods in the country. The main beneficiary of the project is the Ministry of Trade, Directorate General for Customs Enforcement (DGCE). The DGCE is the first national body at the borders tasked with checking personal belongings, goods, and vehicles entering and exiting the country. Therefore, the DGCE plays a critical role in providing the first response for the detection of nuclear and radiological materials on persons, in goods and in vehicles at borders. An international Evaluation Consultant will be engaged for the conduction of the final evaluation.
Commissioned by: Monitoring, Evaluation, Accountability, and Learning (MEAL) Unit, IOM Türkiye
Managed by: Senior National MEAL Officer
Responsibilities
Evaluation Context
The International Organization for Migration (IOM), the United Nations (UN) Migration Agency, was established in 1951 and is the leading intergovernmental organization in the field of migration, working closely with governmental, intergovernmental, and non-governmental partners. With 175 member states, 8 states holding observer status, and offices in 172 countries, IOM is dedicated to promoting humane and orderly migration for the benefit of all.
The IOM established its operations in Türkiye in 1991. IOM’s partnership with the Government of Türkiye (GoT) was formalized in November 2004, when Türkiye became an IOM Member State. IOM Türkiye works closely with the government of the Republic of Türkiye, regional authorities, the UN, donors, and civil society organizations to address migration challenges by implementing programmes through three pillars: Resilience, Mobility, and Governance.
Across the country, IOM Türkiye provides a comprehensive response to the humanitarian needs of migrants, internally displaced persons, returnees, and host communities through direct humanitarian assistance, recreational activities, and various other efforts. Alongside IOM’s role in addressing the needs of migrants during crises, the mission works in close collaboration with the Government of Türkiye to address the longer-term impact of migration, including migrant assistance programmes, labour integration and migration management, immigration and border management, and research and data collection on migrant movement.
PROJECT INFORMATION
Title of the Project:
Improving the Control Capacity of DG Customs EnforcementProject ID:IB0439Implementing Agency:
International Organization for Migration (IOM)
UNSDCF 2021-2025 Outcome:
Outcome 4.2: By 2025, the effectiveness of the international protection and migration management system is improved.
IOM Strategic Plan 2024-2028 Objective and Long-Term Outcome:
Objective 3: Facilitating pathways for regular migration.
Long-Term Outcome 3b: Migration flows and cross-border mobility are well managed, with measures to ensure well-being, including health, security and safety throughout the mobility continuum.
IOM Türkiye Mission Strategy 2021-2025 Priority:Strategic Priority 4 - Governance: Strengthen cooperative development and implementation of evidence-based and inclusive migration governance that addresses migration challenges, leverages opportunities and facilitates safe, orderly and regular migration.Beneficiary (Partner) Institutions:
Lead Institution: Ministry of Interior (MoI), Department of European Union Affairs and Foreign Relations (DIAB)
Beneficiary Institution: Ministry of Trade (MoT), Directorate General for Customs Enforcement (DGCE)
Location of the Project: Border Crossing Points in the Republic of Türkiye (land, sea, and air)Start and End Date of the Project:1 May 2024 – 30 April 2026Project Extensions (if any):N/ATotal Planned Project Budget:
Planned Budget: EUR 1,000,000
Funding Source/Donor:European Union (EU)Overall Objective of the Project:Overall Objective (Impact): To improve the surveillance and control capacity of the customs in the movement of goods in the country.Target Groups:
Primary:
• Ministry of Trade (MoT)
Secondary:
• Ministry of Interior (Turkish National Police)
• Ministry of Energy and Natural Resources (MoENR)
• Turkish Nuclear Regulatory Authority (NDK)
• Turkish Energy, Nuclear & Mining Research Institution (TENMAK)
• Turkish Postal Services (International Mail & Cargo Processing Center )
• Express Courier Companies
• Private Sector Entities
Final Beneficiaries:
Travellers crossing the border and local communitiesEstimated Results (Outcomes and Outputs):
Specific Objective (Outcome): To strengthen the customs control capacity in the areas of nuclear security and smuggling via parcel carriers.
Output 1: Radiation control capacity of DG for Customs Enforcement is improved.
Output 2: Mail and Express Courier Control Capacity is improved.
Evaluation Coverage:The thematic scope of the evaluation covers project activities implemented at DGCE Headquarters, selected border points, mail processing centers, training sites, and study visit beneficiary groups. It includes all technical, operational, capacity-building, and coordination components. The final evaluation aims to comprehensively assess the project’s performance during the period between 1 May 2024 and 30 April 2026.
Evaluation Purpose, Objectives and Scope
This evaluation is being conducted in accordance with IOM’s Evaluation Policy and Guidelines which sets out a number of guiding principles and key norms for evaluation in the organization following the Norms and Standards of the United Nations Evaluation Group (UNEG).
The final evaluation aims to comprehensively assess and demonstrate progress achieved towards the expected results against those stated in the Project Documents and identify the lessons learned and recommendations relevant to the planning, preparation and implementation of activities under the subsequent phase. The purpose of the external final evaluation is to provide a summative assessment encompassing all six core OECD-DAC criteria: relevance, coherence, effectiveness, efficiency, sustainability, and likely impact (including early signs of results, whether intended or unintended, positive or negative) as well as cross-cutting issues.
Specific Objectives of the Final Evaluation
The specific objectives of the evaluation are to:
Assess the extent to which the project has achieved its stated objectives (outcome and impact levels) and outputs, while identifying both negative and positive factors that have facilitated or hampered progress in achieving the project outcomes, including external factors/environment, weakness in design, management, and resource allocation.
Assess the effectiveness of the implementation strategy (i.e. implementation modalities, issues of coordination and partnership arrangements, and synergy among similar projects as well as with other initiatives/programmes of IOM Türkiye).
Assess the management and financial efficiency of the project.
Assess the extent to which the project's outcomes will be sustainable (without the need for external support) and contribute towards the objective of the project.
Generate substantive evidence-based knowledge by identifying best practices and lessons learned that could be useful for remaining project implementation, other interventions at national (scale up) and international level (replicability).
Provide a forward-looking perspective for IOM’s positioning in relation to integrated border management for subsequent phases.
Provide formative recommendations toward design and implementation arrangements of the subsequent phase.
Through a participatory approach, the Evaluation Consultant will actively engage relevant stakeholders, including IOM Türkiye staff, focal points of the national counterparts and beneficiaries from different activities. A tentative Data Collection Mission Agenda is provided under section 10 to indicate the level of effort and in-country travel required for data collection.
Recommendations, emerging from the evaluation, should be strongly linked to the findings of the evaluation and should provide clear guidance to IOM Türkiye and its stakeholders on how they can address them.
Furthermore, the evaluation will have a focus on what worked, what did not work, and why, based on feedback against evidence and provide actionable recommendations, highlight best practices, share lessons learned, and offer valuable insights, which will inform the remainder of the implementation of the project. An essential aspect of the evaluation involves analysing the integration of IOM cross-cutting themes of gender equality, non-discrimination and human rights-based approach into project activities and implementation.
Evaluation Criteria
The OECD-DAC evaluation criteria, relevance, coherence, effectiveness, efficiency, likely impact, and sustainability will guide the final evaluation. All assessments will be conducted in accordance with the OECD-DAC definitions to ensure methodological rigour and consistency with international standards.
• Relevance: The extent to which the intervention's objectives and design respond to the needs, policies, and priorities of beneficiaries at the global, national, and institutional levels, and continue to do so as circumstances evolve.
• Coherence: The extent to which the intervention is compatible with other interventions within the same country, sector, or institution.
• Effectiveness: The extent to which the intervention has achieved, or is expected to achieve, its stated objectives and results, including any differential results across groups.
• Efficiency: The extent to which the intervention delivers, or is likely to deliver, results in a cost-effective and timely manner.
• (Likely) Impact: The extent to which the intervention has produced or is expected to produce significant positive or negative, intended or unintended, higher-level effects.
• Sustainability: The extent to which the benefits of the intervention continue or are likely to continue.
Evaluation Questions
In the light of the evaluation parameters, the Evaluation Consultant is expected to analyze data and share their findings, conclusions and recommendations generated by this analysis. As a reference point for the evaluation, the Evaluation Consultant is provided with indicative key and sub-evaluation questions below; which are expected to be amended, elaborated, aggregated, consolidated and submitted as part of the Inception Report and shall be included as an annex to the final evaluation report described below. In consultation with the Commissioning Unit, in the inception phase, the Evaluation Consultant will further develop and refine the evaluation questions to ensure detailed and specific information is gathered for each criterion.
The evaluation matrix will be reviewed collaboratively with the Commissioning Unit. Any fundamental changes to the evaluation criteria and questions should be agreed upon between the evaluator(s) and the Commissioning Unit and reflected in the inception report.
The sub-questions included in the ToR are intentionally designed to guide prospective evaluation team by outlining the expected scope of the assessment, key thematic areas to be explored, and the breadth of analysis anticipated. They are meant to provide a comprehensive picture of what the evaluation should cover and to support evaluators in developing a robust evaluation matrix. Further refinement and consolidation of evaluation questions is an essential step during the inception phase. The evaluation evaluation team will be expected to prioritize, aggregate, and streamline these questions to ensure analytical depth, coherence, and a focused final report.
The following key questions will guide the evaluation process:
Evaluation CriteriaEvaluation Questions
Relevance
Key evaluation question-1: To what extent was the project’s design, theory of change, and interventions appropriate and aligned with the needs and priorities of target beneficiaries and stakeholders, as well as with national, regional, and international policy frameworks and IOM’s comparative advantages?
Sub-questions:
To what extent were the theory of change applied and interventions designed in the project relevant to serving the needs of the target groups, beneficiaries and stakeholders (MoT, MoI, NDK, TENMAK, IMCPC)?
To what extent were the design and strategy of the project activities aligned with the objectives set in the IOM strategies, UNSDCF 2021-2025, IOM Immigration and Border Governance (IBG) Guidance, Community Engagement & Policing (CEP) Document, Migration Governance Framework, and the EU Integrated Border Management principles?
To what extent does the project fit into the comparative advantages of IOM’s global experience and presence in the area of border management?
To what extent were the participants of project activities representative of public, civil society, academia, private sector and other relevant target groups where applicable?
What opportunities are there to better align the support/activities to the changed context and the needs of the beneficiaries for future needs?
Coherence
Key evaluation question-2: To what extent was the project internally consistent and logically designed, and how well did it complement and create synergies with other relevant interventions and institutional frameworks in border management and customs enforcement?
Sub-questions:
- (External coherence) To what extent did the project create synergy/linkages with other projects and interventions in the country i.e. other projects implemented for border security, customs enforcement and surveillance? How well does the project complement government-led border management systems?
-
(Internal coherence) To what extent are the project’s objectives, outcomes, outputs, and activities logically aligned and mutually reinforcing? How coherent is the project’s theory of change in linking project interventions (e.g. ToTs, cascaded trainings, study visits, team building, assessments) to intended border management outcomes?
Effectiveness
Key evaluation question-3: To what extent did the project achieve its intended outputs and outcomes, strengthen institutional capacities and coordination, and respond effectively to beneficiary needs while enabling adaptive implementation and learning?
Sub-questions:
To what extent has the project achieved its intended outputs and outcomes until the end of project duration? (The Evaluation Consultant is expected to provide detailed analysis of 1) planned activities and results and 2) achievement of results.)
How effective have the project’s interventions (e.g. ToTs, cascaded trainings, study visits, team building, assessments) been in improving border security and surveillance in line with the EU’s integrated border management policies and strategies? To what extent have project activities responded to the needs of target groups and beneficiaries (e.g., national and provincial administrations, private sector, regulatory authorities)?
To what extent have the project partners/beneficiary institutions increased their capacity on implementation of the project interventions such as DGCE’s radiation detection response capability (e.g. alarm handling, NORM identification, emergency protocols) and prevention of smuggling via postal/express courier channels (e.g. risk targeting, profile-based selection, OSINT use)?
How did the project contribute to enhancing inter-agency information sharing and operational coordination?
What are the key factors contributing to project success or underachievement during the project execution? How might this be improved for future programming?
To what extent have monitoring mechanisms supported effective implementation and results achievement?
To what extent and in what ways has ownership - or the lack of it - by the partners/beneficiary institutions impacted the effectiveness of the project?
Efficiency
Key evaluation question-4: To what extent were project resources (financial, human, and time) used optimally to achieve results, and how well did management, partnerships, and adaptive approaches support cost-effective and timely implementation?
Sub-questions:
How well did the resources (funds, expertise, and time) convert into results? To what extent were project outputs delivered on time and with high quality? To what extent has the project ensured value for money?
How efficient are the project’s approaches to improving border security and surveillance (e.g. ToTs, cascaded trainings, study visits, team building, assessments) compared to alternative approaches?
What was the progress of the project in financial terms, indicating amounts committed and disbursed across results (total amounts & as percentage of total) by IOM? Was funding sufficient for the achievement of results? (funding analysis)
To what extent and in what ways has ownership - or the lack of it - by the partners/beneficiaries impacted on the efficiency of the project?
Was there any identified synergy between IOM initiatives/projects that contributed to reducing costs and optimizing resources while supporting results?
How well did implementation arrangements work for achievement of results? Did the project have the necessary coordination mechanisms, implementation arrangements and communication flow to ensure that the allocated resources efficiently converted into the expected outputs?
To what extent did project MEAL systems provide management with a stream of data that allowed it to learn and adjust implementation accordingly?
What type of (administrative, financial and managerial) obstacles did the project face and to what extent has this affected its efficiency?
How appropriate and effective has IOM’s partnership strategy employed by the project been? What factors contributed to its effectiveness or ineffectiveness?
- How well did the project respond to external shocks or contextual shifts? (adaptive management)
Likely Impact
Key evaluation question-5: To what extent is the project contributing, or likely to contribute, to sustainable improvements in border security, surveillance capacity, and broader institutional and systemic outcomes at different levels?
Sub-questions:
What evidence exists of emerging changes in capacity of beneficiary institutions as a result of the project?
To what extent is the project likely to contribute to longer-term improvement of border security and surveillance capacity in line with the EU’s integrated border management policies and strategies?
What broader effects (positive or negative/intended or unintended) are emerging from the project’s interventions at the individual, organizational, or national level?
Sustainability
Key evaluation question-6: To what extent are the project’s results, capacities, and systems likely to be sustained over time through institutional ownership, policy integration, and availability of financial and human resources?
Sub-questions:
To what extent have partners/beneficiary institutions demonstrated ownership of the intervention modalities created by the project? To what extent has this project induced prospect for active policies targeting target groups to be pursued by the beneficiary institutions to improve the overall efficiency of their services?
To what extent are project results embedded within existing institutional systems (e.g. Radiation Control Plan, risk profile portfolios, training curricula, OSINT unit)?
What systems, capacities, and resources are in place to sustain results after project completion? To what extent do legal frameworks and governance structures support continuity of project benefits?
What are the main risks to sustainability (institutional, financial, political, operational)?
Are the legal frameworks, policies and governance structures and processes in place for sustaining project benefits?
To what extent can project approaches be replicated or scaled up across Türkiye or in other contexts? What measures (including exit strategy elements) are needed to strengthen sustainability?
Crosscutting Issues
Key evaluation question-7: To what extent did the project integrate and uphold principles of human rights, gender equality, inclusion, conflict sensitivity, and equity in its design, implementation, and outcomes?
Sub-questions:
To what extent has the project integrated human rights, gender equality, disability inclusion, and Leave No One Behind principles in its design, implementation, and monitoring where applicable?
To what extent are project activities accessible and responsive to the needs of vulnerable groups, including disadvantaged groups of women and persons with disabilities where applicable?
What barriers exist to equitable participation and benefit among stakeholders and beneficiaries?
What measures could strengthen inclusion and equity in future programming?
To what extent has the project applied conflict-sensitive and Do No Harm approaches, particularly in sensitive migration contexts?
To what extent has the project contributed to strengthening the protection and human rights of migrants in Türkiye?
Evaluation Approach and Methodology
The evaluation should be transparent, inclusive, participatory and utilization-focused. The overall
methodology should be implemented following a theory of change approach, framed by the UN/OECD DAC evaluation criteria drawing upon mixed methods (quantitative and qualitative) data to capture direct project results as well as (likely) contributions.
In line with good practice in evaluating this type of complex system change-focused intervention, the overall methodology should be based on three concrete pillars:
- the project’s theory of change