Evaluation of PUM Netherlands Senior Experts Programme, 2017 – 2020

Background
The Ministry of Foreign Affairs (MFA) of the Netherlands commissioned SEO Amsterdam Economics to carry out an evaluation of PUM Netherlands Senior Experts, a volunteer organisation founded in 1978. Through PUM, Dutch volunteer experts with business expertise provide advice and training to small and medium-sized enterprises (SMEs) and local business support organisations (BSOs) in developing countries. The overall aim is to improve the knowledge, practices, and performance of these PUM clients—thereby contributing to the sustainable economic development of SMEs. The evaluation covered the period 2017 – 2020, during which MFA provided PUM with a total subsidy of EUR 45 million. The purpose of the evaluation was two-fold: (1) to assess the effectiveness, relevance, additionality, coherence and efficiency of the use of public funds (‘accountability’), and (2) to draw lessons for future programming (‘learning’). SEO carried out the evaluation between September 2020 and March 2021, including fieldwork in Uganda and Myanmar conducted by SEO’s local consultants.

Results

• Effectiveness: PUM’s senior experts were widely valued for their technical expertise, and were seen as effective in transferring knowledge. There was robust evidence, based on survey data and case studies, that PUM’s advice generally helped to improve its clients’ business practices and business performance (although evidence for the latter was weaker), particularly in Least Developed Countries.
• Relevance: Most stakeholders agreed that the advice provided by PUM experts generally responded well to the actual needs of PUM clients. However, PUM did not always seem to optimally select those clients, sectors, or countries where PUM advice could have the largest impact. PUM’s relevance to MFA’s policy objectives on Private Sector Development (PSD) was generally high but could still be further improved.
• Additionality: Multiple sources indicated that PUM’s advisory services were highly additional to what was available from local markets or the international market. Even when similar services were available from the market, PUM was seen by many stakeholders as contributing to changes in development-relevant outcomes that would not have materialised without PUM. Nevertheless, there is a risk of market distortion arising from the fact that PUM, as a volunteer organisation, offers its advisory services below market prices. To mitigate this risk, PUM should strengthen safeguards to ensure that it remains additional to commercial sources.
• Coherence: There was mixed evidence with regard to PUM’s internal and external coherence. In particular, the coordination with other Dutch PSD programmes could be improved.
• Efficiency: With regard to cost-effectiveness, PUM’s total costs per mission increased slightly between 2017 and 2019, but the number of missions per FTE did not show major efficiency gains or losses during this period. With regard to timeliness, the matching process seemed efficient. However, efficiency gains were still possible with regard to the preparation time before missions and the time needed to arrange follow-ups.

Methods
This evaluation used a mix of quantitative and qualitative methods, using Contribution Analysis as the overall evaluation methodology, and an Embedded Case Study Approach for the case studies. The evaluation followed OECD-DAC criteria and the DCED (Donor Committee for Enterprise Development) definitions of additionality. The main quantitative and qualitative methods included:

• Extensive desk review of programme and project documents and case study specific background literature;
• Reconstruction and development of a new Theory of Change (ToC) for PUM, jointly developed with PUM through a ToC workshop;
• In-depth analysis of monitoring data from PUM’s own systems and secondary data for benchmarking outcomes;
• Statistical analysis of existing survey data from PRIME (an ongoing annual survey implemented by PUM in partnership with Wageningen Economic Research);
• Design, implementation, and analysis of a newly designed online stakeholder survey among a broad range of PUM stakeholders;
• Interviews and focus group discussions (FGDs) with PUM management, PUM local representatives, MFA, and representatives of two comparable Dutch government funded private sector development programmes.
• In-depth case studies, covering 10 PUM projects in two countries (Uganda and Myanmar), carried out in cooperation with local consultants. The case studies involved case-specific desk research as well as in-depth interviews and FGDs with a variety of PUM stakeholders, resulting in two separate case study reports (one per country).