Evaluation of the FMO-MASSIF Fund (2015-2019)

Background
At the request of the Dutch Ministry of Foreign Affairs, SEO conducted the 2015-2019 evaluation of the ‘MASSIF’ investment fund that the Dutch Development Bank (FMO) manages on behalf of the Dutch government. MASSIF’s key objective is to increase access to financial services for micro-, small- and medium-sized entrepreneurs (MSMEs), with a specific focus on the ‘unbanked’, agricultural and rural areas, female entrepreneurs, and innovations in inclusive business. The evaluation had both accountability and learning purposes, and assessed MASSIF’s efficiency, additionality, catalytic and demonstration effects, and end-client impact. It also included an assessment of FMO’s risk management practices in the area of Environmental, Social, and Governance (ESG) risks. In addition to a general portfolio review, the evaluation comprised an in-depth assessment of 20 representative case study investments (in Sub-Saharan Africa, MENA, South Asia, Southeast Asia, Central Asia and Latin America). SEO carried out these case studies in partnership with Oxford Policy Management.

Results
FMO’s management of the MASSIF fund was generally efficient and largely in line with the fund’s mandate and new strategy. The MASSIF investment criteria were generally correctly implemented, and MASSIF performed well on its geographic targets (with around 40 percent of investments in Africa and 25 percent in Asia). As intended, the MASSIF fund overall took more risk than FMO’s commercial fund FMO-A, as a result of which MASSIF’s financial performance was more volatile than that of FMO-A. However, MASSIF’s operating expenses were broadly in line with those of comparable funds.

The 20 case study investments reviewed as part of the evaluation were largely financially additional, and often catalytic. Our cases studied showed that MASSIF was nearly always additional to both (a) the commercial market and (b) the core capital of Development Finance Institutions, including FMO-A. In many cases, the MASSIF investment was a first and necessary step for its clients to access the international capital market. MASSIF also often mobilised additional investments by reducing the perceived or actual risks of investing in these clients. Furthermore, the case studies suggested that Capacity Development (CD) projects financed through MASSIF often added non-financial value. However, FMO could measure and report the effects of such CD projects more systematically. (See also our evaluation of B-CD).

The impact of MASSIF investments at the level of end-clients was difficult to assess, as FMO was still in the process of improving its results framework and impact measurement tools. At the time of the evaluation, FMO already collected some relevant impact indicators at the level of target groups (e.g., the share of agricultural loans, or loans for female entrepreneurs; the number of men and women employed at the FMO client), but this was not yet systematically reported for all investments. Moreover, only limited information was available at higher impact levels, such as changes in their access to finance, employment, or income. And even when such information was available, it turned out to be difficult to attribute such observed changes to MASSIF, except as part of the in-depth investment evaluations that are occasionally commissioned by FMO.

The procedures for classifying and managing the ESG risks of MASSIF investments were generally sound. FMO screened all MASSIF investments for potential ESG risks (such as human rights infringements), and had proper processes in place to mitigate and manage such risks. Going forward, FMO could further strengthen the links between identified ESG risks and CD projects that could reduce such risks.

Methods
This mixed-method evaluation used a range of qualitative and quantitative methods. Using multiple methods and triangulating multiple information sources helped to minimise bias. Furthermore, careful and transparent sample selection procedures helped to ensure that the conclusions depended as little as possible on a particular selection of case studies or interviewees.

The most important methods and sources included:

• Review of programme and project documents, previous evaluations, and other relevant literature.
• Reconstruction of MASSIF’s Theory of Change.
• Semi-structured interviews with relevant FMO staff, including MASSIF management, investment officers, ESG officers, evaluation officers and impact team members.
• Case studies for 20 representative MASSIF investments (in Sub-Saharan Africa, MENA, South Asia, Southeast Asia, Central Asia and Latin America), involving semi-structured interviews with relevant FMO staff, MASSIF clients, co-investors, Board and Advisory Committee members, and other local and international stakeholders (both in person and online).
• Statistical analysis of MASSIF portfolio data and impact data.
• Analysis of secondary data for benchmarking (e.g. national access to finance indicators).