Exploratory Study of Mainstreaming Scaling

, , Announcement, CoP Reports,
Download the paper here

1. Introduction

Despite increasing discussion within the international development community about the importance of scaling, actual scaling activities and the resources devoted to it remain at a low level.  This contributes to the general underperformance of development assistance compared to the need and threatens to prevent the global community from achieving the Sustainable Development Goals (SDGs), delivering on climate change commitments, and achieving the intended results of numerous other global initiatives.

While there are many reasons why scaling is not happening, there is a general consensus that a lack of systematic focus on — and funding for — scaling is a central factor.  This is compounded by the fact that most of the projects and uses which development assistance supports remain largely determined by donors themselves i.e. is supply-driven. In this context, the Scaling Community of Practice (CoP) has proposed to do a major study of mainstreaming of scaling by international development funders (IDFs),[1] hereafter just referred to as mainstreaming. Given the ambition and resources required for such a study, the CoP commissioned preliminary research on mainstreaming to inform and guide that study. This research, the results of which are presented in this note, had five objectives:

  1. determine if a larger mainstreaming study makes sense, and, if so, inform a decision about its intended audience and primary purpose;
  2. confirm or refute the assumption that action by funders to support scaling and to mainstream a focus on scale within their policies and practices remain limited;
  3. do a high-level assessment or landscape analysis of where mainstreaming is happening, the form(s) it is taking, and the characteristics, drivers and obstacles to mainstreaming efforts;
  4. develop tentative hypotheses and topics that could be addressed in greater detail in a full study; and
  5. make recommendations as to the methodology such a study should take and the organizations that would serve as good case studies.

The information sources for the preliminary study were intended to be a literature review, interviews with key informants (KIs) drawn from the CoP and from IDF staff, and a survey of CoP members.  As it turns out, repeated Internet searches revealed that there is no literature on mainstreaming to speak of, with the significant exception of several papers prepared by researchers at Brookings (Annex 1).  As a result, the preliminary study relied largely on seventeen KI interviews (Annex 2) and the results of a fourteen-question survey of CoP members to which roughly fifty people responded.

Before reviewing the findings of the study, we wish to highlight three issues that arose.

  • There was universal agreement that the most valuable approach to a larger mainstreaming study would be a set of several case studies and cross-case lessons. However, the preliminary review suggests that the diversity of the international development finance ecosystem significantly complicates the task of cross-case comparison.  Size (in terms of financial resources) is one major differentiator, but so is the breadth of activities or niche in scaling that different IDFs occupy. Comparing small innovation funders that do not directly fund scaling or support sustainability to a vertical fund that supports innovation and scaling from A to Z is unlikely to yield significant insights.  Another important distinction is between funders that provide only financial resources versus those that also provide technical support or direct (project) implementation.  Finally, funders differ in their mandates, incentives and resources designated to support scaling; very few (e.g., vertical funds)  appear to combine all three.
  • There is widespread conceptual confusion and disagreement both across and within IDFs as to how they define scale, scaling, and successful impact at scale. A larger study will need to have a clear definition of these terms, and a consistent way to judge “good practices” and “institutionalization”, perhaps based on the CoP’s Scaling Principles. Unfortunately, these definitions are likely to conflict with the understanding or current practice in many IDFs.
  • The conduct of a mainstreaming study is likely to be politically sensitive influencing the willingness of organizations to share access and their willingness to support publication and dissemination of the results that emerge. Two proposals to address this sensitivity deserve consideration.  First, doing the study in partnership with internal actors in IDFs, e.g., an evaluation or knowledge management office, and, second,  framing the study as a learning journey designed to assess and advance mainstreaming, rather than just to (critically) assess current practice.  In both cases these approaches would need to include some measures of confidentiality and anonymity and emphasize positive achievements rather than focusing on criticism.

2. Findings

There was universal consensus among all KIs that a larger study on mainstreaming would be important, and that the CoP is well-placed in terms of brand, capacity, and reputation to lead such a study.  Consensus was nearly as universal that the study’s most important purpose was to support and promote advocacy for mainstreaming and scaling in general. Therefore, the study should primarily focus on identifying and analyzing inspirational examples of mainstreaming and be explicitly embedded in a well-resourced and ambitious advocacy effort. (This focus could be complemented by looking at failed efforts or non-efforts, assuming such IDFs could be persuaded to participate).  As such, the primary criterion for choosing topics, questions, methodology and participants should be: what would best serve to promote mainstreaming?

Beyond this agreement, there was substantial disagreement about whether a study should focus only on what might be called internal administrative changes or whether the focus should include looking at how these changes are actually translated into what gets funded and implemented, whether by IDFs themselves, implementing partners, governments or grantees. There was also disagreement about whether the study should include development of a framework for mainstreaming, guidance and tools. Given limitations of time and resources, and challenges to attribution, it is our view that the CoP’s study’s initial focus should be limited to the issues of strategy, internal operations, policies, procedures and resource allocation.  While a study that does not explicitly demonstrate the link between mainstreaming and results might be less credible, and one that does not produce a framework or guidelines less useful, covering these major issues appears to be too ambitious for an initial effort.  Case studies could be included in the form of individual examples of how mainstreaming had translated into scaling projects or impact, but not a comprehensive review.

In terms of the mainstreaming landscape, the study confirms that, in general, mainstreaming of policies and practices designed to increase and improve scaling are not happening in any of the major IDF’s or largely limited to inclusion in mission statements and strategy.  With the exception of a few innovation funders who are extending their support to so-called transition-to-scale grants, most IDFs other than foundations and impact investors are not allocating additional financial or human resources to scaling. Both KIs and survey respondents felt strongly that the way development projects are designed and implemented and managed by IDFs remains a major obstacle to scaling.  This observation was shared by IDFs that are major funders of innovation.[2] There are a few bright spots where mainstreaming has advanced significantly: individual bureaus within a few bilateral agencies; innovation funders now supporting transition-to-scale grants; some vertical funds; and a number of small and medium-sized foundations. However, in most cases where it is occurring, it is still early days, and partial or limited in scope, breadth and depth.  A comprehensive study would need to have realistic goals and expectations given the number of cases available, taking into account actual progress in a selection of those bright spots that might be included in a study.

There was some confusion among some survey respondents in differentiating between the mainstreaming in their own organizations (usually implementing partners, NGOs or consulting firms), and mainstreaming of changes within the IDFs that fund them.  Possibly as a result of this confusion, the survey identified integration into Project Design, Project Implementation and Monitoring, Evaluation, Accountability and Learning (MEAL) as the place where the most progress has been made in mainstreaming.

In contrast to the survey respondents, KIs identified integration of scaling into IDFs’ organizational Mission/Vision/Strategy as the principal area where mainstreaming has occurred to date.  In terms of the sequence of mainstreaming, the KIs suggest that this has translated into greater informal incentives and a culture more oriented towards scaling, and therefore somewhat more scaling in actual Project Implementation.  However, progress on integrating scaling into project design procedures, review and approval processes, MEAL systems, and guidance on how to do all of these, where it has occurred at all, has lagged behind and in most cases been partial. Except for innovation funders, few KIs reported any increased allocation of resources or any change in formal incentives to support scaling; for innovation funders this has been largely composed of grants for transition to scale, access to coaching,  support for participation in incubators and accelerators, or some combination of these.

A central obstacle noted by many was the development project approach used by many IDFs, especially that the procedures for project design and approval or grantmaking do not include or encourage scaling.  Another major obstacle, noted above, are internal disagreements or confusion as to the concepts of scale and scaling.  This goes hand-in-hand with a lack of tools, training and guidance as to what constitutes good practice in scaling, given that it is hard to provide support to achieve  something that is not well-defined.  A fourth obstacle was the inability of IDF’s to consistently partner and collaborate with each other and other stakeholders.  Finally, and perhaps most importantly, many KIs cited internal bureaucratic and administrative rules, regulations and procedures that are often inimical to scaling, such as contracting procedures, though the specific nature of these obstacles varied across different IDFs.

Respondents report that  drivers of mainstreaming scaling agenda are largely internal, particularly initiative and leadership by senior management, and to a lesser extent, internal evaluations that show lack of scaling.  Pressures from governance bodies, external evaluations, or local, in- country actors are cited as either absent or less effective in driving IDFs towards greater integration of scaling into their activities.

At a more granular level, obstacles differ across types of IDFs in terms of whether they have the flexibility, mandate, incentives, capacity and resources to mainstream scaling.  A mandate to support the entire scaling pathway is rare, with the possible exception of vertical funds.

Based on this analysis, an extensive list of research topics, questions and hypotheses emerged.  These include: (i) what are the drivers of mainstreaming, and how to get more senior management to back and lead initiatives; (ii) the role of organizational mandates, incentives, capacity and resources in driving and implementing mainstreaming; (iii) the role of middle management and staff in supporting or opposing mainstreaming, and how to address it; (iv) bureaucratic and administrative rules, regulations, procedures etc. as obstacles (or facilitators) of mainstreaming; (v) the role of partnerships, and how to address the mixed historical record of donor collaboration to promote scaling; (vi) the additional activities, effort and costs that a greater focus on mainstreaming might require; and (vii) the risk of higher failure rates in individual projects or grants, and how those could be addressed, such as investing more resources in a fewer efforts, a portfolio approach, using a funnel or stage gate approach, or some combination thereof.

As noted above, there were disagreements about whether to include assessment of actual scaling activity and impact, and whether to produce a mainstreaming framework, guidelines, and tools.  Regarding the first concern, we believe a feasible compromise would be to include a selected set of projects or grants which would illustrate how mainstreaming translated into specific projects, grants or other examples of good and/or greater efforts at scaling.  With regard to the second concern, the preliminary study suggests that developing a mainstreaming framework, guidance and tools should also be deferred for subsequent attention.

Those with experience in large organizations such as the World Bank, IFAD or vertical funds argued that the resources of bilateral agencies and foundations are too small, and often their mandate too limited,  to do scaling seriously and should not be the type of IDFs selected for case studies.  Conversely, KIs familiar with foundations (except for BMGF), innovation funders, and bilateral agencies felt they had little in common with vertical funds and multi-lateral development banks.  Based on these observations, the preliminary study recommends that a comparative study concentrate on one or at most two types of IDFs such as a group of bilaterals and a group of vertical funds.

The IDFs that appear to be doing the most serious mainstreaming as a group are vertical funds. Vertical funds have major advantages:  some flexibility in their organizational structure and procedures (at least initially); significant funding devoted to a narrow focus; a strong mandate for scaling and supporting the whole scaling pathway; reasonable accountability for impact at scale; and incentives to scale.  The Green Climate Fund has included scaling as a major component in its proposal evaluation criteria, though that has yet to be translated into specific criteria and consistently applied.  IFAD is probably the best example of an earlier undertaking a systematic and extended mainstreaming effort, although implementation has become in many cases a check-the-box exercise.

Many KIs felt that where vertical funds come up short is in their potential influence on other types of IDFs and serving as inspiring and relevant examples.  This view was especially common among KIs working for  bilateral agencies, foundations and innovation funders.  By contrast, multi-lateral banks (MDBs) and vertical funds seem to regard each other as peers and actors from those institutions often engage in benchmarking performance against each other.  Since MDBs  apparently largely believe, at least for public consumption, that they are already working at scale, their motivation to participate in a study seems to be somewhat different than others; more focused on how to do better the things they already do, as well as how to address the issue of sustainability.

Bilateral agencies, innovation funders, and most foundations, consider themselves a breed apart from vertical funds and MDBs.  Innovation funders have tiny financial resources; for those embedded in bilateral IDFs, their funding is often 1/100 of  what is devoted to project funding.  Their mandate is limited to funding innovation  broadly defined, though some have been stretching that to include ‘transition to scale’. They currently face a  big challenge in that they are being held accountable for impact at scale. This challenge is becoming more pressing both because many have been in existence for a decade or more, and because the SDGs they are expected to impact are no longer in the distant future. At the same time, they have neither the resources, capacity nor mandate to go much beyond funding innovations.  Those embedded in bilateral agencies generally lack any systematic pathway for getting even their own agencies to integrate their innovations into larger and better funded project efforts and grants. Given the gap between their mandate and resources on the one hand and their accountability on the other, they are strongly interested in mainstreaming or other solutions, and some are already pursuing this agenda.  In terms of considering them as case studies, their strong interest and incentive means that they may be the easiest to enroll in a larger study. However, like vertical funds, they probably have the least in common with most other IDFs because of their tiny resources, focus on innovations rather than projects, mandate limited to only the front end of scaling pathways and, in most cases, lack of any footprint on the ground.  Also, until very recently, to the extent that they have applied a scaling mindset or scalability criteria to their choice of investments, it has largely been limited to innovations that have technical promise to address important problems that exist at scale.

There are so many foundations of various size and approach that it is difficult to generalize, but there do appear to be a number of them of at least small and medium size that are taking scaling seriously.  Their biggest advantage is their flexibility.  Many focus narrowly on one or a few sectors and/or regions and can change strategy and internal procedures relatively quickly.  However, this latter characteristic separates them from most official IDFs, and most of them have much smaller resources than even small bilateral agencies.[3] Bilateral funders fall somewhere in the middle; more resources, broader mandate, but little accountability for impact at scale and therefore less incentive to mainstream.  In the larger bilateral agencies, some individual sectoral bureaus have done important work on scaling and even mainstreaming; and some bilateral agencies have incorporated a scaling narrative into their stated missions and objectives.

Perhaps influenced by the composition of the CoP’s membership, the majority of survey respondents and KIs suggested the larger study focus on bilateral project funders (or sectoral divisions within them).  There was also interest in including, ignoring the comparability criteria for a moment, vertical funds, IFAD, the Eleanor Crook Foundation (because of its total commitment), CRS, and maybe a couple of divisions of BGMF because the foundation as a whole and its strategies are so influential in international development generally.  However it is important to note that the coverage of this study was limited, and there may be other actors, especially those not based in North America or OECD countries, that should be considered. One notable omission from this study are those IDFs that take more of a venture philanthropy approach, broadly defined, such as Co-Impact, the Imago, Audacious and Jacobs Foundations, or the Skoll Foundation. There are examples of scaling in the Global South where it is happening with the donor because the grantee has really led the process.

3. Conclusion and Recommendations

In conclusion, this study confirms that there is strong support and interest in a more comprehensive study of mainstreaming by the CoP using a comparative case study approach and common definitions of scale, scaling, and good practice in scaling up.  Annex 3 presents a set of detailed recommendations for such a study. The purpose of such a study would be to confirm or modify the conclusions of this exploratory investigation and to promote advocacy for scaling and mainstreaming. To ensure comparability, it should look at similar IDFs, preferably those that would be most influential in supporting advocacy.

While there has been some mainstreaming in many organizations, in the majority of cases this remains small and in its early days; there are few cases where mainstreaming has gone far or deep. Ideally, such a study would combine  a deep dive into cases where mainstreaming is not happening, and the reasons/gaps of why not, with cases where some progress has been made.  However, getting such organizations to participate may be challenging, Therefore, evaluating hypotheses may turn into more of an exercise in highlighting successes and inspirational examples.

In the face of the political challenges in getting IDFs to participate, especially those that have done less or even failed, it seems that a partnership and/or learning journey approach to the study is likely to be most successful in both enrolling participants and producing studies that can be shared and published. This may even allow for mainstreaming failures or non-mainstreamers to be included.  Such a study, if embedded in a well-thought out dissemination and promotion strategy,  could lead in the future to helping IDFs individually and collectively develop mainstreaming strategies, producing a general framework and guidelines, and looking at the impact of mainstreaming on actual scaling and impact. While this likely involves some compromises in terms of rigor and critical analysis, it is hard to see a better alternative that is realistic and feasible given the landscape and other constraints.

 

 

Annex 1. Brookings Studies Relevant for Mainstreaming Scaling in IDFs

Annex 2. Organizations Interviewed for This Exploratory Study

Annex 3. Detailed Recommendations for a CoP Study on Mainstreaming Scaling in IDFs

This study offers the following recommendations for a comprehensive scaling study:

  1. It should focus on studying mainstreaming in terms of organizational changes occurring within IDFs. It should not include either whether mainstreaming is leading to greater actual scaling activity on the ground or impact, nor seek to develop a mainstreaming framework, guidelines or tools. That said, individual examples of projects or efforts that illustrate how mainstreaming is affecting scaling activity would be useful, assuming the additional resources required is minimal. A more thorough study of the impact of mainstreaming on actual scaling and the development of a mainstreaming framework etc. should be left to follow-up studies.
  2. The study methodology needs to use clear and common definitions of scale, scaling, and successful scaling, and a clear approach to what constitutes good practice in scaling, preferably drawing upon and aligned with the COP’s Scaling Principles. It should develop and apply a common methodological approach that allows for cross-cutting comparisons.
  3. The whole project and individual studies should proactively anticipate and address what will likely be significant political challenges and sensitivities without compromising quality or objectivity. To address these political issues, a study should focus primarily on positive accomplishments and examples, and frame obstacles or failings as issues to be addressed in moving forward.
    • It is strongly recommended that, in this vein, the study be framed and conducted as more of a ‘learning journey’ than as an arms-length evaluation.
    • Either separately or as part of a learning journey, the study should be conducted as a partnership between an external, independent team of researchers working with internal partners from within each participating IDF.
    • In the initial phase of the study, it should ensure that there is strong buy-in and support not only from senior leadership but also by middle management and staff, and especially operational divisions.
  4. The study, assuming limited resources, should limit itself to one or at most two types or categories of IDFs so as to allow for comparability and generation of cross-cutting lessons. The characteristic of the various types of IDFs, let alone individual IDFs, vary too widely to allow for meaningful comparison across multiple types of IDFs.[4] The factors to be used to select participants or IDF types should include that they:
    1. have tried mainstreaming and at least made some progress or had some success, with the potential to produce valuable lessons about drivers, obstacles and good practice in mainstreaming. This would include cases where mainstreaming efforts have been minimal or disappointing, assuming such organizations would be willing to participate
    2. currently have (or had) strong incentives to mainstream scaling
    3. have the mandate to support more or preferably all phases of scaling, particularly the phases of going to scale, institutionalization and sustainability, (or alternatively, engage in partnerships to support those phases)
    4. have the financial resources to support scaling, even if to achieve that they are narrowly focused in terms of sectors and countries
    5. are likely to use such a study of their own organization, as well as the broader study, to advance further their own mainstreaming efforts
      • are seen by other IDFs as comparable and as a source of experience and practices that would influence their own actions; would serve as inspirational examples to other IDFs; and therefore be useful to an advocacy strategy to promote mainstreaming and scaling.
  5. Based on those criteria, we recommend that the study focus on vertical funds and large bilateral agencies (or specific divisions within bilateral agencies, including innovation funders and research centers that study or support scaling). At the same time, exceptionally promising or influential examples of mainstreaming that are drawn from other categories, such as the Eleanor Crook Foundation, a small foundation, Co-Impact, an impact investor, or the BMGF (because of how influential it is), should be included, if resources permit. If resources do not permit the inclusion of impact investors, a standalone study perhaps entitled: “Impact Investors and Scaling:  Lessons for Traditional International Development Assistance” would be worth considering.
  6. We recommend that the study examine some or all of the following questions:
    • How is scale, scaling up and successful scaling defined across and within organizations?
      • How do these definitions and/or lack of alignment facilitate or impede mainstreaming and scaling?
      • Do these definitions exist simply for reasons of historical inertia, or are they fundamentally embedded in individual and organizational culture and incentives?
    • What are the major drivers of efforts at mainstreaming?
      • If it is indeed senior management leadership that is key to driving mainstreaming, what motivates senior management to initiate and lead a mainstreaming effort?
      • How can internal assessments of impact at scale (or the lack thereof) or tracking of macro-systemic indicators, be used to motivate mainstreaming, without those evaluations being perceived as threatening by operational divisions?
      • What drives an organization to expand its mainstreaming effort beyond incorporation into its Mission and Vision to other components of mainstreaming?
    • What are the major internal bureaucratic and administrative obstacles to mainstreaming, particularly those that affect project models and cycles? What reforms to those obstacles have been tried (or good practices that already exist), to make them more supportive of scaling,? What has been successful or not, and why?
    • How do efforts and progress in mainstreaming vary depending upon an organization’s niche or role within the entire scaling pathway, official mandate, funding, implementation and technical support mechanisms?
    • How can partnerships and collaboration be used to facilitate scaling in practice, especially given the numerous challenges and historically mixed performance of donor partnerships generally?
    • What has been tried, been successful or not, and why?
      • Are partnerships or handoffs between innovation funders and supporters of going to scale, institutionalization, etc. actually feasible and realistic given what appears to be a disproportionate interest and resourcing of innovation over scaling?
      • Are there practical examples of systematic approaches to innovation-scaling partnerships or hand-offs that have proven successful?
    • What are the additional financial costs, time, effort and other resources required to increase scaling efforts? How can these be justified in terms of potentially greater benefits?
      • What are the risks involved to funders in terms of their success rate and accountability, especially of an increased failure rate for individual projects or grants?
      • How can those risks be mitigated or justified in terms of, for example, greater potential overall impact? Does this require a portfolio approach, a funnel or stage gate approach, or both?

 

 

[1] IDFs include bilateral aid agencies like USAID or GIZ, multilateral development banks like the World Bank, vertical funds that concentrate on one issue, such as the  Green Climate Fund, research and innovation funders such as the various Grand Challenges or CGIAR, foundations like the Bill and Melinda Gates Foundation, and impact investors like Co-Impact. According to David Gartner and Homi Kharas “Vertical funds are global programs for allocating official development assistance that focus specifically on an issue or theme.” In their book chapter, they argue that Vertical Funds, for reasons discussed above, need to be strongly considered as an important or even the preferred vehicle for scaling.  See Gartner and Kharas: “Scaling Up Impact:  Vertical Funds and Innovation Governance”, Chapter 4 in Chandy, Hosono, Kharas and Linn, Getting to Scale: How to Bring Development Solutions to Millions of Poor People: Brookings Institution, 2013. Another institutional form with promise for scaling is impact investors who combine venture philanthropy and a portfolio approach.  As impact investors are not usually considered in the same category as official donors and foundations, we have not included them in this case study, though that could be changed.

[2] Examples of innovation funders include Grand Challenges Canada, USAID’s DIV, CGIAR centers, etc.

[3] Each of the top 24 bilateral donors give more in net ODA, over $500 million, than any individual US foundation except BMGF and Open Society. New Zealand gave more in ODA in 2021 ($600 million), than the Ford, Hewlett or MacArthur Foundations, and by wide margins in most cases.

[4] IDFs vary in terms of the size of their resources, whether they provide funding, implementation, technical support or some combination of the three; the mechanisms, financial and otherwise, they use to provide support, especially implementation;  which phases of scaling pathways they have a mandate to engage in and actually do support; the sectors and types of scaling pathways they work in; and the flexibility they have with all of the previous points.

Post Views: 95
Related Articles
0 Comments

Leave a reply

Your email address will not be published. Required fields are marked *

*

Groups

©2024 MSI

Log in with your credentials

Forgot your details?