Home 9 Editor's Picks 9 Impact Underperformance

Impact Underperformance

What it looks like, and why it matters

Tackling global social and environmental problems is hard. Yet, seven years of survey data collected by the Global Impact Investing Network (GIIN) show that impact investors report overwhelming success in their results. Since the first round of data collection in 2014, the proportion of impact investors reporting falling short of their impact expectations has hovered between only 1% and 3%. Given the known difficulty of measuring impact rigorously, how do we make sense of such confidence?

Impact Performance relative to expectations graphic

To answer this and other questions, the Wharton Social Impact Initiative undertook the largest ever qualitative study of impact measurement in impact investing, interviewing staff at 135 leading impact investing organizations (including asset managers, fund advisers, limited partners, and consulting firms). Among the questions asked was a request for examples of deals where investees showed disappointing impact. The goal of these questions was to dig beneath the survey data and talk directly with investors about how they see and think about impact underperformance.

Why impact underperformance is so important

Socially responsible investors have long tried to avoid or minimize negative impact by screening out investments that perpetuate child labor, deforestation, and other clear negative outcomes. Impact investing field-builders have aimed to differentiate their practice from such negative screening by espousing a more nuanced and granular approach to impact assessment based on metrics and target-setting. Thus, impact underperformance includes negative outcomes but also goes further; it includes positive outcomes that have failed to achieve a predetermined threshold for success.

This all might seem like an academic exercise in clarifying terms, but the distinction between negative impact and impact underperformance should matter to practitioners. If a key element of impact investing is not just screening out categories of investments but also evaluating impact performance quantitatively, and if evaluating impact performance entails detecting and reporting both success and underperformance, then impact investors should be measuring their impact in a way that is sensitive to possible underperformance. In short, whether and how impact investors measure impact underperformance can be seen as a test for whether impact measurement helps to inform actual performance evaluation. As such, it merits much more attention than it has received so far.

The role of metrics in evaluating impact performance

So, what were the results of this research? First, there was a striking inconsistency in how impact investors described their use of metrics for tracking success and underperformance in the deal examples requested in the interviews. While 82% of respondents gave an example of a deal where they determined impact success at least partly with impact metrics, only 24% did so in regard to an example of impact underperformance.

The distinction between negative impact and impact underperformance should matter to practitioners.

Interviewees often described cases of underperformance in terms of commercial failure or mismanagement — poor financial returns, low market penetration, managerial incompetence, etc. Also common was a general feeling of mission-misalignment stemming either from a strategic pivot that led to mission drift or from inadequate due diligence. As one respondent explained in reference to an underperformance example, “For me, personally, it was…a bit of a weird deal from the very beginning in terms of not neatly falling into what you would traditionally envision as a positive impact.”

Four Zeros on signs

Based on these findings, it appears to be common practice to measure success with impact metrics but to gauge impact underperformance either with quantitative indicators of business shortcomings or with qualitative information on organizational characteristics. Quantitative impact data — what we typically associate with impact measurement — are generally not utilized in assessments of underperformance. One of the interviewees recognized this pattern from her experience, noting that “the objective of impact data has not been to assess underperformance [but rather] to share success. If there are some really glaring issues, you would see them…through other operational data.”

Interviewees also reported that the bulk of impact assessment happens before capital is deployed. As one respondent explained, “I think a lot of investors have already made that decision that good things will happen at the time of the investment.” This finding suggests that most impact investors assume impact performance to be positive as long as (1) they believe in their initial designation of investees as impactful enterprises and (2) these enterprises continue to grow after investment. At that point, impact measurement — largely through data on volume, demographics, and short-term outcomes — serves mainly to reinforce these ex-ante determinations of impact.

Applying these interpretations to the survey findings mentioned at the beginning of this article, it is likely that impact investors report low levels of impact underperformance because (1) they are generally confident in how they identify impactful businesses in their screening processes and (2) their investments generally perform reasonably well financially.

Takeaways for the field

It is possible to conclude from these findings that impact investors are failing to evaluate their impact performance meaningfully, since impact metrics are rarely used to assess underperformance. By comparison, detecting and reporting underperformance is a key part of financial analysis. A venture capitalist might draw on “gut feel” or subjective impressions when forecasting future earnings, but if those hunches are wrong, then the financial reports will provide that final verdict in the form of numbers showing underperformance. One might argue that the same standards should apply to impact evaluation.

An alternative conclusion — one more sympathetic to investors — is that professional rhetoric on impact evaluation needs to better accommodate practical realities. Impact is often much more difficult to evaluate than financial performance. From this perspective, the inconsistent use of impact metrics reflects an understandable balancing act – i.e., investors feel compelled to share data on results but also need to avoid draining off time, money, and energy on complex impact evaluations.

We encourage a middle road between these two conclusions.

There is clearly an opportunity for impact investors to draw more on impact data following the deployment of capital to monitor and evaluate performance, even if it is underwhelming performance. Fortunately, there are a growing number of valuable resources for doing so. For example, one of the most exciting developments in the field is the emergence and growth of 60 Decibels, a social enterprise that fields mobile, voice-based surveys to understand customer experience with impact-focused products.

However, even with the power of a tool like 60 Decibels, the fundamental question of “Did my investment have an impact?” is often extremely difficult for investors to answer. For instance, consider the recent impact report that 60 Decibels released on off-grid energy products such as clean cookstoves. The report offers an impressive array of data showing the extent to which clean cookstoves reach people living in poverty, the proportion of customers saying that their lives have “very much improved,” and the percentage of customers who have experienced challenges using the product. There is much to learn from these types of indicators, and committed impact investors should make every effort to collect such information. Still, there is a categorical difference between the data that 60 Decibels produces and the data that researchers have used to rigorously assess whether clean cookstoves affect health outcomes and air quality. The vast majority of impact investors lack the time, training, and funding to emulate these types of evaluations of complex social impact interventions. Given this reality, it often makes more sense to review such research evidence prior to investment in order to inform an overall impact investment strategy.

While there is a place for impact metrics throughout the life of a deal, there are major hurdles to post-investment impact measurement providing the kind of transparency and accountability that investors expect from financial reporting. With this reality in mind, we argue that impact investing field-builders should emphasize the value of sound and comprehensive due diligence, such as checking research evidence to evaluate impact theses and thinking carefully about potential unintended consequences.

Impact investing can benefit from investors assessing and reporting impact underperformance with the same diligence that they apply to cases of success.

Another takeaway is that the field of impact investing can benefit from investors assessing and reporting impact underperformance with the same diligence that they apply to cases of success. It is, of course, understandable that investors may hesitate to volunteer information about falling short of impact goals, but documenting how impact investments miss their marks is central not only to maximizing the evaluative utility of impact measurement but also to maintaining the learning orientation that investors need in order to deepen and scale their impact. Organizations that may face less pressure to impress outside investors (e.g., foundations with program-related investment portfolios) are best positioned to initiate this critical change in how the field views data on impact underperformance.

Closing thoughts

Many different outlooks, strategies, and experiences surfaced across the 135 interviews conducted for this research. However, a consistent theme across these conversations was a profound commitment to combating social injustice and environmental degradation. The challenge facing the impact investing field is how to advocate for a form of impact measurement that effectively captures and optimizes the results of that commitment but that also does not get in the way by setting up unrealistic standards. A vital step in moving toward such practice is to stay grounded in the actual day-to-day work of impact investors by, simply put, talking to them — learning about their hopes, understanding their frustrations, and ultimately uncovering opportunities for advancing best practices.

Maoz (Michael) Brown is Head of Research at the Wharton Social Impact Initiative, where he directs research on impact investing and social enterprise. A focus of Michael’s research has been how impact investors measure, manage, and report their impact. He holds a PhD in Sociology from the University of Chicago.
Lauren Kaufmann is an Assistant Professor of Business Administration at the Darden School of Business at the University of Virginia. Her research focuses on social impact, impact investing, and business ethics. She holds a PhD from Wharton and a MSc from the London School of Economics and Political Science.
Sankalp Global

Related Content

Comments

0 Comments

Submit a Comment

Sankalp Global Summit

Deep Dives

What is Resilience?

RECENT

Editor's Picks

Webinars

News & Events


More News & Events

Subscribe to our newsletter.

Subscribe to our newsletter to receive updates about new Magazine content and upcoming webinars, deep dives, and events.

Access all of Impact Entrepreneur.

Become a Premium Member to access the full library of webinars and deep dives, exclusive membership portal, member directory, message board, and curated live chats.

ie frog
Welcome to Impact Entrepreneur MagazinePlease join us!

While all magazine articles are free to view, you need to choose a membership option to continue using the site. Basic Membership is free and gives unlimited access to articles. Premium Membership brings a host of rich offerings and collaborative opportunities.

Or LOGIN HERE if you are already a member.

Click