Evaluative metrics bring truth in advertising to the social sector

My primary focus is helping organization’s improve their programs based on outcomes metric, but there is no doubt that fundraising and development are on the minds of every agency executive that I work with. I used to shy away from this fact, but I now embrace the desire of agency executives to use evaluative data for fundraising purposes as an opportunity to bring truth in advertising, a particularly exciting prospect as donors become increasingly savvier investors.

It is undeniable that there are plenty of organizations (and people) that overplay their contributions to the public good. But there are also organizations that unwittingly mask their outcomes in poorly defined metrics, framing their social value to donors in ways that are at odds with their own internal organizational decision calculus.

A large affordable housing development and homeless service providing organization I work with is a great illustration of how our choice of outcomes metrics can obscure the real value an organization aims to optimize over.

As is typical in homeless services, this organization reports, among other things, the number of people housed annually. The problem with this metric is that it values housing a mentally ill chronically homeless person who has been on the streets for 19 years the same as housing someone who slipped into homelessness for one month due to momentary economic shocks, like job loss.

Assuming the number of people housed is actually what this organization wanted to maximize, the rational thing to do would be to move away from chronic homelessness and only house those whose homeless spells is likely to be very short. But this is not how the organization actually thinks, or how it makes program decisions.

This agency, again like many homeless service providers, cares deeply about those experiencing the continuum of homelessness, especially those who are chronically homeless. In economic terms, the organization derives more utility, or value, from housing a more difficult to house individual than from a less difficult to house person.

The trick then is to first internally formalize the organization’s utility framework, and then to identify the outcomes metrics the organization is actually optimizing over. For example, instead of the number of people housed, a more meaningful metric might be the number of years of homeless prevented, or the number of years of life preserved that likely would have otherwise been lost. Not only are these metrics better representative of how the organization plans its interventions, it also paints a more complete picture for potential donors.

In cases like this, I embrace the fundraising and development aspirations of the organizations I work with to the extent that helping them better understand their data will actually move them closer (rather than further) from articulating a truer story of the social benefit their agency contributes toward.