Why your 100% success rate does not impress

I have recently noticed a growing number of organizations including “impact” pages on their websites. These impact pages generally include some charts and data on an organization’s supposed impact. In the spirit of openness and a sector that is moving toward managing to outcomes, I’m glad to see organizations putting their data out there for the world to see.

Unfortunately, a lot of this data is crap.

While social value is supposed to be the work-product of a social program, the reality is that frontline organizations trade the pretense of impact (whether actualized or not) for funders’ dollars. These financial arrangements logically put pressure on implementing organizations to report good news. And oh boy do they.

I was reading through the impact page of a nonprofit that aims to reduce recidivism by mentoring prisoners while in prison, and then helping those prisoners find paying jobs when they reenter society. This organization boasts a too good to be true 100% success rate. 100%! None of their clients ever return to prison.

Amazing right? The organization goes on to compare their 100% success rate to the 50% national average recidivism rate. This comparison certainly sounds impressive, but I doubt it really is. In fact, comparing the outcomes of the prisoners this organization serves to the overall prison population average is especially misleading.

In a plucky video on the organization’s website, the chief development officer wisely proclaims that you can’t help someone who doesn’t want to help themselves, and therefore their program only serves those who are motivated to leave prison once and for all.

Therefore, the proper comparison group is not all prisoners, but rather only those prisoners who are motivated never to return to prison. Indeed, the effect of their mentoring and job placement program is intermingled with the personal motivations of especially motivated prisoners.

So – is 100% still an impressive statistic? I don’t know. Maybe it is. But, the people in their program might have succeeded regardless. Indeed, 100% of successful people succeed.

I don’t mean to single this one organization out, but rather use this example to illustrate a dangerous trend. I’m disheartened by the number of organizations I see proclaiming ridiculously astronomical success rates with pure confidence their interventions are the causes of such outstanding results. If this is how we plan on using data in the social sector’s great data renaissance, then we’re in trouble.

Part of growing into a savvy consumer is learning the difference between a scam and the real deal. When we see a deal that looks too good to be true, we tend to be skeptical. And rightly so.

Given this obvious skepticisms to scammy sounding pitches – I’m truly amazed at how many of these exact types of pitches we make in the social sector.

Donors aren’t stupid, and they can tell when something is too good to be true. If you find that your program is 100% successful, instead of declaring to the world that your organization is genius, you’re probably better off reassessing your evaluation, because I’m about 100% sure you’ve made a mistake.