Month: June 2012

Anyone familiar with the concept of an outlier is fairly well inoculated from the seductive aroma of a well penned anecdote. Yet fundraising consultants continue to peddle the tired myth of the winning story, the singular parable that will endow an agency into perpetuity.

We are in the era of savvy donors. While I applaud the sector’s recognition of the importance of outcomes measurement, I’m afraid we are overemphasizing the importance of big numbers versus good measurement practices.

When I was in college I ran a small non-profit that was surprisingly competitive in grant applications. We did not necessarily have a greater impact than those organizations we competed for funding against (quite the contrary in fact). Although we did not have bigger numbers than other organizations, we did have better processes and capacity to collect and learn from our data.

Other organizations provided bigger numbers, we provided more believable ones.

The pressure to report big numbers is more self-imposed than we tend to realize. Foundations and donors are begging organizations not for more fantastically large metrics, but something they have good reason to believe is true, within some well estimated margin of error.

As donors become more sophisticated in their giving, increasingly more agnostic across focus areas, organizations must also become more sophisticated in their internal use of information and more statistically honest in their reporting.

Anyone can make up a big number. 5,439! See, that was easy. But it doesn’t mean anything. If you know it, so do your donors.

Indeed, the brilliance of the dawn of the impact oriented social investor is that organizations would do well to fully embrace evidence based processes for the betterment of program impact and the assurance of future funding.

But having data is not enough, and blindly reporting metrics or wrongly inferring meaning from misleading infographics is plainly transparent to this new crop of social investors. To win their hearts, organizations must win these donors’ minds. And in their minds, they are investors, no longer emotionally manipulatable donors.

Housing First, the approach to chronic homelessness that places people into supportive housing instead of treating them on the streets, has been largely adopted by homeless advocates due to its measurable cost savings. Numerous studies have found that the cost of placing chronically homeless persons in supportive housing is lower than the cost of servicing them on the street.

Housing First’s detractors argue investment in Housing First is crowding out more traditional interventions like homeless shelters, which serve chronic and non-chronic homeless persons alike.

While the research on Housing First has largely focused on the cost of housing versus not housing the chronically homeless, the larger debate is really about how to best allocate the total sum of dollars available for homeless services, chronic and non-chronic.

Ultimately, any social investment is about maximizing or minimizing an outcome given a set of constraints. While cost has largely driven the Housing First rhetoric, another way to think about homeless interventions is to minimize the decrease in life expectancy an individual experiences due to homelessness.

According to Pathways to Housing, chronic homelessness decreases life expectancy by an average of 25 years. Given this assumption, I wondered what the equivalent number of years of life lost for a non-chronic homeless person would need to be indifferent between investing in chronic versus non-chronic homelessness.

In 2011, the estimated national homeless population on a given night was 636,017 people, with 107,148 chronically homeless and 528,869 not chronically homeless. To setup my indifference model, I assumed that the chronically homeless population was a closed population, that is, the same 107,148 people stayed homeless for the year (a knowingly incorrect approximation).

Calculating the number of non-chronically homeless persons is more complicated, the annualized number of non-chronically homeless persons is a function of the average duration of non-chronically homeless persons’ homeless spells (measured in months spent homeless in my model). Mathematically, the approximation of the annual number of unique non-chronically homeless persons is the point in time count divided by the fraction of the year spent homeless.

To get the point of indifference between investing in chronic versus non-chronic homelessness, assuming all we care about is minimizing the total number of years of life lost, we need to find the point where the sum of all years of life lost for chronically homeless persons is equal to the sum of all years lost for non-chronic homeless persons.

The following chart shows the average number of years a non-chronically homeless person would have to lose on account of their homelessness based on the number of months in the year spent homeless. Assuming a twelve-month average duration of homelessness, non-chronic homelessness would have to take an average of five years off of non-chronically homeless persons’ life expectancies to equal the twenty-five years average loss of life for non-chronically homeless persons.

The above calculations make a lot of assumptions, and are better understood as approximations rather than concrete guidelines. Furthermore, the model assumes that we value the total number of lives lost equally, that is, that twenty-five years of life lost for one person is equivalent to one year of life lost for twenty-five people, which might not be how you actually think about the value of life.

I’m not sure what the average duration of non-chronic homelessness is, so I can’t necessarily weigh in on whether this model would suggest a change in the current investment mix in homeless services. Regardless, if the social sector is to be more strategic with its investments, we would do well to carefully consider what outcomes were are maximizing or minimizing over. While the cost savings of Housing First are encouraging, I would hope that our objective has more to do with minimizing harm to humans than to our coffers.

Evaluation and fundraising are two very different worlds. There is a dangerous trend in the social sector to conflate evaluation with fundraising. To be very clear, the skillsets and objectives of an evaluator are different than those of a grant writer and fundraiser, as well they should be.

I care a lot about improving evaluation, and data literacy, in the social sector. It is the only way we will be able to move beyond our current level of collective impact (whatever that level is). As a firm that specializes in helping organizations learn from their evaluative metrics, I often struggle with the best way to position our work.

On the one hand, our customers are significantly more competitive in grant applications because they have a better approach to understanding their outcomes, and improving programming accordingly, than other agencies. But while our customers enjoy a competitive advantage in grant applications, my partner and I have always resisted the urge to encourage organizations to work with us to improve their financial bottom lines.

The reason we have resisted drawing a relationship between fundraising and evaluative metrics is because our job is not to help organizations “tell their story” or “prove an organization is a great non-profit”. We try to help our customers learn from the reality of their program impact (or lack thereof), and to improve their programming based on as true an estimate as we can get of the effects of their interventions.

The fact is that funders are desperately looking for any evidence that their dollars are making a difference in the world. That is why our clients are more competitive in grant applications, because they can demonstrate an enhanced capacity to evaluate their outcomes and learn from their mistakes. This is not the same thing as demonstrating that they are awesome, rather, it is signaling that they are capable of identifying where they are awesome, where they are not, and how to improve.

I saw a job posting on Idealist for a director of grants and evaluation. The job description largely entailed focusing on grant opportunities, with the “evaluation” portion solely dedicated to proving program impact. This is absolutely the wrong way to think about evaluation.

I do believe it makes sense for organizations to dedicate some of their grants budget to evaluation, but not to prove program impact. Instead, any organization that can demonstrate an ability to identify program success and failure, and the capacity to learn from those results, will stand out to evidence starved funders.

The Department of Veterans Affairs and the White House are holding an app competition for mobile applications that connect unhoused persons to social programs. The competition has announced the top five finalists, including the demeaningly named Sherlock Homeless, but triviality has already stolen the show.

The premise of the app competition was flawed from the outset, and is emblematic of a chronic syndrome in the social sector. We are too easily swayed by the trends of the corporate world, time and again believing that if we just copy what those in the corporate sector do we will enjoy success.

In the mid to late 2000s the craze was hiring MBAs into the social sector. If only non-profits were more like businesses! The economic collapse at the hands of MBAs cooled that trend, but alas the app craze filled our empty panacea cup.

As Silicon Valley blazes trails like it did in the 1990s, the social sector has started wondering why there is an app for sharing photos with friends, but there is no app for ending poverty. Well, photo sharing is trivial, ending poverty is not.

But, we can make trivial applications about poverty. That must count for something right?

No.

And yet the White House itself is pushing the misnomer that technology can solve social problems. It cannot. If connecting people to homeless services was simple enough that a part-time developer could solve this problem, Google would have done so a long time ago. Google indexes pretty much every website on the Internet, so why does Google search fail to effectively connect people to services?

Social service agencies do not always have a web-presence, and when they do, they do not adequately maintain their sites with sufficient information to make referrals. That is why 211, and my own company, employ people to manually maintain our resource databases. The problem of maintaining resource data is not a technological one, it is logistical, and there is no app for that.

I am not arguing that there is no place for technology in the social sector. As a firm that uses technology in its work with social sector organizations, obviously I believe there is a place for technical innovation in our work. But slick, shiny apps with ridiculous names and soon to be outdated databases are not what anyone needs.

There is a reason that in the app economy apps sells for a dollar. They are easy to make, and easy to forget. We don’t need apps, we need real solutions.