Why predictions are so difficult to make in the social sector

As 2011 comes to an end and we look forward to 2012, pieces predicting what will happen in the coming year are popular in every industry, including the social sector. While making predictions is no easy task, not even for psychics, making predictions in the social sector is especially difficult.

Most industries tend to have clear industry leaders. These organizations, like Apple, Chevron, and Wallmart are significantly large that they can in many ways move industries on account of their sheer size. In the social sector we have no clear market makers, with the exception of governments. But governments are inherently unpredictable. Democracies are subject to deadlock, compromise, and regular leadership change. Dictatorships, while marked by leadership consistency, lend themselves to rulers with unpredictable and often erratic behavior and management decisions.

Of course, a common metric used for industry predictions is consumer demand. While a company like Amazon can use its vast consumer behavior database and market research to predict the directional winds of what consumers want, addressing needs is a significantly different undertaking. The demand we address tends to be either immediate (homelessness, hunger) or arguably intractable (systematic poverty).

Aggregate social indicators like poverty and unemployment rates are the most popular indicators used for predicting future demand. Yet the reliability of these predictors is questionable considering the regularity of reports about how organizations are blindsided by demand for their services. Indeed, these social predictors are highly imperfect, especially considering our inability to agree on a definition of poverty in the United States and the persistently misleading reports of homeless counts.

While big-data is being put to lucrative use in some sectors, especially technology, the most accurate predictions are based on the aggregation of front-line organizations’ own data. Big-data is itself by definition aggregations of micro-consumer behavior and transactions. The difficulty we face in applying these same tactics in the social sector however is that in order to aggregate meaningful large-scale data-sets, we must first collect and analyze meaningful micro-sets.

To this end, my company is working with organizations helping them setup data collection pipelines and analytic processes so they can better make predictions of both demand and impact. Making increasingly more accurate organizational level predictions is a necessary first step to reliable industry wide predictions.

Despite all the blog posts, books, and questionable models to the contrary, we are a long ways away from being able to predict with any relevance the future of the social sector. Therefore, the only prediction I will make for 2012 is that the future is unclear.