Focus on big data misses the big picture

We are living in an era of data deluge. Social sector talking heads are preaching the promise of “big data”, yet few arguments have been made about how all this data will facilitate impact.

Data is useful only in so far as it helps inform decision making. And while social sector organizations aim to solve big problems, their solutions are decidedly smaller in scale and scope.

Indeed, an organization developing a job training program might look at macroeconomic conditions to determine the total potential market size for an intervention. But unless that intervention covers a large enough geography to significantly influence over all employment (which is rarely the case) the more useful information is the micro, “small data” that an agency collects itself.

I think about so-called big data, like Census indicators, as providing the context for an intervention. It’s the market data that develops the case for action. But once that decision to act is in place, there’s little left for big contextual data to inform.

The more pressing questions, like is this program working and how do I increase impact, cannot be answered by large public databases. Instead the focus needs to be on developing analytical capacity and program specific data collection feedback loops that capture relevant indicators on an iterative basis.

Yet our current obsession with big data and trivial infographics obscures the real promise of an analytically oriented social sector for soundbites and graph porn.

If we want to tackle the big problems, we need organizations to be able to collect and analyze small data sets relevant to their own work. Our wrongheaded focus on large scale data for the sake of seeming analytical obfuscates the real opportunities data affords.