Month: January 2012

Evaluating organizational effectiveness is a growing sub-field of the social sector, with a slew of competing measurement frameworks. Something a lot of these frameworks assess is whether organizations make use of data management system. The idea is that an organization that has a data management system in place is more likely to be data savvy and to actively manage to outcomes.

This might be a reasonable proxy for whether an organization actually incorporates evidence in its practice. But from where I stand, data only has value in so far as it helps an organization make higher impact decisions. Therefore, I propose a more robust approach to evaluating an organizations use of metrics.

If an organization’s behavior before implementing a data oriented approach is exactly the same after implementation, then no value has been added. Data should help inform action, not just confirm prior beliefs. It’s hard to imagine any organization (or individual) that does everything so perfectly that there is no room for improvement.

Effective uses of data collection and evaluative analysis should help drive program improvements. If you can’t identify any changes in your organization’s behavior, then whether or not the organization has a data management system and processes in place to collect information, it has not actually benefited from its data efforts.

More important, just because an organization changes its behavior based on metrics does not necessarily mean it benefited from an effective use of information. Indeed, information should help us make better decisions. In some cases, organizations make poor decisions that are backed by data.

A classic example is the Space Shuttle Columbia disaster. NASA used information to back up its decision to go ahead with a shuttle launch in suboptimal weather, that lead to the disintegration of the shuttle’s O-ring and a subsequent explosion that killed all the astronauts on board.

In this case, NASA made a data-backed decision, but it used its data incorrectly and made the wrong inference, resulting in disastrous consequences. Therefore, it’s not only important to collect your data and use it, but to take care to analyze your data properly, and listen to reason of all parties.

Which leads me to my most important indicator of whether an organization uses information well. Information should inform decision making, but it should not necessarily, on its own, dictate what an organization does. While having a data management system in place is great, and using reasoned analysis in the interpretation of data is better, there is no replacing the judgment of experienced practitioners and the feedback of those we serve.

The best organizations include evaluate metrics as a part of their decision frameworks, but they do not supplant their own judgment for a regression.

As the furor to incorporate metrics in the social sector grows, organizations are feeling the heat to get more data savvy. In principle, this is a good thing. Information should help inform decision making. But there is a big difference between information informing your agenda and allowing it to set it.

Data should inform your answers to questions, but data sets should never determine what questions you seek to answer. Every organization grapples with a myriad of decision problems, from optimizing resource allocations to increasing the social impact of interventions.

The natural role of information is to help us make more informed decisions. But data does not, on its own, answer any questions. And no data set can (or should) determine the most important issues facing an organization. Those questions should be driven by the organization itself and the people it serves.

Yet time and again I see organizations blankly asserting that they need data. Why?

A lot of organizations don’t have a great answer beyond citing the overall direction of our industry. This is a pretty lousy answer, and more importantly leads to half-baked data collection implementations that do nothing to drive organizational change or improve outcomes.

Each organization I work with, before talking about data management systems, what data points to collect, or internal processes for collecting metrics, I ask them what they do and what problems they face. Simply put, you don’t know what data you need until you know what problems you’re trying to address.

I’m afraid the glorification of trivial info-graphics and blanket mandate that organizations should be “data driven” perpetuates a wrong-headed belief that there is inherent value in data. As someone whose whole lively-hood is based in data collection and analysis, let me be as clear as I can, data only has value when it informs a decision.

As a sector, we’d be wise to focus less on the “data, data, data” mantra, and to instead engage in discussions about the issues organizations face, and where metrics can help inform better decision making. Despite the misleading glee of those who proclaim the data revolution will transform the social sector, data itself is nothing but a distraction unless it answers specific questions an organization faces.