Evaluating your organization’s use of metrics

Evaluating organizational effectiveness is a growing sub-field of the social sector, with a slew of competing measurement frameworks. Something a lot of these frameworks assess is whether organizations make use of data management system. The idea is that an organization that has a data management system in place is more likely to be data savvy and to actively manage to outcomes.

This might be a reasonable proxy for whether an organization actually incorporates evidence in its practice. But from where I stand, data only has value in so far as it helps an organization make higher impact decisions. Therefore, I propose a more robust approach to evaluating an organizations use of metrics.

If an organization’s behavior before implementing a data oriented approach is exactly the same after implementation, then no value has been added. Data should help inform action, not just confirm prior beliefs. It’s hard to imagine any organization (or individual) that does everything so perfectly that there is no room for improvement.

Effective uses of data collection and evaluative analysis should help drive program improvements. If you can’t identify any changes in your organization’s behavior, then whether or not the organization has a data management system and processes in place to collect information, it has not actually benefited from its data efforts.

More important, just because an organization changes its behavior based on metrics does not necessarily mean it benefited from an effective use of information. Indeed, information should help us make better decisions. In some cases, organizations make poor decisions that are backed by data.

A classic example is the Space Shuttle Columbia disaster. NASA used information to back up its decision to go ahead with a shuttle launch in suboptimal weather, that lead to the disintegration of the shuttle’s O-ring and a subsequent explosion that killed all the astronauts on board.

In this case, NASA made a data-backed decision, but it used its data incorrectly and made the wrong inference, resulting in disastrous consequences. Therefore, it’s not only important to collect your data and use it, but to take care to analyze your data properly, and listen to reason of all parties.

Which leads me to my most important indicator of whether an organization uses information well. Information should inform decision making, but it should not necessarily, on its own, dictate what an organization does. While having a data management system in place is great, and using reasoned analysis in the interpretation of data is better, there is no replacing the judgment of experienced practitioners and the feedback of those we serve.

The best organizations include evaluate metrics as a part of their decision frameworks, but they do not supplant their own judgment for a regression.