With every organization I work with, I begin the consulting engagement by developing an impact theory. The impact theory is the portion of the theory of change that identifies the causal assumptions of how a set of social interventions is expected to drive particulars social outcomes.
The impact theory is important because it sets the basis for how a program defines success, and how it intends to get there. Used correctly, the impact theory sets boundaries around what data points an organization’s survey instruments should collect.
At Idealistics we use a database system we developed to model an organization’s impact theory. We call this system the Decision Framework, as it models the decision relevant factors an organization faces in trying to maximize social outcomes.
When I put together an impact theory, I spend time speaking with program directors and staff, hearing from them what they believe differentiates their program offerings. I’m most interested in identifying what they believe would be different in the lives of those they serve were their particular program not to exist. In evaluative speak, this is called the counterfactual.
A large service provider I am working with emphasizes the customer service aspect of their work, which includes traditional basic needs human services offerings like clothing, food, and medical services. Many of these services are available in other forms, provided by alternative agencies. However, this organization argues its interventions are unique not just because of the services offered, but the way in which people are treated when receiving assistance.
Social interventions tend to be defined as tangible outputs like medical vaccinations, counseling hours, and food baskets. While traditionally we think about interventions as activities or items that are plainly enumerable, can the way an output is delivered be an intervention in itself?
Brands differentiate themselves on service quality in the marketplace all the time, and charge a premium for it. While we all prefer not to be treated like crap, does better customer service drive better social outcomes? The impact of customer service on social outcomes is an empirical question which likely varies depending on what outcomes an organization intends to affect.
If good customer service is identified as a causal element in driving a set of outcomes, customer service needs to be operationalized to something we can measure, so we can evaluate the possible relationship between intended outcomes and quality customer care.
While an organization might assume there is a positive relationship between good customer service and positive social outcomes, the more interesting is how should an organization respond if the relationship between customer service and positive outcomes does not bear out?
The value of not being a jerk
Ideally everyone would be treated well, and no one would act like a jerk while administering social programs.
But what if customer service made no difference in the outcomes an organization was trying to drive, and good customer service came with a cost? If you could only treat half the number of people with good customer service than with crappy service, but otherwise get the same result, which is the optimal choice?
As we move toward a data driven social sector, these are the kinds of questions we will face, especially if the evidence runs contrary to our prior beliefs. It is easy to assume there is value in providing good customer service (which there might be), but we make a lot of assumptions all the time that on face seem intuitive, all to later uncover unintended consequences.
Used properly, evaluative metrics can help guide organizations make more impactful program decisions, but the real test is how we will react when data suggests what we are doing might be wrong.