April 29, 2009
In Assessing Charities, Are Stories About Beneficiaries Ever Useful?
To educate donors, charities often share stories about people served by their organizations. But Holden Karnofsky writes on the GiveWell blog that those stories rarely provide much insight into whether or not the organizations are doing good work.
“Charities share a small number of stories without being clear about how these stories were selected, which implies to me that charities select the best and most favorable stories they could be telling,” says Mr. Karnofsky.
One “semi-exception,” he says, is the group Interplast’s blog, which “has such a constant flow of stories that I feel it has assisted my understanding of Interplast’s activities.”
Mr. Karnofsky asks: Why aren’t there more charity blogs like this one?
“A charity that was clear, systematic, and transparent before the fact about which videos, pictures, and stories it intended to capture (or that simply posted so many of them so as to partly alleviate concerns about selection) would likely be providing meaningful evidence,” he says. “If I could (virtually) look at five random clients and see their lives following the same pattern as the carefully selected ‘success stories’ I hear, I’d be quite impressed.”
In a reply, Tony Pipa, a nonprofit consultant, says that narrative is an effective way for charities to share information. But he agrees that most nonprofit groups today use narrative purely to market themselves.
“Love your suggestion to combine 100 unedited reviews, and would like to see a foundation take it and fund a pilot of a group of grantees to carry it out, to test its value,” writes Mr. Pipa.
What do you think?
Comments
Commenting is closed for this article.
Previous: Google.org Helps Track Swine Flu
Next: Learning From Career Mistakes
My comment to the Givewell blog post:
From an anthropologist: There is a time and a place for qualitative data. I don’t blame a nonprofit for using stories to illustrate their successes in their marketing materials – why shouldn’t they? And I don’t think it would make much sense to do open-ended interviews with all the clients of a nonprofit (unless it’s very small) – imagine the nightmare of making sense of the data!
That said, I think qualitative and quantitative data should supplement each other more frequently than it does. It would be a very good idea for Givewell to check out its top choices on the ground. Just because an RCT has found a program effective in the past doesn’t mean it is still implemented with fidelity to the effective model…
Qualitative interviews make most sense with a relatively small number of key people in the exploratory phases of an evaluation – say in the case of a process evaluation where the goal is to understand the range of issues volunteers are faced with. But the next step, if the purpose is to understand what’s going on on a larger scale, should be to create structured interviews to determine how widespread the issues that emerged are.
Qualitative exploration also makes sense if quantitative data is mysterious in some way – it’s a great way to begin to find out what may lie behind the numbers.
In short: No numbers without stories; no stories without numbers.
— Ingvild Bjornvold Apr 30, 10:15 AM #
To give fuller context, thought I’d also copy my entire comment to the GiveWell post:
To my mind, narrative is a much better metaphor for capturing the intrinsic value of nonprofits than the quant metaphors of the business world that so many (including GiveWell) seem obsessively determined to apply to nonprofits. However, as you point out, narrative as used by almost all nonprofits now is basically marketing, because there is subjective bias in which stories are told and what of that story is presented. Love your suggestion to combine 100 unedited interviews, and would like to see a foundation take it and fund a pilot of a group of grantees to carry it out, to test its value. I’d also love to see much more discussion and more suggestions of how to make narrative meaningful to measurement/learning by those who are developing models to measure effectiveness.
— Tony Pipa Apr 30, 01:17 PM #
This feels like a somewhat theoretical debate about nonprofit storytelling and desire for transparency. There is a difference between story collection for marketing an organization to individual donors and supporters and story telling to report back on outcomes to funders. But ultimately, it is about what the organization is trying to achieve via the storytelling.
At the end of the day, when the rubber meets the road, nonprofits need stories to make the issues they address, and the outcomes of that work, emotional, relevant and real.
Donors and advocates want to impact people. Stories should be used to inspire consideration AND numbers, results and fiscal management should always be top donor considerations.
I, too, like to hear the “unsuccessful” stories because the best learnings come out of overcoming challenges. When we get it wrong we work harder to learn and do better. I respect an organization that is continuously trying to improve and demonstrates the need to do so.
But let’s not put nonprofits under the burden of having to tell all stories, good and not so good. It is hard enough for them to dedicate resources to telling any stories at all. If they are to be relevant to prospective donors and supporters, they need to bring their work to life in a way that is inspiring, attention grabbing, and relevant.
Risa Sherman
http://cause-nation.com
— Risa Sherman May 1, 01:47 PM #
My organization Keystone Accountability is dedicated to improving the quality of evidence available to understand the difference organizations make when they set out to improve the world in some way. We specialize in cultivating feedback from those who are meant to benefit. From this vantage point, I can briefly share some lessons from our work.
First, the debate has moved beyond qualitative versus quantitative. Those with an interest in these themes may want to look into the current debates in the field of impact evaluation. One articulation of the bleeding edge there is summarized in a 2-page call for action at http://www.keystoneaccountability.org/node/242.
Second, intensive field observation by third-party analysts (as valuable as it is) just won’t scale in a way that will allow us to meet the system-level challenge to get all organizations to put better outcome and impact information into the public domain. We need open review systems like the one that GreatNonprofits operates as this will increase the volume of independent opinions about performance. But we need more than this as well.
Third, and to my mind most importantly, we need feedback data to be collected and aggregated in a way that creates scientifically valid comparisons across organizations. The Center for Effective Philanthropy’s (CEP) Grantee Perception Report™ has done this at the level of the performance of grantee feedback to foundations. Keystone is now working with CEP, a number of leading foundations, human services organizations, and charity rating websites to do this at the level of the primary constituents of social change — those meant to benefit. We need to create a new information infrastructure for comparative feedback in order to do this at scale, perhaps something along the lines of what the customer satisfaction industry has done in the commercial context.
Fourth and finally, there is a very simple thing that we can all do in order to make a giant step forward in this space. We can all adopt and insist upon the feedback principle of public reporting by organizations seeking to create social and environmental benefits. The feedback principle asks organizations to report their results in whatever way they choose, but in addition they should report what those meant to benefit have to say about those alleged results. This feedback would provide a kind of validation to the purported results, and would need to be undertaken in a methodologically valid way. The method for this kind of constituency validation of results is not the hard part here. The challenge is to gain a sufficient appreciation of the necessity of this kind of system upgrade to our theory of public reporting.
— David Bonbright May 3, 10:29 AM #
There is a lot of wishful thinking in these comments, suggesting that non profits have the resources to be constantly collecting and sharing the types of information being suggested.
I think stories are important. I come from an advertising background where stories told daily to potential customers resulted in people coming to our stores to buy our merchandise.
I use blogs to tell stories at http://cabriniblog.blogspot.com and encourage other tutor/mentor programs in Chicago, and other cities, to do the same.
However, I recognize the limited ability of my own organization, and others, to capture, write, and post these stories regularly, so I also reach out to universities, high school, businesses and others who could be adopting non profits, and serving as their “story tellers”. If third party groups took this role, maybe we’d have more and better stories, and maybe we’d collect more of the other types of evidence, too.
Maybe we’d even draw more operating dollars to the programs in the field trying to provide effective services on an inconsistent flow of resources.
— Dan Bassill May 3, 12:48 PM #