Changing the Culture of Philanthropy: a Young Donor's Views

Tuesday, December 11, at 12 noon, U.S. Eastern time

Holden Karnofsky worked for a hedge fund in Connecticut for three years and is now devoting himself to philanthropy. He founded GiveWell and its grant-making arm, the Clear Fund, and is a contributor to the

A transcript of the chat follows.

Peter Panepento (Moderator):
    Welcome to today's discussion with Holden Karnofsky, who is taking your questions on the culture of foundations, the new generation of philanthropists, the value of charity-rating services, and his own efforts to create a new type of grant maker.

You can learn more about Holden here:

http://www.philanthropy.com/free/articles/v20/i05/05000701.htm We've already received some great questions, but we would love to get more. Holden is a fast typist, so we can definitely cover a lot of ground. Click the "submit a question" link to send your question to Holden. And don't be afraid to mix it up. You're guaranteed to get a candid response. Ok. Let's get started.

Question from Edith Asibey, Asibey Consulting | Communication and Advocacy Strategies:
    I conduct evaluation trainings. A reaction I often get from both foundations and nonprofits is: evaluation/measurement is SO much work, and even after all this work, we are still only learning a limited amount of information that may not even accurately portray the complexity of our work. What would you respond? Thanks.

Holden Karnofsky:
    "valuation/measurement is SO much work, and even after all this work, we are still only learning a limited amount of information that may not even accurately portray the complexity of our work"

Sounds completely true to me ... yet doesn't diminish the need for evaluation, because without it, you learn NOTHING. The fact is that intuition is nothing like a reasonable substitute for observation - the whole history of science is the history of that fact hitting us repeatedly in the face.

This is a really good and tough question. By its nature, the question of whether to measure has to be answered without the facts at hand, meaning through intuition, analogy, etc., which is somewhat ironic and frustrating. I made a stab at it here and will do so again another time.

Question from Bob, small nonprofit:
    Would donors be more likely to support charitities that were participating in benchmarking exercises that were designed to reduce their administrative and fundraising costs?

Holden Karnofsky:
    I'm not a fundraiser and I don't have strong views on what donors are likely to do - much stronger views on what they should do. I don't think reducing administrative and fundraising costs should be a priority. At all.

Increasing overhead so you can study the problems charities are working on, and how well-suited their strategies are to address these problems, has far more potential to improve results than reducing overhead so we can spend a few extra bucks on strategies that haven't been critically evaluated.

Question from small non profit:
    I do not understand the charity effective quandry. An annual report should be available and provide direct information. Is an annual report sufficient?

Q. 2

Sme thing on grant makers. Is an annual report sufficient?

Holden Karnofsky:
    Depends what's in the annual report, but in my experience, there's almost never the info I need to assess "what you get for your dollar." For example if we're looking at a job training program, we need to know how many people are placed *sustainably* in jobs (i.e., hold the jobs for a while); we also need a sense of how many of these placements represent lives changed, i.e., how many of them likely wouldn't have been able to get similar jobs on our own. And of course, we need to know the other side too: what the charity literally did, how it spent its money (I'm not talking about silly stuff like "how much went to overhead" but instead about having a concrete sense of how they're translating donations into impact on people's lives).

Typical annual report has almost none of this info, but without it, how can you have any idea what distinguishes one job program for another, or whether any of them are making people better off than they would be otherwise?

Question from George McCully, Catalogue for Philanthropy (Massachusetts):
    The "culture of philanthropy" includes its "vocabulary, conceptualization, rhetoric, infrastructure, and modes of operation". (from our organization's mission statement). What is your critique of each of those, and how would you change them?

Holden Karnofsky:
    I'm not knowledgeable enough about the sector to confidently critique anything but the lack of transparency and public discussion. And it's this problem that stops any other problems there might be from getting addressed.

I have minor quibbles with the vocabulary and rhetoric (people often use jargon and hedge rather than going for maximum meaning and understanding) and intuitions about the infrastructure (I think charities spend too little on overhead, i.e., not enough on organizational self-evaluation), but ... I don't even know why I brought that up cause it's just a distraction. The real problem is the lack of transparency. The nonprofit sector should be a model for govt, for-profit, and everyone else by being the one place where everyone just tells the whole truth about everything. If this happened, so much else would take care of itself. That is the one thing I can confidently say is a problem.

Question from Kathi, small nonprofit:
    Is there necessarily a link between transparency and evaluation?

Holden Karnofsky:
    Unfortunately no. Lots of foundations manage to build up a huge rep for doing evaluation without sharing any of it. To me there is a necessary link between transparency and quality. Transparency is more important than evaluation, more important than anything really, if you believe in your own fallibility.

Holden Karnofsky:
    Hi everyone, I'm excited for this. One thing that is rarely understood about us is that we have low confidence in practically everything we say, particularly about a sector we have little experience with. We are always listening for someone to come along and correct us (and it happens often).

However, these are the two biggest stumpers I know of in the sector right now:

1. Does anyone actually think that "% of money that goes to programs" (above a certain threshold like 50%) is at all relevant to how good a charity is?

2. Why can't/shouldn't every foundation in the world be publishing everything that goes into its decisions, using some version of the www.givewell.net model?

Answers to these questions get 10 bonus points. Also, questions/comments about helping people (as opposed to raising money to help people) get 2 bonus points. Let's go.

Question from Patricia Smith, Earth Share of California, environmental nonprofit:
    Earth Share of California's core source of fundraising is payroll deduction via workplace giving campaigns. Our core group of donors are starting to retire. In a workplace giving campaign environment what can be done to attract donors in the age range of Mr. Karnofsky? How is payroll deduction giving viewed by Mr. Karnofsky's peers?

Patricia Smith
Executive Director
Earth Share of California


Holden Karnofsky:
    Sorry, fundraising strategy is one of the few things I don't have strong opinions about. I feel that 99% of the discussion in this sector is about how to get money, 1% about what to do with it; I'd put the appropriate balance closer to 20%/80%.

If you want to know how to get MY money (and my like-minded friends'), the answer is to make a strong substantial evidence-backed no-BS case. I give once a year, the donation itself takes very little time or effort, and further reducing that time/effort would have no effect on me.

Holden Karnofsky:
    No outstanding questions, so maybe I'll just rant a bit. One of the things that's been annoying me lately is the emphasis on "expertise," the idea that the best way to make decisions is to find someone with lots of miles on them and a good reputation and trust that person. This, in fact, is not that way any successful experts operate. A good expert is able to share all the reasoning that goes into their beliefs and recommendations, in complete detail. A good expert may have a lot of intuitive and unsystemized knowledge, but this knowledge translates into results that *can* be demonstrated (like an investor or athlete who gets great results largely on feel). If someone is claiming to be an "expert" that you should take on *faith*, without demonstration that they actually have gotten results in teh past and without explanation of their reasoning, don't listen. Why? a, because experts disagree, and I can find an expert to endorse any point of view you'd like (or any charity). b, because experience BY ITSELF does not necessarily lead to improved ability to do ANYTHING (I can give unlimited examples of this). Not everyone who puts in their time gets better, and many get worse - that's why you have to look at what an expert says and what they accomplish, rather than riding their resume.

Question from Laura, small foundation that prefers to remain anonymous:
    The foundation I work for, along with every grantmaker I have spoken with, complain about the same thing - "The grant applications we have to review are horrible. Organizations don't follow directions!" About half of these horrible applicants get their grants. Even though we continually ask for what we need from charities to evaluate their requests, are we partly to blame for the problem of horrible applications?

Holden Karnofsky:
    The story I often hear is that foundations are too stingy and narrow with how they invest their money - they want it all going to their project or their program - so the charity ends up with no ability to build organizational capacity. That could be a way in which you're to blame, I don't really know.

More importantly, I'd ask why you're asking me this question instead of asking the charities.

Question from Peter Panepento:
    You have been criticized by some because of your age and the fact that you often speak brashly. Do you see yourself toning down your approach -- or do you think an aggressive style is needed for the type of change you're pursuing?

Holden Karnofsky:
    Yeah, I'm kind of obnoxious. I mean, I try to be nice when it's important to be nice, but most of the time, I just try to be myself, because that makes it much easier to have the passion and energy I need to do something difficult and worthwhile.

Question from Coach Adam, Race wioth Purpose:
    There is little free market competition in the not for profit sector; large organizations focus primarily on defending their donor bases and acquiring new donors rather than working together to solve issues that arguably multiple organizations claim to be committed to solving. (A notable exception could be Robin Hood because of the direct support of its board.) How can we expect to be successful (as defined by having a measurable impact) when so many organizations are working in silos and concerned about another organization taking their donors?

Holden Karnofsky:
    I think the nonprofit community legitimately has good people in it - morally better people than other communities. I think we're capable of going beyond myopia about our own paychecks, and pursuing our missions instead of bottom lines (as we're supposed to, legally). For some reason that isn't happening now when it comes to sharing information, but I think it can, and if it does, people will stop asking how nonprofits can be more like for-profits and start openly wishing that for-profits had as much openness, honesty, and sharing of information as nonprofits.

Question from Cora, family foundation that prefers to remain anonymous:
    How do you recommend that foundations who have given anonymously for generations be more open without totally giving up their anonymity?

Holden Karnofsky:
    I suppose you could publish all your materials under a fake name or send them to another foundation for publishing. I have to be honest, though, I hate the idea of giving anonymously. I like almost every other kind of anonymity because it allows people to speak the truth without fear. But when we're talking about a funder, you should have no fear - just a desire to be honest about what you're finding and a public pride in the values you hold. Both of these things would make your giving better for the world; seems selfish to give them up out of shyness.

(There could be some reason for being anonymous I'm not thinking of though, so let me know.)

Question from Kristine, nonprofit:
    What if we REALLY included the people/communities that we "help" to let us know how we're doing, set benchmarks, describe our efficiency (or lack thereof) of meeting their wants/needs?

Holden Karnofsky:
    This is a really interesting issue to me because there are 2 ways of including clients in our strategy. Way 1 is to ask them what they'd prefer and how they like the programs. I think we should ALWAYS do this because it's too easy to miss something obvious by just not asking the person you're trying to help. But, clients don't always know what will make them better off. They don't, it's a fact. We don't know either. When we're tlaking about how to get someone from poor to rich, they don't know how to do that or they would have done it. So Way 2 to engage our clients is to continue the dialogue well BEYOND the point of service. As long as charities continue to lose track of clients the second they stop working with them, and don't find out what ultimatley happens to them and how their lives go, there's only so much they can truly be said to be "including" them.

Question from Jeremy Gregg, Central Dallas Ministries:
    Over the past few years, particularly in response to disasters such as Hurricane Katrina, we have witnessed a disproportionate growth in the number of health and human service organizations compared to the overall growth of sustainable funding for such work. In most other sectors, this would lead to a period of rapid consolidation through mergers and acquisitions. Do you foresee this happening in the non-profit sector? If so, what advice can you offer that will help non-profits to successfully navigate through such transitions?

Holden Karnofsky:
    Do companies really merge that often? The much, much more common way to get rid of redundancy in the private sector is through competition. People seem averse to this in the nonprofit sector, but ...

I've heard a lot of nonprofits complain that everyone wants them to merge, and understates the complexities of merging. This sound right to me. When you merge, who's in charge? How do the different philosophies and approaches confront each other? Merging seems incredibly difficult and complex to me. There are times for it, but it seems naive to say that charities should merge every time they serve similar populations in similar ways. Let a thousand flowers bloom and let people try different approaches ... as long as we eventually figure out the good ones and get rid of the others (something I don't believe happens often right now).

Peter Panepento (Moderator):
    We've reached the halfway point of what has been a very active and interesting conversation. I know Holden is looking for challenging questions -- so please keep them coming. He says he types 120 words per minute. Let's test him a bit and see if he can keep up with your questions.

Question from Laura, Chicago arts partnerships in education:
    How does your approach to assessment tackle the power dynamic between foundations and nonprofits to determine transparent assessments of nonprofits? How do you think foundations can approach assessment to give nonprofits ownership of the research, so that they are can improve their services?

Holden Karnofsky:
    Power dynamic hasn't been a big problem for us, because our grants are puny so I don't think anyone is knocking themselves out for us. If we were bigger I'd be very concerned about charities' doing things that are bad for themselves and their mission in a misguided attempt to "Give the customer what he wants." And I'd address this in the same two ways we do now:

1. Act as a funder, not a partner. Find the best, give unrestricted grants, and let them do their job. Share opinions, but don't make funding conditional on following them.

2. Discussion.clearfund.org lets applicants converse with us anonymously. Notice how much better this is than the anonymous surveys other foundations brag about. A survey has no back-and-forth; this does. Anyone got a good reason why there should be ANY foundation in existence that doesn't provide its applicants an online forum to have back-and-forth anonymous conversations?

Question from Tom Belford, The Agitator blog:
    Based on your research to date, what percentage of nonprofits would you say can really document actual outcomes (i.e. prove their strategies are working) as opposed to process milestones?

Holden Karnofsky:
    There's a spectrum. A lot of our Employment Assistance guys can show that they are placing people in jobs that they keep, which is great, but none of them have done a good analysis of the question of the counterfactual (i.e., comparison group). A lot of saving-lives-in-Africa charities can show that a particular project worked, but only Population Services International is able to give us a true bird's-eye view of EVERYTHING they're doing.

It's also important to keep in mind that a ton of charities might be willing and able to track outcomes, but the funding isn't there. I really prefer to pick on foundations rather than picking on charities. Charities do hard work and know much more about it than I do. Funders, I feel pretty confident saying they are dropping the ball on some major things (looking for things that have already worked; being transparent).

Question from Suzanne Perry, Chronicle of Philanthropy:
    I want the bonus points for asking about helping people. You just published the results of your evaluations of charities working to save lives in Africa. What are some of the things you learned while doing this research? Anything surprising?

Holden Karnofsky:
    It was hard to be surprised by anything because we knew how little we knew, going on. So, one interesting thing I've learned is that most of the diff between Africa and us - as far as mortality - is in infant health, and the rest is AIDS, TB, and maternal care. That affects my opinion of the cause - I feel more confident now that saving a life is really saving a life, not just saving someone who will die the next day. That matters. And, learning this somehow seems health parity seem a bit more achievable - there is really a key 5 years and a key 3 other issues we need to focus on. Then again, a huge question that's been raised - for the next round, I hope - is how widespread non-fatal debilitation is. Some of the numbers I've seen are just INSANE, implying that 20-50% of people in certain areas have debilitating malnutrition or elephantiasis or blindness or something. We need to learn more about this and what we can do about it. How many bonus points did I say that was worth? +2 for you

Question from Nemisis:
    You talk a lot about evaluation OTHER organizations. Do you evaluate yourselves (GiveWell)? What kind of critia do you use for this evaluation?

Holden Karnofsky:
    We are obsessed with evaluating ourselves. We have put time and expense into our website specifically so that we can track who takes action using our website (including donations) and where they come from. Ultimately, my measure for evaluating GiveWell will be (money we raise and spend + money we cause others to spend [which we track using the Donate buttons on our website]) * (difference between the best charity we can find and "average" charity, in terms of life outcomes) = our imapct on life outcomes.

Now, to be clear, we are a startup, and at htis point we need people to fund us who believe in what we're doing WITHOUT the evidence. Thus, we would not quality for a GiveWell prize. But that's OK. We aren't the only funder in the world, and we believe others (including our donors) can and should fund unproven things that might work - AS LONG AS THEY'RE ACCOMPANIED BY RIGOROUS EVALUATION TO SEE WHETHER THEY DO WORK. Our role is to find the ones who've already reached critical meass and documented ourselves, and the difficulty of doing so implies to me that other funders haven't done a very good job with this.

Question from Monica:
    Where would a philanthropist go to fund his project other then getting funds from a charitable foundation?

Holden Karnofsky:
    I wish I knew. I think your options, as a startup (rather than a brand name charity), are foundations and personal connections. But others know far more about fundraising than I do.

Question from Cora, Family foundation that prefers to remain anonymous:
    Thanks Holden - it's not that the donors of the foundation are shy, it's that they have opened themselves up in the past and they've had more development directors stalking them than Britney has paparazzi. Sometimes when charities are chasing the money they don't always think of the donor's comfort level and privacy preference. So the donors went underground.

Holden Karnofsky:
    Sorry to hear that. This might be a stupid/naive question, but why not just institute a rule that organizations making unsolicited phone calls will be disqualified or something? Sorry, just thinking off the top of my head how we would deal with this kind of problem if it really got out of hand.

Question from Peter Panepento:
    Should more foundations be willing to offer operational support with their grantmaking? And why do you think they have been so leery to do so?

Holden Karnofsky:
    All I know about why they haven't done so is what I've heard, stuff like "they want to feel directly responsible for Project X" or "They are silly." Yes, I think foundations should find great organizations and give them great funds and let them do their job. I don't think that someone sitting in an office rather than on the ground is the right person to design the programs, and I include myself in that.

Question from Sean Stannard-Stockton, Tactical Philanthropy Blog:
    Holden, it seems to me that with your recent coverage in the Chronicle, you are becoming a leading spokesperson for the concept of transparency in philanthropy. I'm all for what you're trying to achieve. However, you have maintained a style of writing and speaking that many people find to be rude, arrogant and unbecoming. Now you have every right to be yourself, but it seems to me that given your public persona as a spokesperson for transparency you are effectively teaching people that a lack of politeness and a general lack of respect for people you do not agree with go hand in hand with the concept of transparency. So my question for you is do you think that being polite and respectful are at odds with the concept of transparency and honesty? For foundations to engage in transparency, must they be prepared to drop social decorum and attack each other's ideas and criticize each other as "bad philanthropists" when they don't agree?

Holden Karnofsky:
    Sean, there is a huge difference between being blunt and being disrespectful. I believe that telling people what I honestly think is being respectful, more so than trying to baby and manipulate them. But are there cases in which I'm actually disrespectful, and say things that are personally insulting rather than just the honest expression of opinion? If so, I need to learn about them and understand them and change my ways. I appreciate any feedback you have on this. I want to do whatever is necessary to show people respect and get along with them, as long as it doesn't mean hiding my true opinions. I don't think there is any contradiction here - it can be done - though it may be something I need to work on.

Question from University:
    Are there any circumstances under which you think evaluation is NOT necessary or prudent? Could it be the case that you just learn too little for too much cost when something seems to work anyhow?

Holden Karnofsky:
    Sure, there are cases. It's hard to answer this in the abstract, of course any idea is capable of going too far. If you ask me about a specific issue, though, I'll argue for why we need to know more than we do (unless I think we don't).

Question from Coach Adam, Race with Purpose:
    For the two bonus points, people and organizations do what they are incented to do. Until organizations are judged primarily on impact (e.g., shareholder return and ROI as measure by the impact on their community), we will be focusing on navel gazing. Donor education is paramount, but then again donors are also incented to give money away primarily for tax reasons rather than due to the impact to which they are contributing. What if the IRS Code gave higher deductions for organizations that actually were making a positive difference, that would swing funds toward those organizations pretty quickly. It would also complicate the IRS code in a time when simplification is desired.

Holden Karnofsky:
    No bonus points for you - this isn't a GOOD reason. I agree with your account of why the problem exists, although I don't think the IRS is the right party to solve it. I think we can solve it together by rewarding the right things.

Question from George McCully, Catalogue for Philanthropy (Massachusetts):
    Hi Holden, thanks for taking my earlier question. Seems to me there is a second real and indubitable fundamental, problem for the sector, and that is inadequate money coming into it—this is why our focus is on programs and so little is spent on evaluation, and why that amount is not likely to increase substantially except among those with room in their budgets. Do you have any interest strategically in increasing charitable giving?

Holden Karnofsky:
    Not really. People give a ton of money right now. I think people would give even more, not less, if they had good information, see http://blog.givewell.net/?p=99

Also, trying to raise the amt of giving is something that so many people are obsessed with, people as capable as I. I don't feel I have much to add to this endeavor. By contrast, getting better about HOW WE SPEND IT has unlimited potential to improve our effectiveness and this barely seems to be being worked on at all - so that's where I feel my efforts are better allocated.

Question from Kristine again:
    What do you see re: the potentials and pitfalls of online funding/fundraising for transparency, accountability and community participation? Any interesting trends in the forseeable future?

Holden Karnofsky:
    I've never really seen the Internet as being as central to transparency and hwat we're doing as others do. If it weren't for the Internet, we'd do the same thing and publish a paper instead. It would be way slower and way worse, like everything you do with worse technology, but I don't think the Internet is what magically allows our project to exist, and it certainly isn't an excuse for the opacity of foundations to date.

I mean, the Web makes about a trillion things easier and faster. But th emost important changes I want can happen with or without it.

Question from Grant Schoenebeck, UC Berkeley:
    Say you are some huge charity like Red Cross, what do you do to evaluate your programs? How would you evaluate something like Katrina relief efforts?



Holden Karnofsky:
    Katrina is a tough one, very different from what we've been doing because it's not the kind of thing where you can create a proven, effective, scalable intervention and repeat it - you have to respond to the issues as they come up. I don't think there's much place for longitudinal studies and randomized trials in evaluating Katrina relief ... at the same time, since what you're trying to do is fairly simple, the satisfaction of clients is a much better proxy for outcomes than it is for something like employment assistance or sex education (in which case the things they say rihgt after the program may have little to do with whether they're really headed for a better future). I think one approach to disaster relief is simply to make sure you know which strategies and people are being used in each area, and survey the people in each area to see how they felt about the aid they received, and compare and contrast and try to learn things. I haven't thought about this much.

Holden Karnofsky:
    Guys, are we all together on this? The "straw ratio" (percentage of revenue spent on programs as opposed to administration) is meaningless? Charity Navigator is a farce? Are we all sick of the way the media repeats these same lies over and over again? What can we do about it? How can we break the spell that's been cast on all these reporters? Serious question. This is one of the few times I would consider joining some club or signing some petition or being in some "movement" or "protest." Ideas?

Question from Karen, small national nonprofit:
    How do you respond to yesterday's report in the Wall Street Journal on philanthropy? It seems to say "we as donors need to hold orgs more accountable by demanding reports and analysis of their success and demanding that they don't hide their numbers," but in the same breath, states "we must hold organizations to a low 'overhead and administrative' cost." The article touts organizations who publish 118 page progress reports, but damns those who spend too much on overhead. (where do they think the costs of publishing a 118 page report falls?) I think this perpetuates the tendency for organizations to overfund programs and underfund capacity and evaluation by making them feel bad about their "overhead" costs. Thoughts?

Holden Karnofsky:
    Agreed. We just need to get rid of the ridiculous notion that overhead is bad. It's an insane idea, it generally takes about 30 seconds to explain why it's insane, and all of us in this sector recognize that the notion needs to be scrapped. I agree that charities should document themselves, I disagree that they should keep overhead low.

Question from Laura, small private foundations:
    Holden, I don't think I'll get bonus points for this answer, but -"Why can't/shouldn't every foundation in the world be publishing everything that goes into its decisions, using some version of the www.givewell.net model" I love the givewell model, as much as I know about it to date. The private foundations I work with want to be totally anonymous. And at the same time be effective donors. We look at every minute financial detail of a grant applicant to determine if they're "managing themselves well" if they're a good place for their employees, if they make an impact in the community - all very general. We mine for that information. And when we get it, we keep it because we don't want anyone to know about us (the donors want to be private people). A real dilemma.

Holden Karnofsky:
    As I've said earlier, I'm still not totally clear on why you can't just send it to another foundation for publishing, under a pseudonym. And I still think it's morally better to be non-anonymous, though I understand this isn't always practically possible.

Question from Tom, community foundation:
    Did you read Journal Report in The Wall Street Journal yesterday? If so, what is your reaction and how useful to you feel it is that resources could be directed toward services or toward a 118-page "progress report" as created by the American Cancer Society?

Holden Karnofsky:
    I don't care about the number of pages, but I think we need to take the same attitude toward the world of aid that we take toward the natural world, the business world, etc. No matter how good your intentions, and no matter how much sense an idea makes in your head, the world is infinitely more complicated than you think and you don't know something's working until you've looked directly at it. In the world of aid, looking directly at your results in much harder than in other worlds, but it's still necessary.

Question from Sean Stannard-Stockton, Tactical Philanthropy Blog:
    Holden, you've stated similar answers to my previous question in the past. But the fact is, many established people in the philanthropy sector think that you are arrogant and disrespectful. You must know that.

Being truthful is not enough, if you want to have an impact you must get people to act on what you say. If you are perceived as rude or arrogant, it become very easy for established players to tune you out and claim you are not important.

Do you think it is more important to tell the truth or to get the sector to change?

Holden Karnofsky:
    I continue to deny that telling the truth and getting along with others aren't logically exclusive. I might be making them exclusive, by being a jerk. But honestly, Sean, I haven't heard much of what you're saying. When writing our press release, I was looking for people saying nasty things about us (so I could quote them) and I found a whopping two and neither of them even really dislikes us. I could easily just be completely missing a huge set of people who hate us. I am honestly not aware of people who hate us at this point, aside from the troll comment here and there. If you are aware of these people, and you can help me find out about them and what they're saying, please do.

Peter Panepento (Moderator):
    We're approaching 1 p.m. on the East coast, but we still have time to squeeze in a question or two. Send them in now while you still have time.

Question from Tom Belford, The Agitator blog:
    As opposed to a online pile of 990s, does something exist like a public clearinghouse for program evaluations?

Holden Karnofsky:
    Not that I know of, or we would have been using it. It mighit exist STRUCTURALLY (I think Phil Cubeta once pointed me to a website *inviting* people to publish their evals), but it doesn't exist as a usable resource, that I know of. In fact the program evaluations generally aren't even public, let alone not centralized. I think the first pressing need is for foundations to make their reports public, then someone can create a clearinghouse.

Question from Peter Panepento:
    Evaluation is a concept that is easy to understand and carry out for larger organizations with ample resources. But what about the vast majority of very small nonprofit groups that operate with razor-thin budgets and very small staffs? Is it possible for these smaller groups to measure their impact? If so, how?

Holden Karnofsky:
    It all depends what they're doing. There are certain activities that don't need to be evaluated (I'd put hospice care in this category, though there should at least be surveys to make sure that the people are happy with the care). When it comes to more ambitious things like trying to get people jobs or educations or save lives in another culture, facts about what works must work there way in somehow. Two ways for a small nonprofit to get them in there: 1. Do things that are already known to work, even if the way we know is through othes' evaluations. (I'm surprised at how few charities used this strategy, which we invited them to use, on our app. You do'nt need to show the literal results of your activities - just show that similar activities have worked in other times/places.) 2. If yo'ure really doing something that has no precedent AND that is trying for difficult-to-observe outcomes (i.e., you'll have no way of knowing whether it's working unless you measure, and you have no strong empirical reasons to think it'll work) ... I think you just need to suck it up and do evaluation. I honestly would rather a charity blow half its budget on evaluation than spend twice as much on things that have BOTH never been tried AND will never be tested.

Holden Karnofsky:
    To the two people from anonymous foundations: if you send your materials to us anonymously, we will publish them for you on our website in a special marked-off section, so others can learn from them without bugging you.

Holden Karnofsky:
    Elie is disappointed with my answer on evaluating relief orgs. He wants to look at how fast they get relief to people. Reasonable comment.

Holden Karnofsky:
    Elie thinks it was wrong of me to call Charity Navigator a "farce." He rightly points out that it's very useful as a search tool. That's true, we use it all the time. I also think it's useful for making sure the charity you're about to give to is a total, total fraud that isn't even trying to hide its fraudulence, although if htis is even a question, you clearly haven't thought very hard about where to give. What I think is a farce is the notion that the ratings are a good way to evaluate charities, i.e., that a 4 star charity is better in any way than a 2 star charity.

Holden Karnofsky:
    3 minutes left!!!! Hurry up, or you'll never get to ask me anything!!

Holden Karnofsky:
    P.S. If you ever want to ask me anything or send a suggestion, email me

Peter Panepento (Moderator):
    We do have one final question in the queue. Holden is vigorously typing to answer it now.

Question from University:
    A charity without an evaluation.

It seems that in helping disadvantaged youth many orginazation try to play a small part in a larger picture. For example, flying kids out to silicon valley to see the opportunities there. Give them tours of a few comapanies, etc. I don't think they think they will change these kids, but the hope is that with enough additional experiences for these kids, things will add up and make a difference. How can they be expected to evaluate this? Should they just give up.

Here, because they see themselves as a small part of a larger picture, true evaluation (for each of these parts) would cost a lot.

Holden Karnofsky:
    You just need to have some empirical reasoning for how what you're doing fits into that picture and why it's important enough to matter and why the other pieces work the way you say they do. Some people call this a Theory of Change - for example, you could show that having a positive attitude toward life leads to better results later on (using academic research - P.S., I think you'll find that it does not support this at all), and that your activities lead kids to have a more positive attitude toward life (at least a couple weeks afterward, though).

Please don't interpret me as saying that you need to DIRECTLY measure the connection between what you do and life outcomes. But the Theory of Change needs to be clearly articulated and you should be able to support each of the more questionable claims (such as the idea that a positive attitude leads to better academic success - a claim that I for one would dispute) empirically.

Holden Karnofsky:
    It's very possible to make a strong, concrete argument for an intervention without any direct measurement. But having that *empirical* component - data from somewhere, not just the logic that "it must work" (because there's really almost nothing that "must work" in problems as difficult as the achievement gap etc.) is essential.

Question from Jennifer-Nonprofit in Colorado:
    Do plan to invest some of your money to help nonprofits fund evaluation?

Holden Karnofsky:
    So far, we're so tiny that we can't do this. Our strategy instead is to be part of the incentive to do evaluation. But would we like to? Yes, absolutely. If we had more money to grant, I would like to give a certain size "prize" to the best proven/effective/scalable method of helping people we can find, then spend the rest funding evaluations of things we think are promising but unproven. That'd be nice.

Peter Panepento (Moderator):
    Thanks to everyone who submitted questions today for Holden Karnofsky. And a special thanks to Holden for answering so many of your queries (and adding a few editorial comments of his own). We'll be back after the first of the year every Tuesday at Noon Eastern time to discuss important issues in philanthropy. Our next scheduled chat is Tuesday, January 8, when a panel of experts will answer your questions about what your charity or foundation should expect in 2008.