What's the Solution to the Evaluation Quandary

Tuesday, April 24, at 12 noon, U.S. Eastern time

Suzanne Callahan founded Callahan Consulting for the Arts and the Laboratory for Arts Evaluation, Washington organizations that help

A transcript of the chat follows.

Stacy Palmer (Moderator):
    Hello. Welcome to The Chronicle of Philanthropy's online discussion with Suzanne Callahan, an expert on evaluation. She will be taking your questions and comments throughout the hour. I'm Stacy Palmer, editor of The Chronicle, and we are delighted that so many of you are interested in talking about this topic. To submit a question, just click on the button on this page. Suzanne, thank you for joining us.

Question from Lesa Lessard Pearson, nonprofit consultant, EMBA candidate:
    The mission, vision and goals of nonprofits are not usually exactly the same. It is critical that both funders and fundees understand what bottom line success looks like for themselves and for the other or risk setting up unrealistic expectations from the get-go. The cornerstone of any solid partnership requires a degree of trust and honesty — good negotiating skills are also a big plus. Would you agree that far too often, the focus is on the money (both from the giving and receiving end) rather than the actual effectiveness of the partnership? Would you agree that a healthy step would be to pre-evaluate the goals and expectations of the partnership both from a process and progress point of view before any money is exchanged?

Suzanne Callahan:
    In an ideal world, yes. This is why I practice and believe in the participatory approach to evaluation, which requires stakeholders to sit down at the same table and discuss what success means to them before a program begins, rather than relegate evaluation to the end, when a final report for a grant is due. Terms such as "effective" and "useful", which are typical "measures" of a program's success, have completely different meaning, depending on the project and its context.

To your list of trust and honesty, I would add flexibility, meaning that that all stakeholders for a project may need to compromise in order to complete the project. Here is an example of all three attributes, drawing from the arts organizations in my book, Singing Our Praises, commissioned by the Association of Performing Arts Presenters. A presenter [the term that describes the theaters where you go to see performances] works with a wide roster of artists and audiences each season. Arts presenters reflect on their own shared values before they plan their seasons, and must implicitly trust the creative process of the artists they select — even when they don't know the "outcome" because the art/performance itself hasn't even been created yet!

And, partnership happens literally every day — in making decisions about set design, costumes, brochure costs, music, and much more, not to mention in the rehearsal process itself. The creative problem solving — in the form of trust, honesty and flexibility — that happens every day in a theater would be great training for Fortune 500 execs. And, as the firm Arts Action Research reminds us, the arts field's track record of on-time deliverables is stellar: have you ever been to a theater when the performance did not happen within 15 minutes of when it was supposed to? That happens due to the reason you suggest: solid partnerships.

Question from Dan Moore, GuideStar:
    Should evaluations be made public? By the foundation? By the nonprofit? What issues are implicated? Privacy, legal concerns about privy of contracts, potential for harm to reputation? Would this be a good thing or a bad thing — if evaluations became public.

Suzanne Callahan:
    Most of the evaluations I've worked on are public. In reporting, public or otherwise, there are ethical issues that should be followed, which are on the Web site of the American Evaluation Association. It sounds like what you are asking about is negative evaluation findings. Most of the organizations I've worked with have been comfortable revealing their own shortcomings, provided that a) it's stated in a constructive and respectful way, rather than a "gotcha" exercise and b) they have the opportunity to review and comment on those negative findings, and state how they would address their shortcomings. The participatory approach, which is described above by Lesa Pearson, allows for this. What we sometimes do is create two versions of the report, an internal one that goes into more detail in a manner that serves staff and board, and a shorter one that is easier understood by an external audience.

What's key is discussing with the organization ahead of time — before the evaluation is conducted — how the information will be used, and what they would want done with any negative findings. In both of the case studies in my book, there were less-than-perfect findings that were intentionally included (thanks to the leadership of the arts organizations involved). Without negative findings, we'd have reams of paper that revealed nothing about how to improve programs, which would limit the usefulness of evaluation.

Question from Barbara Levenson, religious nonprofit:
    How do you identify and take into account outcomes that seem to be a result of the program but that would have happened anyway?

Suzanne Callahan:
     Michael Quinn Patton, a leader in the evaluation field, claims that evaluation is, in many ways, an art form, and I agree.

As your question suggests, we have to be our own devil's advocates in designing our evaluations, in order to be sure we don't overstate the impact of our programs. We have to ask the tough questions that you are posing — questions that can't always be fully, or simply answered.

In your scenario, I would also ask: How do you know the outcomes would have happened anyway? Is the fact that they'd happen anyway attributable at all to the prior work of the organization that is claiming to [now] cause those outcomes? Sometimes outcomes take longer to achieve than we wish, and attaining them now may be the result of earlier good work. This is one of the reasons why evaluators tend to distinguish short-term from long-term outcomes.

Conversely, in your scenario, can you definitely attribute the attainment of your outcomes to other circumstances, outside of the program itself? Our role as evaluators are to state all the variables that might influence a program and its outcomes, and, to the extent possible, determine what came about as a result of our program. But that's much easier said than done. As Mr. Patton says, "context is everything," meaning that we are charged to take into account the external environment in which a program is operating.

The questions you raise are the kinds that evaluators debate all the time. And, since humans live not in a laboratory, but in a world full of influences that are acting on any situation at any given moment, we can’t always be sure that our program alone caused the kinds of change we want to claim. Because our programs aren't conducted in test tubes, we often can’t isolate them from each and every influence — nor would we want to if those influences cause positive results.

Katherine Newcomer of George Washington University gave me great advice, years ago: When conducting an evaluation, simply say what you did. In your instance, say what you think your outcomes were, what the outside influences might be, how you controlled for them, what you found out, what you think know, and what still might be open to question. Getting advice from someone who is trained in evaluation might help. Being honest and transparent about what you did in your evaluation and what you found out is the best way to handle ambiguity.

Your query raises a good point that may apply to other questions asked today: I can give you my take, but you may get a different answer if you ask a different evaluator. We don't always agree, often because the answers aren't cut and dry.

Question from Phyllis Jones, The National AIDS Fund:
    This is not a question as much as it is a comment. Your article is right on the money (both figuratively and literally). Evaluation is expensive and time consuming, therefore if funders want grantees to engage good evaluators, develop sound evaluation tools, be trained to gather valid data and receive good data analysis, then adequate funds need to be earmarked for those activities, separate from dollars to support program and delivery of services.

Also evaluation needs to be frontloaded into programs. The evaluation design ideally should closely follow program design and evaluation tools should be ready at the start of implementation. It is far too common that evaluation is an afterthought or something that is initiated after a project is launched and has had time to "get the quirks worked out".

What funders and practitioners don't realize is that they could really benefit from lessons learned if an evaluation process is in place and ongoing, as they "work on the quirks". Unintended outcomes are what truly inform programs and give rise to innovation and modification, so that practitioners can adapt with programmatic changes to improve outcomes.

We have to stop looking at evaluation results as success or failure and shift to looking through the lens of intended and unintended outcomes. Only then can we give practitioners the ability to critically look at their programs, in real time, and make necessary adjustments without the fear of losing funding because they didn't "stick to the course" as outlined in their proposal.

So now the questions:

How do you see the field of philanthropy changing to incorporate deliberate, thoughtful, and informative evaluation as an essential component to grantmaking, especially taking into account how costly it can be in comparison to the dollars they are giving to support provision of services?

Secondly, how do we change the culture of agencies to stop fearing evaluation and look at it as a technical assistance tool rather than a tool to measure program failure or success?

Thank you for your article.

Suzanne Callahan:
    I think funders realize that they are not always getting what they want from evaluations, but aren't yet quite sure what to do in response. It's a big, complex tug of war and well-intentioned funders are caught in the middle. I don't see only the field of philanthropy needing to change. Nonprofits need to change, too. They need to increase their understanding of, and willingness to take on, evaluation as a core value, to make it an ongoing part of operations, rather than to view it only as a funding requirement.

The first step, for all who are holding any part of the tug-of-war "rope" is to drop the rope and talk honestly and openly about how change is articulated and measured. Many of the folks who have the insight to do that are the nonprofits themselves. But in order for grantees to feel safe being honest, funders must grant them some degree of immunity from being "punished" for less-than-glowing results. Like so many things, it all starts with honest and open communication. What funders can do is to convey to grantees their openness to hearing the full truth, without immediately cutting funding.

More directly to your second question, in my experience as a consultant to funders and nonprofits, maintaining a relationship of trust and open communication with clients, in which they view us as their ally, is absolutely key, even when the information we gather isn't altogether positive. That trust gets established way before the data is collected. Compare it to a visit to the doctor: when most of us make the decision to go to a doctor, we want to know the truth about what's wrong so we can do what we can to heal. It doesn't mean it's an easy message for the doctor to deliver, and the message might mean we have to change our diet and start exercising. But when the doctor is on our side, and we are working as a team, it's a far more pleasant process that's likely to generate positive results. Even if we don't do anything about it (which, as I've learned from doctors through the Society for Arts in Healthcare, is their biggest problem!)doctors owe us an honest assessment, rather than to tell us what we want to hear. In evaluation, we have to be open and ready to hear the truth — and willing to act on it.

Question from Simeon C. Goodwin, Ph.D., The Family Place, Dallas, TX:
    Regarding program outcomes, how common is it for non-profits to use confidence intervals in reporting their findings? Also, do you know if these programs use anything other than a 95% or 99% confidence interval?

Suzanne Callahan:
    It's not all that common, unfortunately. In the nonprofit field, there's a gap in knowledge about sampling procedures in general as well as statistics. This is a an area where a bit of outside assistance (which might even come from a pro-bono statistician or a professor from a local university) would be of help in increasing the validity of evaluations.

The most common error I see is that organizations rely on sample sizes that are far too small to draw conclusions. It's not that nonprofits or funders want to deceive anyone, it's just that they don't understand.

Question from Lisa Walker, IGI LLC:
    With an understanding that flexibility and a balance of quantitative and qualitative reporting provide optimum results, I am wondering what common evaluation standards and reporting formats exist for particular giving areas so that less of an administrative burden is placed on not-for-profit organizations and cumulative progress and gaps can be identified for specific giving areas.

Suzanne Callahan:
    There is no single standard, or set of standards, for any giving area, partly because funders themselves have different missions and priorities. But there have been, in some fields, broad guidelines for both applying for and evaluating grants that can be aggregated and tracked over time. The NEA uses a common application and final reporting forms in their granting, which is accompanied by project-specific reporting requirements, and I've integrated their ideas into my own practice. Generally speaking, common reporting and guidelines make a lot of sense but can meet with mixed results; they allow for aggregation of information, they ease the review process because funders and review panels are familiar with the forms, and can even make it easier for the grantee, who only has to fill out one set of paper. But they may or may not allow much room for individual interpretation, depending on the funder and their own internal review process.

Question from Jeannette Schultz, National Endowment for Financial Education:
    How do you convince upper management that evaluation can work when they believe that this component can make them look bad?

Suzanne Callahan:
    That's a tough one. Sometimes, ironically, the "conversion" of management comes when a funder requires an evaluation, which they later find to be useful to their organization. Often evaluations reveal positive program results in ways in which management is not aware. Also, the issue of trust is key; to the degree possible, skeptical management must trust the evaluator and the process. What helps immensely is gathering information that is clearly viewed as useful to management, rather than merely required by an outside source.

Question from Sal Alaimo, PhD student-IUPUI / Consultant:
    If the funding world is truly interested in adequate evaluation that will be used to improve programs, why isn't there more support for evaluation capacity building (ECB) that is utilization-focused? Evaluation needs are driven at the program level, so ECB would be based on what questions about the programs you're trying to answer. It seems to me that funders can play an important role in ECB and use evaluation as an important part of their "investment" in programs, organizations and communities as they often like to say.

Suzanne Callahan:
    I can only agree, wholeheartedly! For folks who might be new to the term, utilization-focused evaluation is highlighted in one of Michael Patton's books by the same name. Its mantra is "intended uses by intended users," meaning that evaluations should be designed and [according to Patton] "judged by their utility and actual use, or how real people in the real world apply evaluation findings and experience the evaluation process."

I can't tell you why there isn't more support for capacity building but I can only agree with you that there should be, both for funders and nonprofits. And, a number of funders, such as the Wallace Foundation, Doris Duke Charitable Foundation, and the Chicago Community Trust, Dance/USA, the Association of Performing Arts Presenters, the National Performance Network, and Creative Capital, have invested in forming cohorts of their grantees, who get together regularly as a requirement of the receiving funding. Grantees get training, learn about each others projects and discuss how programs are going. This is a great model on many different levels.

Question from Bill Harris, Facilitated Systems:
    In reading your article, you seem to be encouraging both sides to be flexible in seeing what actually happens, not just focusing on what they planned. Is that correct? If so, there seems to be another level: figuring out how to look with open eyes for unplanned results. Is that correct? Is there any science to that, or is it all art?

Suzanne Callahan:
    What you imply is correct: the best attribute is a clear, objective set of eyes and ears, which come from the stakeholders involved in a project — be they funders, nonprofits, client/members, etc. But there is some science, too: Patton suggests a list of outcome categories that can guide our thinking and frame the ways in which our programs measure change. It has helped me out immensely and can be adapted to any field.

Question from David Devlin-Foltz, Aspen Institute:
    Your call for setting more reasonable expectations seems especially apt in the case of evaluating policy advocacy efforts. Our planning and evaluation guides for foundations and grantseekers (at ContinuousProgress.org) urge them to stop trying to claim sole credit for bringing about a change. The tools instead help groups try to document their contribution to a favorable outcome, and urge foundations to accept that as sufficient. Have your clients -- both foundations and NGOs - been willing to call "contribution" a success, for evaluation purposes?

Suzanne Callahan:
    David, credit goes to you and Aspen Institute for developing tools that encourage a more reasonable approach. Aspen's tools, which effectively combine qualitative and quantitative data, were influential in my own thinking about evaluation, many years ago when I worked on their Self-Employment Learning Project. Changed policy is an example of a long-term outcome, which can be based on years, even decades, of advocacy and hard work on the part of many organizations. As for my own experience, sadly, the answer to your question is no — funders tend to want guarantees of the results of their funding.

Here's the two reasons why folks try to claim sole responsibility for achieving outcomes: 1) because they've been strongly encouraged to do so in order to get funding, and 2) they don't understand enough about evaluation, versus research, to realize that they are overstating their own impact.

Whether it's stated outright in funding guidelines that require grantees to have measurable outcomes, or whether it's implied in descriptions of past grants that culminated in grandiose results, grantees are often just plain scared that if they don't make a bold claim, they won't get the grant. Whether consciously or unconsciously, when the philanthropic system in our country encourages big results, promises money for those results, and won't accept a contribution toward an outcome as being enough, it tempts folks to overstate their success. Anything that Aspen or others out there can do to encourage us all to take a reality check would be helpful.

We can learn from the science field in this regard: I doubt if the NSF or NIH insists that their researchers must find a cure to lymphoma in order to be funded to conduct an experiment about the disease. These agencies invest in studies, and abandon efforts, all the time, in order to contribute to the greater good of advancing knowledge and ultimately curing disease.

The dialogue about short, versus long-term outcomes needs to go a lot further. A few months back, Kenard Wing had a great piece in the Chronicle about this topic.

Stacy Palmer (Moderator):
    We're getting lots of great issues to discuss, but just a reminder that if you would like to submit a question, hit the "ask a question" link on this page.

Question from Ian Wilhelm, Chronicle of Philanthropy:
    Suzanne, if foundations were to lower evaluation expectations, do you think they would provide more grants for nonprofit operatings costs? Foundation leaders tell me that they understand the need for paying for such overhead and administrative expenses for their grantees, but that they prefer to provide program support because it's easier to measure its results. Thanks.

Suzanne Callahan:
    I certainly hope they would. It's sad to think that grantees have to prove themselves again every funding cycle, in order to attain funds. These days, I consider gen-op to be leadership funding: it is rare for funders to take the time to get to know an organization so well, and trust the nonprofit so implicitly to know how to spend money. There are those funders who bend over backwards to get to know their grantees and programs, and they are to be commended. But, it's not that easy for funders, either; remember that they often have to report to their own boards about results, and reporting the results of gen-op is difficult, if not impossible, to do. Thus the tug of war I described in my Opinion piece.

Question from Bronwyn Mauldin, Consultant:
    It's been my experience that when government or foundation funders set the performance outcomes, many nonprofits either see it as an excuse not to spend time setting their own outcomes that are different from those, or they think they can't. Has this been your experience? How can we help nonprofits realize that they can and should set their own goals, data collection systems, etc.?

Suzanne Callahan:
    Whether intentional or not, the outcomes set by funders become the carrot that is dangled in front of nonprofits. While it's only fair that funders tell nonprofits what they are looking for, it becomes tempting for nonprofits to try to deliver those outcomes, not out of laziness, but because they want the funding. That's been my experience. My work, and writing attempts to get nonprofits involved in the process, and come up with data collection systems that are doable. But that doesn't always mean they are easy, nor quick. There are many evaluators who feel the same way. Most of us see ourselves as allies in the process.

Question from Beverly Parsons, InSites:
    In response to Sal's question, you mentioned funders that bring grantees together. Do you know of any situations where grantees work together to develop their own evaluation that spans their multiple sites? Do you know of situations where this is part of developing their own evaluation capacity?

Suzanne Callahan:
    Wow, wouldn't more of that be great? The only one I know of was through the Chicago Community Trust, in their Dance Initiative which was active several years ago. They brought grantees around the same table and really listened to what was said, and made funding decisions that were based in large part on that information.

Question from Paul Frankel, American Humane:
    Hi! How best can we discuss the issue of "dosage" with donors and funders? In other words, how can we explain that a million dollar afterschool program may not significantly change the lives of vulnerable kids in a very short time (or ever)? Thanks!

Suzanne Callahan:
    This is one of the biggest problems that evaluators face, as we are the folks who are supposed to "prove" results. You said it yourself: we have to start talking about it, and not promising to change the world. And funders have to better understand these limitations. As for where and when those discussions take place, it depends on the field, but an annual conference might be a good place to start, as is the media. And a big thanks to the Chronicle for letting us begin to talk about this!

Question from non profit:
    How exactly do you think foundations should lower their expectations in evaluations results? Why would it be positive for them? Can you give an example for an education project other than the arts ? This may lead to points of negotiation with foundations.

Suzanne Callahan:
    By not expecting nonprofits to change the world with a modest grant. By giving more financial support for conducting evaluations. The best example from education is the debate about No Child Left Behind and the over-reliance on testing.

Question from Janet Curtis, Consultant:
    How do you handle evaluation requirements when the project is really about changing the hearts and minds of a particular constituency, such as immigration reform or climate change — issues that will outlast the grant period?

Suzanne Callahan:
    This is true in much of the work that nonprofits do. We are, after all, contributing to the greater good for society, in a wide range of ways. In an earlier post, I had commented on the difference between short- and long-term outcomes, and the former leads to the latter, but sometimes not for decades. Mapping out those connections, as nonprofits view them, is key.

Question from Laurel Burnham MA Children's Trust Fund:
    The struggle for us is finding grants that will fund the evaluations we would like to do. Many grants require evaluations without providing the necessary funding to do said evaluation. There is a strong desire for powerful results; however, we have a difficult time finding anyone who wants to pay for the often costly evaluation required to get those results?

Suzanne Callahan:
    I can't agree with you more. One of the points my piece made was that funders need to support the real costs of evaluation.

Question from Karen Strong, Prison Fellowship:
    We work with an initially controlled but later highly transient population — people in prison and post-prison. Impact evaluation is complex and costly due to the correctional approval needed for studies in prison and with parolees, and also due to the difficulty of keep track of a cohort for a reasonable study period after leaving prison. What suggestions do you have for how an organization like ours can find the financial resources AND the professional expertise to undertake credible impact evaluations? We've done some studies in the past, but at this time precious resources are directed at our actual services and not at funding evaluators. But we do need the evaluations!

Suzanne Callahan:
    You sum up a big dilemma — one that faces most fields — implementation versus learning how to implement better. One of the key principles of participatory evaluation is focusing on one, or several, aspects of a program rather than evaluating the whole thing (which most organizations could never afford to do). And there's a limit to how many longitudinal studies we can do, due to cost. So I would pick carefully, selecting one or several programs and the most cost effective ways, may be the best you can do.

Question from Jennifer Clark, Economic Opportunity Studies:
    I have a follow-up comment to the comments made by Bronwyn Mauldin and by Suzanne. Yes, funders frequently do set the performance outcomes. WE should put a heavy emphasis on "performance," because the message from funders often seems to be: How much improved well-being can the program produce? Not infrequently, program providers are expected to predict their "performance" in their grant application. In my experience, nonprofits are too busy trying to make a reasonable prediction (meaning: set a target that is high enough to be impressive but not too high to be unattainable), hoping they can hit it if they get the grant, fretting if they do get the grant and it seems as if they are going to miss the target.... Who in the program has any time to think about the value for the program of setting its own goals?

I agree with Suzanne that it is fair for funders to tell nonprofits what they are looking for, but when they say that they are looking for the maximum number of X (whatever result is desired) in return for their dollars — even if setting a target and trying to hit it is not at all a rational way to measure a program's impact — then program providers are "stuck" with that approach to evaluation, and there is no reward for them (or for their evaluators!) for thinking in other terms.

Suzanne Callahan:
    

Question from small non profit:
    what steps do you recommend a non-profit to do in order to learn more about how to evaluate programs well and regularly?

Suzanne Callahan:
    1) get some training. 2) access funding for it if at all possible. 3) consider working with outside help, until you are at a point when you feel you understand the principles well enough to take on the work yourself. 4) take advantage of free methods, such as regular staff and member meetings. You'd be amazed how rarely nonprofits just ask the folks in a program how well they think things went. You can do this informally, and often. Start by just asking!

Question from Esther Landau, The Walden School:
    We use the simplest of reporting mechanisms with our funders, as we are still fairly small potatoes. However, having just received an NEA grant (for which I must attend an evaluation training session next month) I would like to get caught up quickly on the more sophisticated terminology I see popping up in this discussion today (What the heck is a "confidence interval"?) Can you point me toward a book for beginners, perhaps an "Evaluation for Dummies"?

Suzanne Callahan:
    Okay, apologies for self-promotion, but the sidebars in my book and the chapter entitled Try this at Home are designed just for that purpose. And, there are other great sources. Visit the Kellogg Foundation Web site, and download their free evaluation guide, as well as the University of Wisconsin materials, and those by Innovation Network. Most evaluators and a number of funders are out there, trying to help you!

Question from Terry Plain, Missouri Foundation for Health:
    Hello,As a foundation professional, a part of my work is developing funding programs based on existing/emerging health needs in our community. We want to evaluate the effectiveness of our overall endeavors but are not sure when to bring in an external evaluator that can advise us on overall goals and measure of progress. When would you recommend grant makers utilize external evaluators?

Suzanne Callahan:
    When you feel that a) there are major implications for the decisions you will be making, in terms of your programs' cost and/or design, and b) you lack the internal skills or resources to feel confident about the manner in which you are making those decisions. You might start by contacting several consultants and asking them out they would work with your staff and program.

Comment from Lisa Walker, IGI LLC:
    In response to Beverly Parsons' question. I know that Altria (where I used to work) conducted a series of measurement and evaluation pilots in the giving areas of nutrition for people with HIV/AIDS, domestic violence prevention and dance. A sample of grantees were given evaluation grants and technical assistance and Altria program managers participated in the program which was conducted in partnership with the Rensselaerville Institute (http://www.rinstitute.org/) during a two-year period.

The program was a successful small step in organizations working with peers in their fields and a funder to evaluate impact as individual organizations and to develop shared indicators for the field. I was wondering where else that may be happening and if there is a forum to share learnings from these types of initiatives that are happening in different giving areas and locations around the country.

Question from Bill Harris, Facilitated Systems:
    Phyllis asks an interesting question. As she poses it, programs design the program and may regard the evaluation component as external to the system that is the program. Yet many common views of systems include the feedback, often as a key, intrinsic component of the system (think of the cruise control on your car). Are there ways to help programs understand that designing effective programs can (and, arguably, should) means designing both the delivery of services and the feedback (evaluation) that monitors, controls, and seeks to improve that delivery? What have you seen that works in practice?

Suzanne Callahan:
    If I understand your question, you are talking about a process evaluation, as opposed to an impact evaluation. Most of the ones we design are a hybrid of the two. Process evaluations ask questions about how program management and delivery works. Though there's no formula [to any of this] process questions can be incorporated into evaluations, and information can be culled from line managers, other staff and/or program participants. Sometimes, these are the instances in which staff will buy into an evaluation as they realize their jobs might get easier.

Question from From a private-sector trained evaluator:
    Which non-profit agencies or foundations have worked to integrate common private-sector metrics into non-profit evaluations...an example would be the "Balance Scorecard."

Suzanne Callahan:
    Balanced Scorecard is the one I see most often. The British Council, a government agency of the UK, found ways to incorporate what they need to know into questions we asked during an arts residency at the Walker Art Center (MN) and the Wexner Center for the Arts (OH). And, based on what audiences said, the DC office of the British Council gave feedback on rewording some of their questions to their sister agencies in other countries. As for non-arts agencies, I am not sure of the specifics, but hear this discussed by evaluators in other fields regularly. Ask the AEA for specific examples.

Question from Ruth Bowman, Ph.D.; Anne Ray Charitable Trust:
    Are there leaders in the international arena of grant makers or nonprofits that offer constructive insight or examples of evaluation planning and reporting that is notably utilization focused?

Suzanne Callahan:
    Ruth, that's a great question, and I'd like to refer you to the American Evaluation association at www.eval.org. They cohost an international conference each year and would know the answer.

Stacy Palmer (Moderator):
    I'm afraid that we have reached the end of the hour. Suzanne, thank you for all the great information and comments and thanks to all of our readers for the outstanding questions. If you have any additional suggestions for the editorial staff at The Chronicle or comments for Suzanne Callahan, please send an e-mail message to editor@philanthropy.com. We hope all of you will join us for future online conversations.

Suzanne Callahan:
    Thanks for having me Stacy. I am only thrilled that there are so many questions, suggesting that a dialogue has been sparked.