Measuring What Counts: Meaningful Evaluation for Family Foundations (November 2011 FGN)


By Anne Mackinnon

Editor’s note: This month’s feature story is an excerpt from the new edition of our Passages issue paper series. A full copy of this issue paper, complete with specific tips, tools and interviews, is now available for members of our Friends of the Family Program and subscribers to the Family Philanthropy Online Knowledge Center.

Family foundations are in business to make a difference. As one family foundation leader put it, creating a family foundation is a “powerful statement about wanting to achieve impact.” Yet family foundations often get painted unfairly as having little impact, perhaps because they aren’t always very good at understanding or describing the impact they have, even to themselves.

Much has been written about how foundations can better understand their impact. But little of the published work deals with the particular characteristics of family foundations. Philanthropic families tend to take a more personal approach to grantmaking. This approach reflects both their passions for causes and organizations, as well as the donor’s intent. Family trustees often have close relationships with—and knowledge of—the work of their grantees. For that reason, many family foundations take a more informal approach in their evaluation efforts.

The use of formal evaluations to measure impact varies widely among family foundations, but a growing number employ some form of assessment strategy. Some family funders find the evaluation process useful for articulating how they want to make a difference and what difference they believe their grantmaking will produce—in other words, what, in their view, is worth measuring.

Unfortunately, formal evaluation can be very expensive. Many family foundations believe they lack the resources to evaluate the work they support. A question for all funders is: how do we assess impact in cost-effective ways? If formal evaluation is not the answer, or not the whole answer, what are our alternatives?

______________________

A question for all funders is: how do we assess impact in cost-effective ways? If formal evaluation is not the answer, or not the whole answer, what are our alternatives?

______________________

To improve communities, foundations must ask themselves what impact they are having. Those featured in the National Center’s new Passages issue paper on this topic have chosen a variety of paths to evaluation. Each foundation made its choice with a disciplined eye toward the value of what would be learned and the integrity of the process of inquiry; in this issue of Family Giving News, we feature two of these foundations’ stories.

The W. Clement and Jessie V. Stone Foundation: Aligning Evaluation with Values

When Sandra Treacy came on as executive director of the W. Clement and Jessie V. Stone Foundation, the trustees assured her that they were fully engaged and prepared for growth. “Be strategic,” they urged.

Treacy responded by organizing a “learning journey” for herself and the board. They settled on three program areas, as well as a specific focus within each. They also agreed that they would give preference to projects that foster innovation and scale up good practice.

With a staff of three, including herself, and assets of approximately $108 million, Treacy leads the foundation in awarding between 50 and 60 grants per year, averaging about $50,000 each. The lean staffing model fits well with the foundation’s commitment to fund organizations rather than projects, Treacy observed, but it also poses a problem: “projects are much easier to assess than the overall health of an organization.”

Over time, the foundation has learned to identify organizational characteristics that align with the foundation’s values in its fields of interest. The staff and board look for evidence of these attributes when assessing potential grantees or considering how much an organization has grown over the course of a grant. Characteristics include:

  • Leverage. One grantee piloted a writing curriculum. The local school district was convinced of its effectiveness and expanded the pilot throughout the district.
  • Scale. Two organizations involved nonprofit intermediaries in supporting school innovation. Having figured out how to “embed” nonprofits in the work of school reform, the groups scaled up the strategy to reach more schools.
  • Partnership/linkage. Three grantees are moving together on a shared agenda.
  • Learning. The foundation prefers to fund organizations that will grapple with complex issues and share their lessons with the field.

Grantees are required to submit annual reports, summarizing their activities and what they’re learning. But reports don’t necessarily encourage deep reflection by grantees or yield lessons that help the foundation understand the complexity of issues like improving teacher quality or building a cadre of leaders for early childhood education. The foundation therefore experiments with tactics to promote learning among grantees and its own staff and board including grantee convenings, site visits before and during a grant and use of expert panels.

“If we knew what the silver bullet was, we’d all be funding it,” says Treacy, referring to her grantmaking peers. “There is no one meta answer. What we can do is chip away at the problem and share what we’re learning.”

The Dresher Foundation: Patience Leads to Shared Focus

The early story of The Dresher Foundation follows a familiar pattern. Established in 1988, the foundation held relatively modest assets and made grants mainly to organizations known to its trustees. Then, upon the death of the donor in 1999, the foundation’s assets went from $10 million to $65 million virtually overnight. For the next year or two, the trustees weren’t sure how to proceed. They eventually adopted a focus on human services and education and decided to fund only local groups in the Baltimore area and in Harford County, Maryland.

The trustees are extremely busy people, says Robin Platts, the foundation’s executive director and sole staff member, but they were “very receptive to wanting to do this right. They were slowed down initially,” she recalled, because they were “hesitant to rule things out.” It took them a couple of years to accept that it was okay to not fund certain things.

The process of adopting a focus was helped enormously by the trustees’ first retreat, which was facilitated by an external consultant. Soft-spoken, laid back, and open-ended, the facilitator basically let the board members do all the talking. “They heard things from themselves that they wouldn’t have heard from others,” said Platts. Her advice to other foundations struggling to find a direction is “patience”.

The Dresher Foundation sees itself as mission driven, proactive, and committed to assessment and learning. With one staff member, how do they manage? First, the trustees recognize that not all outcomes are measurable and not all activities need to be assessed. As residents of the community where their funding is directed, the trustees see the impact of their grants. They’re touched by the people they serve. The board is often “more interested in anecdote than empirical data.” So, for example, it mattered that the donor’s wife had the experience of meeting an employee of a local nail salon whose son had received a Dresher Foundation scholarship.

Where the foundation really zeroes in on assessment is in its customized post-grant reports. Each post grant report includes a list of approximately 10 questions, to be answered at the end of the grant.

How Foundations Learn, Analyze and Evaluate

An interest in impact can lead a family foundation to look for ways to extend its own learning, through both conventional and unconventional means. For example, some use independent evaluators to assess the projects they support, often with explicit, ongoing learning objectives for both grantees and the foundation. Some develop expertise at tracking and analyzing data in their fields or communities to inform their own grantmaking and the efforts of their grantees. Some cultivate candid dialogue with grantees about organizational effectiveness and invest in building their organizational capacity. For these foundations, small size and proximity to local institutions can be real assets.

A hard look at costs and benefits can sometimes lead a foundation to decide against evaluation, but not necessarily at the expense of structured reflection. For example, The Seabury Foundation devoted the majority of its funding over a five-year period to a community-building program in a midwestern city. The foundation helped local agencies build organizational and leadership capacity, in hopes of having an impact on the neighborhood as a whole. The foundation’s trustees asked themselves how they would know if they had accomplished anything at the end of the initiative. They came up with three potential ways: hire a consultant to interview the agency heads and write a report, develop a questionnaire that grant recipients could respond to anonymously, and spend a day together reviewing what they did.

“We costed it out,” said Boyd McDowell III, the foundation’s director, “and realized that the consultant could cost $35,000. We decided we’d rather give that money away.” A questionnaire, they agreed, wouldn’t produce good retrospective information. The trustees therefore decided to conduct their own internal review. They discovered a few indicators of success that seemed significant. First, they found evidence of the sustainability of their work in the fact that the local leadership Forum, formerly sponsored by the foundation, was continuing to meet 18 months after the last grants were made. Second, they knew that they had played a role in solidifying a handful of weak but important organizations. Third, they had seen “the community rally around the community,” an intangible outcome that was nonetheless visible to their local eyes.

The Seabury Foundation also made a considered decision to devote relatively little of its own resources to tracking “what’s happened and whether or not what the grantee did with our money worked or not. We’re of the belief,” McDowell explained, “that we’re not experts in what these programs do. We can educate ourselves, but the people out there know the most.” Instead, the foundation prefers to ask how its grantees assess themselves—something, McDowell noted, “a good agency will always be doing.”

______________________

“We’re of the belief that we’re not experts in what these programs do. We can educate ourselves, but the people out there know the most.”

______________________

To learn more about the paths other family foundations have taken to assessing the work of their grantees, and to improving their communities, please download the new Passages issue paper on this topic, Measuring What Counts: Meaningful Evaluation for Family Foundations, available in the National Center’s Family Philanthropy Online Knowledge Center.

About the Author

Anne Mackinnon is an independent consultant to foundations and other nonprofits, specializing in strategic communications, organizational development, and education policy. (This paper is based on research conducted for the National Center for Family Philanthropy by Janice Molnar.)

This entry was posted in Management and Operations, Boards and Governance, Grantmaking and Evaluation and tagged , , , , , . Bookmark the permalink.

Share Your Comments Here!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s