I second the notion. We need a better rubric for evaluating nonprofit
effectiveness; we'll all get there faster if we collaborate on the
metrics we're using, instead of letting the powers-that-be regale us
with "admin overhead vs. program" and other favorite nursery rhymes.

For an organization as large as the Red Cross, this must be a
monumental data analysis exercise; even knowing the types of data
gathered would be helpful. How do you judge and measure your
effectiveness? Is is purely subjective, or is there a more analytical
method of figuring out that your disaster relief programs were more
effective or more efficient this year than last year?

Looking forward to your responses. Sean, thanks for blogging this; I
might not have seen this otherwise. This portal has the potential to
change how nonprofits are seen online; it certainly changes the game
considerably.

  Dave.

Hello, folks, it's Maura again.  I am afraid I am not a statistician,
so I am really not familiar with the kind of analysis you are asking
for.  We do collect standard annual service data from our chapters,
such as the number and types of disasters to which they have
responded, the number of people they have trained in first aid, CPR,
disaster preparedness, etc.  We know how many people donate blood (a
frighteningly small percentage of those who are eligible).  Today is
Saturday and I am replying from home, so I don't have access to the
numbers at the moment.  Maybe you could provide me with a link to an
explanation of what you have in mind.
Before I close, I just want to say that the blood supply is always low
around the holidays because people are busy.  So if you can donate,
please do.  You truly could be saving lives.  And please be careful
about fire safety this holiday season.

It's Maura yet again!  I've been thinking about Sean's and Raindrop's
questions.  Like I said, statistics are not my game.  But even if they
were... how would we compare the community impact of, say, a unit of
blood, a person learning a lifesaving skill like CPR, a person
sheltered in a disaster, a meal delivered to someone cleaning up after
a disaster, etc.?  And when it comes to efficiency, a meal served in a
shelter setting is obviously going to cost less than a meal delivered
on one of our trucks to a recovering neighborhood, so how do you
compare?  Some vital services are quick and easy to deliver; others
are hard and expensive, but equally vital.  Costs can also vary widely
from year to year.
And if you are comparing one charity with another... how do you
compare, say, the impact of taking a school kid to the symphony with
the impact of funding research, which perhaps fails, on some deadly
disease?  Our missions are all so different!

Yes, your missions are all different. But I'm not asking you to
compare yourself to other charities. I like the way Dave put it above:

"How do you judge and measure your effectiveness? Is is purely
subjective, or is there a more analytical method of figuring out that
your disaster relief programs were more effective or more efficient
this year than last year?"

I don't know enough about the Red Cross to really know the best way to
understand how effective you are. But I assume you know if you are
effective. So how do you know that? The answer might not be
statistical, but there must be some way that you know that it makes
more sense for me as a donor to fund you than to cut a check to FEMA.

Hello -- this is Ike, and I am a regional communicator for the Red
Cross.  I stumbled across this over the holiday break.

I understand what you are talking about, with regards to our internal
measure of "effectiveness."  Unfortunately, you're asking us the
equivalent of choosing a favorite child.

Such a metric would be arbitrary, and could be easily fashioned to
highlight whichever line of service we wished to justify.  In doing
so, number-crunchers would ask the question "Why in heck is Red Cross
involved in things that AREN'T as high-payoff as _______?"  Just look
at the numbers.  Why be involved in disaster relief when blood
provides the higher "impact?"  Or vice versa?

We're dealing with two different dynamics here.  As a large multi-
purpose humanitarian organization, we've got a tradition being
involved in a number of different activities.  Disaster, blood,
service to armed forces, preparedness, first aid/safety, and some of
the international initiatives Maura described.  Whether we like it or
not, there is a significant slice of America that expects the Red
Cross to play a role in each of those arenas.  Public expectation
drives part of our mission.  In some circumstances, we have made a
promise to be there (like immediate disaster relief).  In others, we
end up getting involved because people think that's what we're
supposed to do, and no one else is stepping up (like the Safe and Well
website partnership.)

The second dynamic is our volunteers.  Some only have an interest in
disaster.  Some only want to teach first aid classes.  Some want to
volunteer to drive needed units of blood from the storage centers to
the hospitals.  As a volunteer-led group, we'd alienate so many people
who are truly volunteering their time to make it all work.

Are you really asking us to pick the one most effective line of
service, and do that to the exclusion of the rest?  Because applying a
universal metric to all the lines of service is an invitation to
starting feeding some and starving others.  That would be akin to
comparing the costs of helping 10 families in an apartment fire versus
10 single-home families spread out on different nights.  Yes, one is
more "cost-effective."  That doesn't mean it's time to abandon the
rest.

I think the key element you are dancing around here is the way we
handle donations.  If someone wants to donate just to local fires in
their local chapter jurisdiction, we can assure that happens.  If
someone wants to donate just to Services to Armed Forces, their wishes
are respected and followed through.  We look at the business model of
each of those lines differently, asking first "Are we meeting this
mission?" and "Can we meet it more efficiently another way?"

Thanks so much for jumping into the conversation. I'm not asking you
to choose anything. I'm just asking how the Red Cross tracks whether
you're doing a good job.

For example, at my firm, Ensemble Capital Management we look at hard
numbers like revenue growth, assets under management and assets per
client. We also look at softer measures like visibility in the media
and online, depth of relationships with referral sources, and client
satisfaction. You can put good numbers on the first set, but not on
the second.

All I'm asking the Red Cross is how do you know if you are doing a
good job? What do you track? And how do you compare yourself? For
instance, what if I asked you why my money could do more good by
donating it to you than donating it to another similar organization or
even to FEMA? If an investor or prospective client asked me why I
thought that Ensemble was a better investment or firm to hire than our
competitors, I could speak to the issue for hours, citing both hard
data and soft qualities. I'm just asking the Red Cross the same
question.

Sean, this is a great topic for debate.  You've probably checked out
GiveWell.net, which studies charities in particular fields and ranks
them on their effectiveness. Founded by two hedge-fund analysts,
GiveWell applies business techniques in evaluating the nonprofits
within specific categories. But the question of whether one should
rank charities - and thus discourage people involved with the ones
that don't rank at the top but do great work nevertheless - is a
philosophical one. The New York Times wrote a balanced and thoughtful
piece - http://www.nytimes.com/2007/12/20/us/20charity.html - on this
subject and on GiveWell in particular.
Dorian Adams (www.JustCauseIt.com - the social-networking website for
the greater good)

Hi Ike and Maura,

I think (hope) I can clarify a little based on a recent post I had in
a blog:

I completely agree that there is no universal metric: Teaching a child
to read is important. Giving a pregnant mother nutritional supplements
to avoid premature childbirth is important. Providing disaster relief
after a hurricane is important. That's not the point. The point is:
Given that you want to teach children to read - how do you know that
you are doing a good job, that you are spending your funds in a wise
manner and using the appropriate teaching methods, partnering with
communities in a way that makes new procedures stick long term, and so
on? Given that you want to provide nutrititional supplements to
pregnant women, how do you evaluate whether you are doing it in the
best way possible and how do you find out in what areas you could do
better? Are you targeting the geographies where the problem is worst?
Are you focusing only where infrastructure makes it easy to assist?
And so on.

In other words: The question is not "what is the equivalent of a
universal profit measure for non-profits?" (How much goodness-per-
dollar are you providing?) - the question is how can we measure or
evaluate or judge our success in achieving the outcomes we are aiming
for - however narrowly these need to be defined to make sense?

Imagine two non-profits both trying to teach children in the same area
how to read, but one using volunteer teachers from Canada and the
other spreading a new teaching method to local teachers using local
volunteers. Both strategies can have their strengths and weaknesses,
we can have our personal convictions concerning what we think is a
good strategy - but how can we find out? What data or whose judgment
could be used for these two charities to discuss their alternative
strategies for achieving the same purpose? What would it take to
(rationally) convince one to abandon its approach and switch to the
other?

Put differently: We cannot assume out of hand that every idea someone
has funded is a sensible way of approaching a problem, and even good
approaches can be improved. What do we use to compare two approaches
and to evaluate suggested improvements?

See more here: http://lifeyears.blogspot.com/2008/02/yes-we-should-measure-impact-bu...