I think (hope) I can clarify a little based on a recent post I had in
a blog:

I completely agree that there is no universal metric: Teaching a child
to read is important. Giving a pregnant mother nutritional supplements
to avoid premature childbirth is important. Providing disaster relief
after a hurricane is important. That's not the point. The point is:
Given that you want to teach children to read - how do you know that
you are doing a good job, that you are spending your funds in a wise
manner and using the appropriate teaching methods, partnering with
communities in a way that makes new procedures stick long term, and so
on? Given that you want to provide nutrititional supplements to
pregnant women, how do you evaluate whether you are doing it in the
best way possible and how do you find out in what areas you could do
better? Are you targeting the geographies where the problem is worst?
Are you focusing only where infrastructure makes it easy to assist?
And so on.

In other words: The question is not "what is the equivalent of a
universal profit measure for non-profits?" (How much goodness-per-
dollar are you providing?) - the question is how can we measure or
evaluate or judge our success in achieving the outcomes we are aiming
for - however narrowly these need to be defined to make sense?

Imagine two non-profits both trying to teach children in the same area
how to read, but one using volunteer teachers from Canada and the
other spreading a new teaching method to local teachers using local
volunteers. Both strategies can have their strengths and weaknesses,
we can have our personal convictions concerning what we think is a
good strategy - but how can we find out? What data or whose judgment
could be used for these two charities to discuss their alternative
strategies for achieving the same purpose? What would it take to
(rationally) convince one to abandon its approach and switch to the
other?

Put differently: We cannot assume out of hand that every idea someone
has funded is a sensible way of approaching a problem, and even good
approaches can be improved. What do we use to compare two approaches
and to evaluate suggested improvements?

See more here: http://lifeyears.blogspot.com/2008/02/yes-we-should-measure-impact-bu...