|
Although there are no "right" or
"wrong" evaluation criteria, there are better and
worse ones, or at least more useful and less useful ones. The
characteristics of good evaluation criteria are (adapted from
Keeney and Gregory, 2005):
-
Accurate and Unambiguous, meaning that
a clear and accurate relationship exists between the
criteria and the real consequences.
-
Comprehensive but concise, meaning
that they cover the range of relevant consequences but the
evaluation framework remains systematic and manageable and
there are no redundancies.
-
Direct and ends-oriented, meaning they
report directly on the consequences of interest and provide
enough information that informed value judgments can
reasonably be made on the basis of them.
-
Measurable and Consistently Applied to allow
consistent comparisons across alternatives. This means the
criteria should be able to distinguish the relative degree
of impact across alternatives. It does not exclude
qualitative characterizations of impact, or impacts that
can't be physically measured in the field.
-
Understandable, in that consequences and
trade-offs can be understood and communicated by everyone
involved.
-
Practical, meaning that information can
practically be obtained to assess them (i.e., data, models
or expert judgment exist or can be readily developed).
-
Sensitive to the Alternatives under
consideration, so that they provide information that is
useful in comparing alternatives.
-
Explicit about Uncertainty so that they
expose differences in the range of possible outcomes
(differences in risk) associated with different policy or
management alternatives.
While not a strict requirement, it is good
practice to check that the criteria are also additive - or more
formally, preferentially independent (Keeney, 1992). This means
that they contribute independently to the total performance of
an alternative. When criteria are preferentially independent,
simplified decision modeling tools can legitimately be used. And
because preferential independence is almost always implicitly
assumed, it is best to make sure the assumption is valid to
avoid errors of logic. If it is not valid, then more complex
analysis is required. There will be trade-offs to make in
selecting criteria. For example, the most "direct and
ends-oriented" criteria tend to be less "operational'
as they are difficult to estimate or model. The most
"accurate" may not be understandable to non-technical
decision makers.
Next>> |
Good evaluation criteria share a number of
characteristics