Define clearly your ‘model risk’ metric and a benchmark

The first step to assess model risk is the definition of a complete model risk metric.
What are the criteria used to qualify a model as ‘good’ or ‘bad’? Which goal should a
model pursue? The answer can vary widely across applications. For pricing purposes,
one might consider minimizing the difference between the results of a model and
market prices or a given benchmark. The latter is not always possible: for instance,
for stock options, the Black and Scholes (1973) model appears to be relatively robust
and is widely accepted as a benchmark, while for interest rate options, there are
many different models of the term structure of interest rates, but little agreement on
any natural benchmark. For hedging purposes, one may want to minimize the
terminal profit and loss of a hedged position, its average daily variation or its
volatility. But others will also focus only on the maximum possible loss, on the
drawdown or on the probability of losing money. And for regulatory capital, some
institutions will prefer minimizing the required capital, while others will prefer to be
safe and reduce the number or probability of exceptions.