Overreliance on statistical models

The asymmetric distribution of credit returns makes it more difficult to measure
credit risk than market risk. While banks’ efforts to measure credit risk in a portfolio
context can represent an improvement over existing measurement practices, portfolio
managers must guard against over-reliance on model results. Portfolio models can
complement, but not replace, the seasoned judgment that professional credit personnel
provide.
Model results depend heavily on the validity of assumptions. Banks must not
become complacent as they increase their use of portfolio models, and cease looking
critically at model assumptions. Because of their importance in model output, credit
correlations in particular deserve close scrutiny. Risk managers must estimate credit
correlations since they cannot observe them from historical data. Portfolio models
use different approaches to estimating correlations, which can lead to very different
estimated loss distributions for the same portfolio.
Correlations are not only difficult to determine but can change significantly over
time. In times of stress, correlations among assets increase, raising the portfolio’s
risk profile because the systematic risk, which is undiversifiable, increases. Credit
portfolio managers may believe they have constructed a diversified portfolio, with
desired risk and return characteristics. However, changes in economic conditions
may cause changes to default correlations. For example, when energy and Texas real
estate prices became highly correlated, those correlation changes exposed banks to
significant unanticipated losses. It remains to be seen whether portfolio models can
identify changes in default correlation early enough to allow risk managers to take
appropriate risk-reducing actions.
In recent years, there have been widely publicized incidents in which inaccurate
price risk measurement models have led to poor trading decisions and unanticipated
losses. To identify potential weaknesses in their price risk models, most banks use
a combination of independent validation, calibration, and backtesting. However, the
same data limitations that make credit risk measurement difficult in the first place
also make implementation of these important risk controls problematic. The absence
of credit default data and the long planning horizon makes it difficult to determine,
in a statistical sense, the accuracy of a credit risk model. Unlike market risk models,
for which many data observations exist, and for which the holding period is usually
only one day, credit risk models are based on infrequent default observations and a
much longer holding period. Backtesting, in particular, is problematic and would
involve an impractical number of years of analysis to reach statistically valid conclusions. In view of these problems, banks must use other means, such as assessing
model coverage, verifying the accuracy of mathematical algorithms, and comparing
the model against peer group models to determine its accuracy. Stress testing, in
particular, is important because models use specified confidence intervals. The
essence of risk management is to understand the exposures that lie outside a model’s
confidence interval.