Backtesting

MARK DEANS
The aim of backtesting is to test the effectiveness of market risk measurement by
comparing market risk figures with the volatility of actual trading results. Banks
must carry out backtesting if they are to meet the requirements laid down by the
Basel Committee on Banking Supervision in the Amendment to the Capital Accord to
incorporate market risks (1996a). If the results of the backtesting exercise are
unsatisfactory, the local regulator may impose higher capital requirements on a
bank. Further, when performed at a business line or trading desk level, backtesting
is a useful tool to evaluate risk measurement methods.

Backtesting is a requirement for banks that want to use internal models to calculate
their regulatory capital requirements for market risk. The process consists of comparing
daily profit and loss (P&L) figures with corresponding market risk figures over a
period of time. Depending on the confidence interval used for the market risk
measurement, a certain proportion of the P&L figures are expected to show a loss
greater than the market risk amount. The result of the backtest is the number of
losses greater than their corresponding market risk figures: the ‘number of exceptions’.
According to this number, the regulators will decide on the multiplier used for
determining the regulatory capital requirement.
Regulations require that backtesting is done at the whole bank level. Regulators
may also require testing to be broken down by trading desk (Figure 9.1). When there
is an exception, this breakdown allows the source of the loss to be analysed in more
detail. For instance, the loss might come from one trading desk, or from the sum of
losses across a number of different business areas.
In addition to the regulatory requirements, backtesting is a useful tool for evaluating
market risk measurement and aggregation methods within a bank. At the whole
bank level, the comparison between risk and P&L gives only a broad overall picture of
the effectiveness of the chosen risk measurement methods. Satisfactory backtesting
results at the aggregate level could hide poor risk measurement methods at a lower
level. For instance, risks may be overestimated for equity trading, but underestimated
for fixed income trading. Coincidentally, the total risk measured could be approximately
correct. Alternatively, risks could be underestimated for each broad risk
category (interest rate, equity, FX, and commodity risk), but this fact could be hidden
by a very conservative simple sum aggregation method.
Backtesting at the portfolio level, rather than just for the whole bank, allows individual market risk measurement models to be tested in practice. The lower the
level at which backtesting is applied, the more information becomes available about
the risk measurement methods used. This allows areas to be identified where market
risk is not measured accurately enough, or where risks are being taken that are not
detected by the risk measurement system.
Backtesting is usually carried out within the risk management department of a
bank where risk data is relatively easily obtained. However, P&L figures, often
calculated by a business unit control or accounting department, are equally important
for backtesting. The requirements of these departments when calculating P&L
are different from those of the risk management department. The accounting principle
of prudence means that it is important not to overstate the value of the portfolio, so
where there is uncertainty about the value of positions, a conservative valuation will
be taken. When backtesting, the volatility of the P&L is most important, so capturing
daily changes in value of the portfolio is more important than having a conservative
or prudent valuation. This difference in aims means that P&L as usually calculated
for accounting purposes is often not ideal for backtesting. It may include unwanted
contributions from provisions or intraday trading. Also, the bank’s breakdown of P&L
by business line may not be the same as the breakdown used for risk management.
To achieve effective backtesting, the risk and P&L data must be brought together
in a single system. This system should be able to identify exceptions, and produce
suitable reports. The data must be processed in a timely manner, as some regulators
(e.g. the FSA) require an exception to be reported to them not more than one business
day after it occurs.
In the last few years, investment banks have been providing an increasing amount
of information about their risk management activities in their annual reports. The
final part of this chapter reviews the backtesting information given in the annual
reports of some major banks.