Review of backtesting results in annual reports

Risk management has become a focus of attention in investment banking over the
last few years. Most annual reports of major banks now have a section on risk
management covering credit and market risk. Many of these now include graphs
showing the volatility of P&L figures for the bank, and some show backtesting graphs.
Table 9.2 shows a summary of what backtesting information is present in annual
reports from a selection of banks.
Table 9.2 Backtesting information in annual reports
Risk
management Backtesting
Company Date section P&L graph graph
Dresdner Bank 1997 Yes No No
Merrill Lynch 1997 Yes Yesa No
Deutsche Bank 1997 Yes No Nob
J. P. Morgan 1997 Yes Yes Noc
Lehman Brothers 1997 Yes Yesd No
ING Group 1997 Yes No No
ABN AMRO Holding 1997 Yes No No
Credit Suisse Group 1997 Yes Yes Yes
Sanwa Bank 1998 Yes Noe Yes
a Merrill Lynch’s P&L graph is of weekly results. It shows 3 years’ results year by year for comparison.
b Deutsche Bank show a graph of daily value at risk
c J. P. Morgan gives a graph of Daily Earnings at Risk (1-day holding period, 95% confidence interval)
for two years. The P&L histogram shows average DEaR for 1997, rebased to the mean daily profit.
d Lehman Brothers graph is of weekly results.
e Sanwa Bank also show a scatter plot with risk on one axis and P&L on the other. A diagonal line
indicates the confidence interval below which a point would be an exception.
Of the banks that compare risk to P&L, J. P. Morgan showed a number of exceptions
(12 at the 95% level) that was consistent with expectations. They interpret their Daily
Earnings at Risk (DEaR) figure in terms of volatility of earnings, and place the
confidence interval around the mean daily earnings figure of $12.5 million. The shift
of the P&L base for backtesting obviously increases the number of exceptions. It compensates for earnings that carry no market risk such as fees and commissions,
but also overstates the number of exceptions that would be obtained from a clean
P&L figure by subtracting the average profit from that figure. Comparing to the
average DEaR could over or understate the number of exceptions relative to a
comparison of each day’s P&L with the previous day’s risk figure.
Credit Suisse Group show a backtesting graph for their investment bank, Credit
Suisse First Boston. This graph plots the 1-day, 99% confidence interval risk figure
against P&L (this is consistent with requirements for regulatory reporting). The graph
shows no exceptions, and only one loss that even reaches close to half the 1-day,
99% risk figure. The Credit Suisse First Boston annual review for 1997 also shows a
backtesting graph for Credit Suisse Financial Products, Credit Suisse First Boston’s
derivative products subsidiary. This graph also shows no exceptions, and has only
two losses that are around half of the 1-day, 99% risk figure. Such graphs show that
the risk figure measured is overestimating the volatility of earnings. However, the
graph shows daily trading revenue, not clean or hypothetical P&L prepared specially
for backtesting. In a financial report, it may make more sense to show actual trading
revenues than a specially prepared P&L figure that would be more difficult to explain.
Sanwa Bank show a backtesting graph comparing 1-day, 99% confidence interval
risk figures with actual P&L (this is consistent with requirements for regulatory
reporting). Separate graphs are shown for the trading and banking accounts. The
trading account graph shows only one loss greater than half the risk figure, while
the banking account graph shows one exception. The trading account graph shows
an overestimate of risk relative to volatility of earnings, while the banking account
graph is consistent with statistical expectations
The backtesting graphs presented by Credit Suisse Group and Sanwa Bank indicate
a conservative approach to risk measurement. There are several good reasons for this:
Ω It is more prudent to overestimate, rather than underestimate risk. This is
especially so as market risk measurement systems in general do not have several
years of proven performance.
Ω From a regulatory point of view, overestimating risk is acceptable, whereas an
underestimate is not.
Ω Risk measurement methods may include a margin for extreme events and crises.
Backtesting graphs for 1998 will probably show some exceptions.
This review of annual reports shows that all banks reviewed have risk management
sections in their annual reports. Backtesting information was only given in a few cases,
but some information on volatility of P&L was given in over half the reports surveyed.