How can you detect model risk?

Unfortunately, we have to admit that there is no unique method to detect model risk.
The essential problem is that the model-building process is a multi-step procedure,
whereas model risk is generally assessed at the end of the chain. The result is an
amalgamation of conceptually distinct discrepancy terms. All successive errors in
the model-building process are aggregated into a single real-valued variable, typically
the average difference between some empirically observed value (for instance, an
option price) and the result of the model. This makes it difficult to detect which
aspects of the model, if any, are seriously misspecified.
However, some signals should be carefully monitored, as they are good early
indicators of model risk. For instance, a model with a poor performance out of the
sample while its performance was excellent in the sample, or time-varying parameters
that vary too much should arise suspicion. Using all the degrees of freedom in a
model to fit the data often results in an over-parametrization. If we need, say, two
degrees of freedom and we have three or more available, we can simply use the third
degree to capture the model errors as a time-varying component.

It is now widely known that models with time-varying parameters have an excellent
in-sample performance, particularly for pricing and hedging purposes. But all the
model risk is concentrated into the time-varying parameters. This is particularly true
if the parameter is not observable, such as a mean-reversion for interest rate, a riskaversion
parameter for a given investor, or a risk premium or a given source of risk.
In a sense, these models are built specifically to fit an arbitrary exogenous set of
data.