January 10, 2012
Let us turn our attention to the accuracy of the WLI growth figure pivoted about -2.638. Why -2.638? This number comes from back testing that mathematically optimizes the accuracy of the recession calling model from three aspects: AUC (area under the curve) accuracy, NBER capture rate, and false positives. There are others one might use, such as lead and lag accuracy, but all three of these metrics are crucial measurements of a recession forecasting/dating system, and they are sometimes mutually exclusive. For example a high AUC does not guarantee a high NBER capture, and a high NBER capture does not guarantee a low false positive rate. So your optimization technique has to find the trigger that gives the best blend of these three metrics. For the above example, if we try to raise the NBER capture rate, the false positives increase. And if we try to change the threshold to a number that reduces the false positives to less than 4 then the NBER capture rate drops dramatically.
Let’s review these metrics before moving on:
- “AUC accuracy” is how many of the sample points (2,292 weeks for this particular exercise) the MODEL correctly categorized as being either a recession week or a non-recession week. The AUC for the WLI growth metric pivoted about the trigger of -2.638 is 0.904, meaning the model correctly categorised 90.4% of all weeks since 1967. Conversely, this means it had an error rate of 9.6% – some 220 weeks were incorrectly categorized by the model (either by calling a recession when there was none or saying there was no recession when there was indeed one). In statistical terms, this is known as the percentage correct or “area under the curve” of the model.
- “NBER capture” refers to how many of the 360 NBER dated recessionary weeks since 1967 the MODEL correctly categorized. The figure is 86.1%, meaning 13.9% or some 50 weeks were completely missed by the model. In statistical terms this is known as the sensitivity of the model.
- “False positives” are how often the model flagged a recession when there was none. When determining false positives for the model we ignored cases in which the model flagged a recession up to 20 weeks before the actual recession occurred, since this is a good thing – the model is giving us several months advance warning of a recession that did indeed materialize. We also ignored the current recession reading to the right of the chart, as we have no way of knowing if we are indeed in a recession right now(we are pretty convinced we are not!).
Is there a better model than the WLI growth metric?
So is the WLI Growth metric provided by ECRI is the best oscillator to use? To find out, my firm performed an optimization procedure to find out what WLI percentage rate-of-change period yields a model that best blends the three performance metrics discussed previously. The result is below:
We found that a three-period simple moving average of the 52-week WLI percentage rate-of-change produced the best results. The chart shows that this new model still flags every NBER recession, but eliminates three false positives. AUC accuracy is 2.8% better (64 additional accurate weeks), although NBER capture drops 4.16% (15 recession weeks). The overriding benefit, however, is that this indicator would have yielded only one false positive in the last 40+ years. We cannot over-emphasise how important this improvement is – it leaves you with a system you are more likely to trust and act upon (although in our prior article we warn against the use of a single indicator for recession dating.)
Display article as PDF for printing.
Would you like to send this article to a friend?
Remember, if you have a question or comment, send it to .
