Weekly Thoughts: Data Wildcatting
Here is something that caught our eye this week:
Data Wildcatting
In a previous life, we spent our days thinking about things like style factors, arbitrage strategies, and smart beta. We relied on some of that experience this week when reading about an ongoing debate regarding the efficacy of investment strategies that apply quantitative filters to data in an effort to outperform the market. While at first glance it seems that an ever increasing amount of technology-based analysis should result in higher quality, more rigorous strategies, the reality is much more nuanced, as Bloomberg explains:
“The core of the problem is that it’s hard to beat the market, but people keep trying anyway. An abundance of computing power makes it possible to test thousands, even millions, of trading strategies. The standard method is to see how the strategy would have done if it had been used during the ups and downs of the market over, say, the past 20 years. This is called backtesting. As a quality check, the technique is then tested on a separate set of ‘out-of-sample’ data—i.e., market history that wasn’t used to create the technique.”
The increasing availability of data, however, can actually complicate matters, as anybody who has read an investment prospectus knows, past performance is not an indicator of future results. The problem with backtesting an exponential amount of information is that it introduces, and perhaps even increases, the risk of finding noise and not signal. In the industry, these initially attractive but ultimately spurious results are the result of “data-mining.” AQR’s Cliff Asness explains:
“Data mining, that is searching the data to find in-sample patterns in returns that are not real but random, and then believing you’ve found truth, is a real problem in our field. Random doesn’t tend to repeat so data mining often fails to produce attractive real life returns going forward. And given the rewards to gathering assets, often made easier with a good ‘backtest,’ the incentive to data mine is great.”
The problem with data mining is that with enough trial and error, one can find justification for almost any investment strategy. For instance, Bloomberg highlights how backtesting using United Nations data found that the best predictor of S&P performance was Bangladeshi butter production. As a consequence, sometimes strategies with impressive backtests can significantly underperform expectations when establishing a live track record.
We will readily admit that we are not qualified to opine on the finer points of quantitative investment strategy creation since we are now more landscapers than quants. That said, it struck us that Chenmark itself is chasing a similar goal — namely the continuous pursuit of data-driven strategies that can lead to increased profit over time. In a sense, our growing small business portfolio is its own “out-of-sample” test of some of our core investment philosophies. We believe strongly in, and feel there is ample historical precedent for, the wealth creation potential of cash-generative businesses that are managed prudently for the long term with a significant focus on capital allocation. That said, applying such concepts to smaller companies generally, or to our particular subset thereof, is largely unproven. Moreover, in small business, establishing a structure to ensure there is data to analyze is often a larger challenge than the analysis itself. We need to think of ourselves more as data wildcatters before we can be worried about whether we are mining or not. To be successful in such an uncertain environment, we concentrate on operational principles that have stood the test of time. On this point, Asness offers some valuable insights:
“At AQR we pride ourselves on minimizing data mining. Nobody in our field is perfect on this front but we’ve had the discipline to walk away from good-looking factors we don’t trust. We have no desire to find things that have worked in the past that won’t work going forward…. We believe in a small subset of things that have worked out-of-sample through time, out-of-sample across geography, out-of-sample across asset class, and importantly, are explained by an economic story that’s not just ‘the data says so.’”
While the team at AQR focuses on identifying securities that demonstrate value and momentum among other factors, we believe Asness’s framework for establishing what works can be generalized to business operations more broadly. As we evaluate prospects and oversee portfolio operations, we will continue to look for factors we believe signal business stability such as recurring revenue, management depth, and history of profitability, while leaving plenty time for some good old-fashioned gut-checks along the way.
Have a great week,
Your Chenmark Capital Team
