PrintAt Commonwealth, we evaluate managers of all sorts on many criteria, statistical and qualitative. We never know, really, whether the analysis will work—which is why we constantly review and manage our recommended list—but we know, from experience, most of what to look for in most types of funds.
The problem comes when we consider new approaches, where we don’t have a long track record to work with. Recently, this has been the case with multiple alternative investment-style funds, many of which have been introduced over the past several years. These strategies are new to the public space, and new to many of the companies offering them, and they simply don’t have the track records to allow the usual quantitative analyses.
One way to look at them is the pony approach, inspired by the old story about the optimist who, when presented with a large pile of horse manure, grabbed a shovel and started digging, on the premise that “there must be a pony in there somewhere!” We don’t take this approach, but we don’t ignore it either. After all, many successful strategies were once new and unusual—frontier markets, international investing, and even stocks themselves.
So how can we balance the two risks, of buying a heap of manure versus missing the pony? This is the problem investors face when looking at past data in any area. You have an enormous pile of, well, information, and your task is to find the pony—if, in fact, one is really there.
At its simplest, market rules of thumb provide a good example of the problem. Does the January effect, for example, actually work? How about “sell in May and go away”? On a more complex level, we face the same challenge when evaluating quantitative “alternative” strategies that have recently become available to retail investors.
I’m experiencing my own version of this problem right now, while working on a research project designed to generate diversified portfolios that have minimal drawdowns as a primary objective. The main risk I wanted to avoid was data mining—that is, running models against the data until I found one that worked. You will never see a bad backtest because only the good ones are selected for display.
Backtested models have a notoriously poor record when deployed in the real world. Many are used solely because of the historical record, without any fundamental supporting reasons why they not only did work but have to work. Be very suspicious of any argument that starts with history as its primary justification.
I therefore started off with two requirements. Any method I used had to be based on a priori assumptions that made fundamental sense, and it had to have some kind of real money-based track record that I could use as a reality check. To avoid the backtesting effect, I also had to specify the model ahead of time, rather than selecting based on what worked in the data. Finally, I had to test the model on data outside of that used to develop it, known as out-of-sample testing.
The project is going well, and I’m pleased with the results so far. But the basic process—specify the initial model, test on out-of-sample data, and work where possible with real-world experience—stands as a sound basis for approaching any strategy you want to evaluate.
