The Independent Market Observer

3/28/14 – How to Evaluate Investment Track Records

Posted by Brad McMillan, CFA®, CFP®

Find me on:

This entry was posted on Mar 28, 2014 11:30:00 AM

and tagged Commentary

Leave a comment

At Commonwealth, we evaluate managers of all sorts on many criteria, statistical and qualitative. We never know, really, whether the analysis will work—which is why we constantly review and manage our recommended list—but we know, from experience, most of what to look for in most types of funds.

The problem comes when we consider new approaches, where we don’t have a long track record to work with. Recently, this has been the case with multiple alternative investment-style funds, many of which have been introduced over the past several years. These strategies are new to the public space, and new to many of the companies offering them, and they simply don’t have the track records to allow the usual quantitative analyses.

One way to look at them is the pony approach, inspired by the old story about the optimist who, when presented with a large pile of horse manure, grabbed a shovel and started digging, on the premise that “there must be a pony in there somewhere!” We don’t take this approach, but we don’t ignore it either. After all, many successful strategies were once new and unusual—frontier markets, international investing, and even stocks themselves.

So how can we balance the two risks, of buying a heap of manure versus missing the pony? This is the problem investors face when looking at past data in any area. You have an enormous pile of, well, information, and your task is to find the pony—if, in fact, one is really there.

At its simplest, market rules of thumb provide a good example of the problem. Does the January effect, for example, actually work? How about “sell in May and go away”? On a more complex level, we face the same challenge when evaluating quantitative “alternative” strategies that have recently become available to retail investors.

I’m experiencing my own version of this problem right now, while working on a research project designed to generate diversified portfolios that have minimal drawdowns as a primary objective. The main risk I wanted to avoid was data mining—that is, running models against the data until I found one that worked. You will never see a bad backtest because only the good ones are selected for display.

Backtested models have a notoriously poor record when deployed in the real world. Many are used solely because of the historical record, without any fundamental supporting reasons why they not only did work but have to work. Be very suspicious of any argument that starts with history as its primary justification.

I therefore started off with two requirements. Any method I used had to be based on a priori assumptions that made fundamental sense, and it had to have some kind of real money-based track record that I could use as a reality check. To avoid the backtesting effect, I also had to specify the model ahead of time, rather than selecting based on what worked in the data. Finally, I had to test the model on data outside of that used to develop it, known as out-of-sample testing.

The project is going well, and I’m pleased with the results so far. But the basic process—specify the initial model, test on out-of-sample data, and work where possible with real-world experience—stands as a sound basis for approaching any strategy you want to evaluate.

Subscribe via Email

New call-to-action
Crash-Test Investing

Hot Topics

New Call-to-action



see all



The information on this website is intended for informational/educational purposes only and should not be construed as investment advice, a solicitation, or a recommendation to buy or sell any security or investment product. Please contact your financial professional for more information specific to your situation.

Certain sections of this commentary contain forward-looking statements that are based on our reasonable expectations, estimates, projections, and assumptions. Forward-looking statements are not guarantees of future performance and involve certain risks and uncertainties, which are difficult to predict. Past performance is not indicative of future results. Diversification does not assure a profit or protect against loss in declining markets.

The S&P 500 Index is a broad-based measurement of changes in stock market conditions based on the average performance of 500 widely held common stocks. All indices are unmanaged and investors cannot invest directly in an index.

The MSCI EAFE (Europe, Australia, Far East) Index is a free float‐adjusted market capitalization index that is designed to measure the equity market performance of developed markets, excluding the U.S. and Canada. The MSCI EAFE Index consists of 21 developed market country indices.

One basis point (bp) is equal to 1/100th of 1 percent, or 0.01 percent.

The VIX (CBOE Volatility Index) measures the market’s expectation of 30-day volatility across a wide range of S&P 500 options.

The forward price-to-earnings (P/E) ratio divides the current share price of the index by its estimated future earnings.

Third-party links are provided to you as a courtesy. We make no representation as to the completeness or accuracy of information provided on these websites. Information on such sites, including third-party links contained within, should not be construed as an endorsement or adoption by Commonwealth of any kind. You should consult with a financial advisor regarding your specific situation.


Please review our Terms of Use

Commonwealth Financial Network®