Magazine article Modern Trader

Better Trading System Analysis: You Obviously Shouldn't Judge a Trading System by Its Marketing Materials. but, Not So Obviously, Neither Should You Judge It by Many of the Performance Measures Frequently Accepted throughout the Industry. A More Complete Analysis of Trading System Performance Is Necessary

Magazine article Modern Trader

Better Trading System Analysis: You Obviously Shouldn't Judge a Trading System by Its Marketing Materials. but, Not So Obviously, Neither Should You Judge It by Many of the Performance Measures Frequently Accepted throughout the Industry. A More Complete Analysis of Trading System Performance Is Necessary

Article excerpt

Many of us have endeavored for years to find publicly available trading systems that turn profits and live up to marketing claims. However, standard hypothetical performance reports have many shortcomings, can be misleading and are often meaningless.

This article will list what additional questions should be asked when analyzing systems, discuss which additional metrics can provide insight, and illustrate pitfalls encountered when trying to determine performance of a portfolio of multiple markets and systems.

MISSING IN ACTION

There are many non-measured factors that can influence the accuracy of hypothetical results. Traders must learn not only to view system analytical reports holistically, but must address questions not answered by standard analytical reports.

How many variables are used by the system? Mathematically, the more variables used by a system, the more closely it will mirror historical performance and the less likely it will accurately predict the future. The number of variables used (often referred to as "filters") is one of two factors that facilitate the misguided practice of "curve fitting." For example, a trading system containing eight variables, once optimized, should be able to predict historical market movements with 100% accuracy, but such a system would be a worthless predictor of future market movements. A robust trading system should have no more than two variables. Examples of variables include the following, denoted by x: If the market moves x% of yesterday's range above yesterday's close then buy; if the price equals the highest high of x days then buy; if the moving average of x bars crosses above the close then sell.

[ILLUSTRATION OMITTED]

What types of optimizations were used and what does the optimization report show? When a system developer creates a system, he often starts with a basic principle, yet has no idea what parameters should be used. For example, a system developer may decide to use a percentage breakout to buy and sell but does not know what that percentage should be. So, the developer will try different percentages to see what works best. This is considered "system optimization"--the second component of "curve fitting." Optimizing a system using small increments (such as, 0.01%, 0.02%, 0.03% ... 100%) significantly increases the risk of curve fitting and consequently decreases the chances of the system working in the future. Additionally, we need to know what the optimization-report looks like. Say we optimize a system using increments of 1% and the process finds that 36% is the best performing percentage. We need to know if 35% and 37% performed similarly because market conditions change daily. If 36% is most profitable, yet 35% and 37% result in a loss, we do not have a robust system. We want to see a nice smooth curve in our optimization reports (that is, 35% and 37% should be almost as profitable as 36%).

How many markets does the system trade profitably? Another measure of system robustness is how well the system trades across multiple markets using the same parameters. The differences between markets and sectors are analogous to the changes in behavior that a single market may experience over time--some markets move quickly while others move slowly, some markets trend while others typically channel sideways.

How many years of data were used to generate the performance report and how many sample trades are represented? First, you want to ensure that there is enough history to cover bull, bear and flat market conditions. Second, you want to ensure enough trades have been taken within each market to be statistically significant (at least 30 trades). Note that system developers often provide a limited number of years of hypothetical results because the system simply does not function in other years. Longer-term systems should present 20 years of results or the life of the futures contract.

What is the trade frequency and duration? …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.