updated 12 July 2016

Who is this Guy ?

Proprietary/Principal Trading

Brian Peterson:

Proprietary Trading:

  • proprietary or principal traders are a specific "member" structure with the exchanges
  • high barriers to entry, large up-front capital requirements
  • many strategies pursued in this structure have capacity constraints
  • benefits on the other side are low fees, potentially high leverage
  • money management needs to be very focused on drawdowns, leverage, volatility

R in Finance trade simulation toolchain

Backtesting, art or science?



Back-testing. I hate it - it's just optimizing over history. You never see a bad back-test. Ever. In any strategy. - Josh Diedesch (2014), CalSTRS



Every trading system is in some form an optimization. - Emilio Tomasini (2009)

Moving Beyond Assumptions

Many system developers consider

"I hypothesize that this strategy idea will make money"

to be adequate.

Instead, strive to:

  • understand your business constraints and objectives
  • build a hypothesis for the system
  • build the system in pieces
  • test the system in pieces
  • measure how likely it is that you have overfit

Constraints and Objectives


  • capital available
  • products you can trade
  • execution platform


  • published or synthetic?
  • what are the limitations?
  • are you held to it, or just measured against it?


  • formulate objectives for testability
  • make sure they reflect your real business goals

Building a Hypothesis

To create a testable idea (a hypothesis):

  • formulate a declarative conjecture
  • make sure the conjecture is predictive
  • define the expected outcome
  • describe means of verifying/testing

good/complete Hypothesis Statements include:

  • what is being analyzed (the subject),
  • dependent variable(s) (the result/prediction)
  • independent variables (inputs to the model)
  • the anticipated possible outcomes, including direction or comparison
  • addresses how you will validate or refute each hypothesis

Building Blocks


  • select the instruments to trade
  • categorize market characteristics that are favorable to the strategy


  • quantitative values derived from market data
  • includes all common "technicals"


  • describe the interaction between filters, market data, and indicators
  • can be viewed as a prediction at a point in time


  • make path-dependent actionable decisions

Test the System in Pieces

How to Screw Up Less

Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise. - John Tukey (1962) p. 13


Fail quickly, think deeply, or both?


No matter how beautiful your theory, no matter how clever you are or what your name is, if it disagrees with experiment, it’s wrong. - Richard P. Feynman (1965)

Things to Watch Out For, or, Types of Overfitting

Look Ahead Bias

  • directly using knowledge of future events

Data Mining Bias

  • caused by testing multiple configurations and parameters over multiple runs, with adjustments between backtest runs
  • exhaustive searches may or may not introduce biases

Data Snooping

  • knowledge of the data set can contaminate your choices
  • making changes after failures without having strong experimental design

Measuring Indicators

A good indicator measures something in the market:

  • a theoretical "fair value" price, or
  • the impact of a factor on that price, or
  • turning points, direction, or slope the series

Make sure the indicator is testable:

  • hypothesis and tests for the indicator
    • standard errors and goodness of fit
    • t-tests or p-value
  • custom 'perfect foresight' model built from a periodogram or signature plot

If your indicator doesn't have testable information content, throw it out and start over.

Measuring Signals

Signals make predictions; all the literature on forecasting is applicable:

  • mean squared forecast error, BIC, etc.
  • box plots or additive models for forward expectations
  • "revealed performance" approach of Racine and Parmeter (2009)
  • re-evaluate assumptions about the method of action of the strategy
  • detect information bias or luck before moving on





Measuring Rules

If your signal process doesn't have predictive power, stop now.


  • rules should refine the way the strategy 'listens' to signals
  • entries may be passive or aggressive, or may level or pyramid into a position
  • exits may have their own signal process, or may be derived empirically
  • risk rules should be added near the end, for empirical 'stops' or to meet business constraints

Beware of Rule Burden:

  • having too many rules is an invitation to overfitting
  • adding rules after being disappointed in backtest results is almost certainly an exercise in overfitting (data snooping)
  • strategies with fewer rules are more likely to be robust out of sample

Parameter Optimization




Walk Forward






Measuring the Whole System

Net profit as a sole evaluation method ignores many of the characteristics important to this decision. - Robert Pardo (2008)