updated 12 July 2016

Who is this Guy ?

Proprietary/Principal Trading

Brian Peterson:

Proprietary Trading:

  • proprietary or principal traders are a specific "member" structure with the exchanges
  • high barriers to entry, large up-front capital requirements
  • many strategies pursued in this structure have capacity constraints
  • benefits on the other side are low fees, potentially high leverage
  • money management needs to be very focused on drawdowns, leverage, volatility

R in Finance trade simulation toolchain

Backtesting, art or science?

 

 

Back-testing. I hate it - it's just optimizing over history. You never see a bad back-test. Ever. In any strategy. - Josh Diedesch (2014), CalSTRS

 

 

Every trading system is in some form an optimization. - Emilio Tomasini (2009)

Moving Beyond Assumptions

Many system developers consider

"I hypothesize that this strategy idea will make money"

to be adequate.

Instead, strive to:

  • understand your business constraints and objectives
  • build a hypothesis for the system
  • build the system in pieces
  • test the system in pieces
  • measure how likely it is that you have overfit

Constraints and Objectives

Constraints

  • capital available
  • products you can trade
  • execution platform

Benchmarks

  • published or synthetic?
  • what are the limitations?
  • are you held to it, or just measured against it?

Objectives

  • formulate objectives for testability
  • make sure they reflect your real business goals

Building a Hypothesis

To create a testable idea (a hypothesis):

  • formulate a declarative conjecture
  • make sure the conjecture is predictive
  • define the expected outcome
  • describe means of verifying/testing

good/complete Hypothesis Statements include:

  • what is being analyzed (the subject),
  • dependent variable(s) (the result/prediction)
  • independent variables (inputs to the model)
  • the anticipated possible outcomes, including direction or comparison
  • addresses how you will validate or refute each hypothesis

Building Blocks

Filters

  • select the instruments to trade
  • categorize market characteristics that are favorable to the strategy

Indicators

  • quantitative values derived from market data
  • includes all common "technicals"

Signals

  • describe the interaction between filters, market data, and indicators
  • can be viewed as a prediction at a point in time

Rules

  • make path-dependent actionable decisions

Test the System in Pieces

How to Screw Up Less

Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise. - John Tukey (1962) p. 13

 

Fail quickly, think deeply, or both?

 

No matter how beautiful your theory, no matter how clever you are or what your name is, if it disagrees with experiment, it’s wrong. - Richard P. Feynman (1965)

Things to Watch Out For, or, Types of Overfitting

Look Ahead Bias

  • directly using knowledge of future events

Data Mining Bias

  • caused by testing multiple configurations and parameters over multiple runs, with adjustments between backtest runs
  • exhaustive searches may or may not introduce biases

Data Snooping

  • knowledge of the data set can contaminate your choices
  • making changes after failures without having strong experimental design

Measuring Indicators

A good indicator measures something in the market:

  • a theoretical "fair value" price, or
  • the impact of a factor on that price, or
  • turning points, direction, or slope the series

Make sure the indicator is testable:

  • hypothesis and tests for the indicator
    • standard errors and goodness of fit
    • t-tests or p-value
  • custom 'perfect foresight' model built from a periodogram or signature plot

If your indicator doesn't have testable information content, throw it out and start over.

Measuring Signals

Signals make predictions; all the literature on forecasting is applicable:

  • mean squared forecast error, BIC, etc.
  • box plots or additive models for forward expectations
  • "revealed performance" approach of Racine and Parmeter (2009)
  • re-evaluate assumptions about the method of action of the strategy
  • detect information bias or luck before moving on

 

 

 

quantstrat::distributional.boxplot()

Measuring Rules

If your signal process doesn't have predictive power, stop now.

 

  • rules should refine the way the strategy 'listens' to signals
  • entries may be passive or aggressive, or may level or pyramid into a position
  • exits may have their own signal process, or may be derived empirically
  • risk rules should be added near the end, for empirical 'stops' or to meet business constraints

Beware of Rule Burden:

  • having too many rules is an invitation to overfitting
  • adding rules after being disappointed in backtest results is almost certainly an exercise in overfitting (data snooping)
  • strategies with fewer rules are more likely to be robust out of sample

Parameter Optimization

 

 

PerformanceAnalytics::chart.Histogram

Walk Forward

quantstrat::walk.forward

 

 

 

quantstrat::chart.forward

Measuring the Whole System

Net profit as a sole evaluation method ignores many of the characteristics important to this decision. - Robert Pardo (2008)

blotter::chart.MAE

  • evaluating backtest/simulated trades can tell you more about the dynamics of your system
  • Maximum Adverse/Favorable Excursion (MAE/MFE) looks at how far a trade moves for/against you before it is closed
  • blotter is probably used as often in the real world for post-trade analysis of production trades as it is for backtests

Using Trade Statistics

All trading and backtesting platforms (should) provide trade statistics:

  • number of trades w/ gross and net P&L
  • mean/median, standard deviation of trading P&L per trade
  • percent of positive/negative trades
  • Profit Factor : absolute value ratio of gross profits over gross losses
  • Drawdown statistics
  • start-trade drawdown (Fitschen 2013, 185)
  • win/loss ratios of winning over losing trade P&L (total/mean/median)

Dangers of aggregate statistics:

  • hiding the most common outcomes
  • focusing on extremes
  • not enough trades or history for validity
  • collinearities of overlapping "trades"

see blotter::tradeStats and blotter::perTradeStats

Using Returns

  • Returns create a standard mechanism for comparing multiple strategies or managers
  • Choice of the denominator matters

Sample Analyses:

  • tail risk measures
  • volatility analysis
  • factor analysis / factor model Monte Carlo
  • style analysis
  • comparing strategies in return space
  • applicability to asset allocation

see botter::PortfReturns to extract returns from cash P&L and all of PerformanceAnalytics

Asset Allocation

  • we tend to do asset allocation studies only after strategies are in production.

  • backtests are most often done on 1-lots, and initial scaling is done ad-hoc.

  • strategy daily returns become returns of a synthetic asset (the strategy) as inputs to optimization

  • optimizer should use your business objectives as the portfolio objective

PortfolioAnalytics is used extensively by traditional asset managers, but it was developed to optimize capital allocation for trading strategies

Detecting Backtest Overfitting

Did we over do it?

A big computer, a complex algorithm and a long time does not equal science. - Robert Gentleman

  • White's Reality Check : from White (2000) and Hansen (2005)
  • k-fold cross validation : improves single hold-out model by randomly dividing the sample of size T into sequential sub-samples of size T/k.(Hastie, Tibshirani, and Friedman 2009)
  • CSCV sampling (combinatorially symmetric cross validation): "generate \(S/2\) testing sets of size \(T/2\) by recombining the \(S\) slices of the overall sample of size \(T\) ". (Bailey et al. 2014, 17)
  • Multiple Hypothesis Testing looks at Type I vs Type II error in evaluating backtests and at appropriate haircuts based on these probabilities. (Harvey and Liu 2015 )

Monte Carlo and the bootstrap

Sampling from limited information

  • estimate the 'true' properties of a distribution from incomplete information
  • evaluate the likelihood (test the hypothesis) that a particular result is
  • not the result of chance
  • not overfit
  • understand confidence intervals for other descriptive statistics on the backtest
  • simulated different paths that the results might have taken, if the ordering had been different

see Jasen Mackie's excellent work in blotter::mcsim

History of Monte Carlo and bootstrap simulation

  • Laplace was the first to describe the mathematical properties of sampling from a distribution
  • Mahalanobis extended this work in the 1930's to describe sampling from dependent distributions, and anticipated the block bootstrap by examining these dependencies
  • Monte Carlo simulation was developed by Françoise Ulam and John von Neumann as part of the hydrogen bomb program in 1946 (Richard Rhodes, Dark Sun, p.304)
  • computational implementation of Monte Carlo simulation was constructed by Nicholas Metropolis on the ENIAC and MANIAC machines
  • Metropolis was an author in 1953 of the prior distribution sampler extended by W.K Hastings to the modern Metropolis-Hastings form in 1970
  • Maurice Quenouille and John Tukey developed the modern 'jackknife' in the 1950's
  • Bradley Efron described the modern bootstrap in 1979

Monte Carlo Output

Output of blotter::mcsim may be plotted:

blotter::plot.mcsim

 

 

blotter::hist.mcsim

Conclusion

  • understand the business context you operate in
    • constraints
    • benchmarks
    • objectives
  • separate the components of the strategy
  • construct testable hypotheses at each step of the process
  • evaluate the components separately
  • test yourself often

Work in Progress

  • xts::plot.xts conversions (GSoC 2016, student Eric Hung)

  • enhancements to Monte Carlo

  • additional round turn trade classifications in blotter

  • more functions for detecting and measuring overfitting

  • simulation of random trade entry/exit points that meet the constraints of a given strategy

  • additional post-trade analysis and reconciliation functionality

All of this functionality is a community effort. If you want to get involved, please contact me.

Installing quantstrat

PerformanceAnalytics,PortfolioAnalytics,FinancialInstrument, xts, and quantmod are all on CRAN.

Development for these packages is on github, and

Install devtools

Install PerformanceAnalytics and FinancialInstrument using install.packages

install.packages('FinancialInstrument')
install.packages('PerformanceAnalytics')

Install blotter and quantstrat as well as the current versions of xts and quantmod from github:

require(devtools)
install_github('joshuaulrich/xts')
install_github('joshuaulrich/quantmod')
install_github('braverock/blotter')
install_github('braverock/quantstrat')

Thanks

Thank You for Your Attention

 

Thanks to my team, and my family, who make it possible.

©2015-2016 Brian G. Peterson brian@braverock.com

Code to apply the techniques discussed here may be found in the R blotter, quantstrat, PerformanceAnalytics, and PortfolioAnalytics packages.

The paper this presentation is largely drawn from is available here: http://goo.gl/na4u5d

All views expressed in this presentation are those of Brian Peterson, and do not necessarily reflect the opinions or policies of DV Trading.

All remaining errors or omissions should be attributed to the author.

Resources

Bailey, David H, Jonathan M Borwein, Marcos López de Prado, and Qiji Jim Zhu. 2014. “The Probability of Backtest Overfitting.” http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326253.

Fitschen, Keith. 2013. Building Reliable Trading Systems. John Wiley & Sons, Inc.

Hansen, Peter R. 2005. “A Test for Superior Predictive Ability.” Journal of Business and Economic Statistics.

Harvey, Campbell R., and Yan Liu. 2015. “Backtesting.” SSRN. http://ssrn.com/abstract=2345489.

Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Second Edition. Springer.

Pardo, Robert. 2008. The Evaluation and Optimization of Trading Strategies. Second ed. John Wiley & Sons.

Racine, Jeffrey S, and Christopher F Parmeter. 2009. “Data-Driven Model Evaluation: A Test for Revealed Performance.” Mac Master University. https://editorialexpress.com/cgi-bin/conference/download.cgi?db_name=FEMES09&paper_id=152.

Tomasini, Emilio, and Urban Jaekle. 2009. Trading Systems: A New Approach to System Development and Portfolio Optimisation.

Tukey, John W. 1962. “The Future of Data Analysis.” The Annals of Mathematical Statistics. JSTOR, 1–67. http://projecteuclid.org/euclid.aoms/1177704711.

White, Halbert L. 2000. “System and Method for Testing Prediction Models and/or Entities.” Google Patents. http://www.google.com/patents/US6088676.