Video summary

How I Develop Trading Strategies | Permutation Tests and Trading Strategy Development with Python

Main summary

Key takeaways

Technology

Summary of Technological Concepts & Workflow (Trading Strategy Development with Python)

The video presents a generic 4-step workflow for developing and validating trading strategies in Python using permutation / Monte Carlo tests to detect overfitting and data-mining bias. The workflow is designed to be applicable to most strategy types.

1) Create Bar-Level Strategy Returns (Stable Objective Function Foundation)

Early example strategy: a moving average crossover.

This step converts trading logic into a position vector per bar:

  • 1 = long
  • 0 = no position (or analogous encoding)

Then it computes bar-attributable strategy returns:

  • Calculate close-to-close returns
  • Shift returns forward one bar
  • Multiply shifted returns by the position signal to attribute P&L to each bar

Why bar-level returns? Trade-level metrics are less stable. Bar-level returns support more stable objective functions such as:

  • Profit factor
  • Sharpe ratio

2) Optimize an Example Strategy Using In-Sample Performance

The moving average crossover is replaced with the Donchian channel breakout strategy:

  • Long when the current close is the highest over a lookback window
  • Short when the current close is the lowest over the lookback window

This requires a lookback parameter, optimized via grid search over possible lookbacks.

Example optimization:

  • Data: hourly Bitcoin, 2016–2019
  • Best lookback found: 19
  • Best in-sample profit factor ≈ 1.08

The video emphasizes interpretation of in-sample results, framed as questions like:

  • “Is it excellent?”
  • “Is it obviously overfit?” (e.g., implausibly perfect win rates or patterns that look too good)

3) In-Sample Monte Carlo Permutation Test (Reject “Strategy is Garbage” Null)

Goal: determine whether strong in-sample performance comes from real signal or from data-mining / selection bias created by optimization.

Null hypothesis

  • The strategy idea is worthless (“garbage”).

Method

  1. Permute the price path while preserving statistical properties of relative intrabar structure, but destroying meaningful temporal patterns.
  2. Re-optimize the strategy on each permuted dataset.
  3. Compare objective values from:
    • the real data
    • the permuted datasets

Permutation algorithm (high-level)

  • Operates on OHLC(V) inputs (OHLC used in the example)
  • Computes log-relative prices per bar relative to the bar open
  • Shuffles relative intrabar components (“gaps” handled separately)
  • Reconstructs permuted OHLC by combining:
    • shuffled relative values
    • opens derived from prior-close structure
  • Preserves certain endpoints / trend structure while scrambling the path

Outcome (Donchian breakout)

  • With 1,000 permutations, only a few matches achieved profit factor as good as the real strategy
  • Reported p-value ≈ 0.3%
  • Interpreted as a pass, suggesting real patterns exist beyond mining bias

Interpreting p-values here: “How often would random/permuted versions look this good if the strategy were garbage?”

4) Walk-Forward Test + Walk-Forward Permutation Test (Future Generalization)

Even if in-sample results look good, the strategy must survive out-of-sample walk-forward testing.

Walk-forward optimization/testing procedure

  • Uses rolling training and periodic re-optimization:
    • Training window: 4 years
    • Refit frequency: every 30 days (chosen for practical performance)
  • Produces walk-forward trading signals for 2020

Walk-forward result (Donchian breakout):

  • Profit factor ≈ 1.04
  • Expected to be worse than in-sample

Walk-forward permutation test (detect luck from future data)

This adapts the permutation logic to focus on future segments:

  • Permutes data only after the first training fold
  • Rationale: if the strategy is worthless, walk-forward performance on permuted “future” data may still look decent by chance

Outcome:

  • With 200 permutations, reported p-value ≈ 22%
  • Interpreted as not convincing (high chance of success occurring via luck)

Practical decision

The speaker would not trade the Donchian strategy with optimized lookback given these results, because they suggest it likely “sucks.”

Overfitting Demonstration (Decision Tree Example)

A deliberately overfit example uses a decision tree classifier:

  • Target: whether price goes up or down over the next 24 hours
  • Features: three basic indicators derived from price differences
  • Tree settings encourage overfitting:
    • minimum samples per leaf set very low

After training:

  • Predictions generate long/short positions
  • Profit factor is computed

Result:

  • In-sample performance looks overly good
  • In-sample permutation testing rejects the model (it performs similarly on permuted data)
  • Conclusion: the performance is driven by overfitting / data-mining bias

Guidance / Principles Emphasized

Use stable objective functions

  • Prefer bar-level returns for objective calculations (e.g., profit factor, Sharpe)

Use permutation tests to reduce false confidence

Permutation tests help avoid:

  • Data mining bias (optimizing on one dataset)
  • Selection bias (reusing validation and choosing the best after many tries)

Decision thresholds (rule-of-thumb)

The video suggests aiming for low p-values:

  • In-sample:
    • aiming for < 1% (speaker’s guidance)
  • Walk-forward:
    • more lenient, but still expects low p
    • example guidance:
      • ~5% for a single year is “maybe acceptable”
      • stricter if multiple out-of-sample years

Don’t treat p-values as “targets” to game

Tests can be manipulated if you continually adjust strategies to force a pass.

Main Sources / Speakers

  • Main speaker: the video creator (name not provided in subtitles)
  • Referenced book: Permutation and Randomization Tests for Trading System Development — Timothy Masters (PhD in statistics; described as highly authoritative)

Original video