🧬 Elisabeth

A grammar-driven, walk-forward-validated trading strategy discovery engine

Elisabeth doesn't trade. She invents strategies — millions of them — and keeps only the ones that survive out-of-sample.

---

💡 The Idea

Most algotrading tools ask you to write a strategy and then test it. Elisabeth flips this around.

You give her a grammar — a formal description of every legal way to combine indicators, prices, comparisons and boolean logic. She then generates candidate strategies by the thousand, backtests each one against real market data, and ruthlessly filters them through a walk-forward validation gauntlet. What comes out the other side are strategies that didn't just fit the past — they generalised to data they never saw during search.

The trick at the heart of the project: a strategy is a genome, and the grammar is the genetic code. A fixed-length array of integers (codons) is used to deterministically walk the production rules of a Backus–Naur Form grammar, expanding non-terminals until a complete, syntactically-valid trading rule emerges. Change the genome, get a different strategy. Search the genome space, search the strategy space.

((CPRICE low= EMA[100]) AND (RSI[14] hig= 35)) [GO_LONG]
;
((RSI[14] low= 60) OR (CPRICE hig= EMA[100]))    [GO_SHORT]

An example of the generated syntax — a candidate, not a validated winner: go long while price holds above the 100‑EMA and RSI dips through 35; flip short when RSI pushes through 60 or price loses the EMA. Whether a rule like this actually pays after costs is exactly what the validation gauntlet (and the research below) decides — and for classic 1h signals the answer so far is "no".

---

🗺️ How It Works

flowchart LR
    A[BNF Grammar<br/>algotrading_doublerule.bnf] --> B[Genome<br/>128 codons]
    B --> C[BnfRandomStrategyGenerator<br/>grammar expansion]
    C --> D[Strategy Expression<br/>GO_LONG ; GO_SHORT]
    D --> E[PostFixProcessor<br/>shunting-yard]
    E --> F[FinancialStrategyBacktestExecutor<br/>TA4J indicators + rules]
    F --> G{WalkForwardValidator<br/>4 quality gates}
    G -->|passes| H[JSON results<br/>output/]
    G -->|fails| X[discarded]

Loading

The end-to-end pipeline, as orchestrated by ElisabethAlgotradingMain:

1. Load OHLC bar data for each ticker from the Massive.com API (Polygon-compatible REST).
2. Split each series into an in-sample training window and an out-of-sample test window (default 70/30).
3. Generate candidate strategies on the training data — expanding the BNF grammar from random genomes, backtesting each, and keeping those that clear the fitness floor (profit, trade count, win-ratio).
4. Validate every survivor on the test data through walk-forward validation — only strategies that generalise pass.
5. Emit the ranked survivors as structured JSON under output/.

Why walk-forward validation matters

It is trivially easy to find a strategy that looks brilliant on historical data — and worthless going forward. This is overfitting, and it is the single biggest way backtests lie. Elisabeth defends against it by judging each candidate only on data it never touched during search. A strategy passes WalkForwardValidator only when all four gates hold on the test window:

Gate	Default	Meaning
Test profit	≥ 1.0	Must not lose money out-of-sample
Test Sharpe	≥ 0.0	Risk-adjusted return must be non-negative
Max drawdown	≤ 0.50	Cannot bleed more than 50% peak-to-trough
Consistency	≥ 0.3	Test profit must track train profit (no regime-luck)

These gates are necessary, not sufficient. They are computed gross of costs and don't, by themselves, compare a strategy against simply holding the asset — so the research harness also benchmarks every strategy against buy-&-hold and a configurable transaction-cost model (see below). A strategy can clear all four gates and still be closet-beta or cost-negative.

---

🧩 The Grammar Is the Search Space

The whole behaviour of the optimizer is defined by one file: src/main/resources/algotrading_doublerule.bnf. Edit the grammar, and you change what kinds of strategies can exist — no Java changes required.

<trading-rule> ::= (<signal>) [GO_LONG] ; (<signal>) [GO_SHORT]

<signal> ::= <var> <op> <value>
           | <var> <op> <var>
           | (<signal>) AND (<signal>)
           | (<signal>) OR (<signal>)

<op>  ::= low= | hig= | xup= | xdn= | eq=
<var> ::= CPRICE | VOLUME | RSI[<tf>] | EMA[<tf>] | MACD[<tf>,<tf>] | SUPERT[<tf>,<tf>] | ...

• 76 technical indicators are wired in — moving averages (SMA/EMA/DEMA/TEMA/HMA/KAMA/JMA…), oscillators (RSI/CMO/Stochastic/Williams %R…), volume (OBV/MFI/CMF/Klinger…), volatility (ATR/Bollinger/Keltner/Donchian/Squeeze…), trend (ADX/Aroon/Vortex/SuperTrend/Ichimoku…) and more.
• Comparison operators low= / hig= / xup= / xdn= / eq= express crosses and thresholds. ⚠️ Their runtime semantics are inverted relative to their names (low= maps to "over / cross-up", hig= to "under / cross-down") — a documented quirk preserved for backward compatibility.
• Boolean composition (AND / OR) lets signals nest arbitrarily deep, bounded only by genome length.

Genome internals live in core/genomes/: RandomGenome (fixed-size, 128 codons) and DynamicGenome (variable). Codons are read modulo the number of available productions; when the genome is exhausted it wraps, a classic grammatical-evolution technique.

---

🚀 Quick Start

Prerequisites

• JDK 21+
• Maven 3.9+
• A Massive.com API key (set as the MASSIVE_API_KEY environment variable)

Build & test

# Full build — compiles, runs unit tests, installs the jar
mvn clean install

# Unit tests only
mvn test

# A single test class / method
mvn test -Dtest=BnfRandomStrategyGeneratorTest
mvn test -Dtest=BnfRandomStrategyGeneratorTest#testGenerateStrategy

Run the optimizer

export MASSIVE_API_KEY=your_key_here
mvn spring-boot:run

Elisabeth will load data for the configured tickers, hunt for strategies, and drop ranked JSON results into output/. Data is automatically refreshed every 3 hours while the app runs.

Docker

mvn clean install
docker build --build-arg JAR_VERSION=SNAPSHOT -t elisabeth .
docker run -e MASSIVE_API_KEY=your_key_here elisabeth

---

⚙️ Configuration

All knobs live in src/main/resources/application.properties and can be overridden via environment variables:

# --- Market data (Massive.com) ---
massive.api.key=${MASSIVE_API_KEY}
massive.tickers=X:BTCUSD,X:SOLUSD,X:HYPEUSD
massive.timespan=minute          # minute | hour | day
massive.multiplier=30            # e.g. 30 minute bars
massive.history.days=730         # how far back to load

# --- Walk-forward validation ---
validation.train.ratio=0.70      # 70% train / 30% test
validation.test.min.profit=1.0
validation.test.min.sharpe=0.0
validation.test.max.drawdown=0.50
validation.test.min.consistency=0.3
validation.candidates.max=10     # survivors to keep per ticker

# --- Output ---
output.dir=output

Fitness floors used during candidate generation (profit / trade-count / win-ratio) are defined in core/main/Configurator.

Transaction costs: backtests are gross by default. Pass -Dbacktest.fee.per.side=0.001 (0.1% per side) to evaluate strategies net of fees via TA4J's LinearTransactionCostModel. Turnover-heavy strategies that look profitable gross routinely turn negative net — always check the net figure.

---

📦 Output Format

Each run emits a per-ticker JSON summary. Beyond the surviving strategies, it records the walk-forward folds and the buy-&-hold benchmark (bhProfit) for each window — so you can immediately see whether a strategy beat simply holding the asset, and whether it did so across both bull and bear regimes:

{
  "track": "longshort",
  "ticker": "X:BTCUSD",
  "timeframe": "1h",
  "folds": [
    { "fold": 1, "testFrom": "2025-03-18T10:00:00Z", "testBars": 1754, "bhProfit": 1.279, "regime": "bull" },
    { "fold": 3, "testFrom": "2025-08-11T14:00:00Z", "testBars": 1754, "bhProfit": 0.914, "regime": "bear" }
  ],
  "strategies": [
    { "expression": "((CPRICE low= EMA[100]) AND (RSI[14] hig= 35)) [GO_LONG] ; ..." }
  ]
}

---

🏗️ Architecture

Module	Responsibility
core/bnf/	BNF grammar parser + genome-driven strategy generator (@NotThreadSafe; each stream element gets its own generator).
core/genomes/	Genome abstractions (RandomGenome, DynamicGenome). Max codon 128, wrap threshold 8.
core/strategy/processor/	Stateless, thread-safe shunting-yard infix→postfix converter and the typed token hierarchy.
core/FinancialStrategyBacktestExecutor	Evaluates a postfix expression against a TA4J BarSeries, building Rule trees from indicator comparisons.
core/validation/	WalkForwardValidator, MetricsCalculator (Sharpe, drawdown, consistency), ValidationResult.
core/data/	MassiveDataProvider (REST client) + BarSeriesFactory.
core/output/	JsonFileEmitter and the RunResult / StrategyResult records.
core/main/	Spring Boot entry point + runtime-mutable Configurator.

Performance & safety notes

• Candidate generation runs over a Stream, capped at roughly 25 evaluated : 1 kept so a run always terminates even when few candidates pass.
• Each backtest runs in a dedicated daemon thread pool with a 3-second hard timeout — some nested TA4J CachedIndicator chains can stall for minutes, so Elisabeth would rather skip a strategy than block the pipeline.

---

🔬 Research Mode

The integration test FinancialStrategyIntegrationTestIT doubles as a research harness for evaluating hand-authored hypotheses with rolling, multi-fold walk-forward, a buy-&-hold benchmark, net-of-cost evaluation, and a reversible long/short mode. Findings live under research/.

Current finding (BTC 1h): across random search and hand-authored hypotheses, no net-of-cost edge has been found for classic indicator signals — long-only or long/short. The directional signal that exists gross is smaller than realistic transaction costs. See research/btc_1h_findings_v3.md. The hunt continues at higher timeframes and other instruments.

# Integration tests are skipped by default. Enable them (needs MASSIVE_API_KEY):
mvn verify -DskipITs=false

# Run only the authored-hypotheses research tool:
mvn failsafe:integration-test -DskipITs=false \
  -Dit.test='FinancialStrategyIntegrationTestIT#authoredStrategiesWithWalkForward' \
  -Dit.strategies.file=research/btc_1h_hypotheses.txt

Test naming conventions: unit tests use the Test*.java / *Test.java prefix (surefire); integration tests use the *IT.java suffix (failsafe) and live under src/test/.../integration/tests/.

---

🧪 Tech Stack

Layer	Choice
Language	Java 21
Framework	Spring Boot 3.3.7 (Undertow, Actuator)
Technical analysis	TA4J 0.22.6
Build	Maven (surefire + failsafe)
Boilerplate	Lombok
Container	Docker (eclipse-temurin:21-jre-alpine)
Data source	Massive.com REST API (Polygon-compatible)

This codebase was recently modernized from its original Java 8 / Spring Boot 2.7 / TA4J 0.11 incarnation — see docs/superpowers/specs/ for the layer-by-layer migration design.

---

⚠️ Disclaimer

Elisabeth is a research and educational tool. Backtested performance is not indicative of future results, generated strategies are not financial advice, and nothing here should be deployed against real capital without independent validation. Trade at your own risk.

---

Built by Gianluigi Davassi.