This repository contains the code for a master's thesis on Signature-Q-Learning, a novel Q-learning variant for history-dependent environments, applied to an algorithmic order execution problem. It includes the custom ABIDES-based gym environment, training/testing scripts, and Jupyter notebooks to reproduce all results.
Most reinforcement learning (RL) methods assume that the environment follows Markov Decision Process (MDP) dynamics, where the current state is sufficient for learning an optimal policy. Many real-world problems, however, exhibit history-dependent characteristics—transitions and rewards depend on past states and actions—rendering standard MDP-based methods inadequate. The thesis addresses this challenge by introducing History-Based Decision Processes (HDPs) as a formal modelling framework and proposing Signature-Q-Learning, a model-free, online, off-policy algorithm that approximates optimal Q-functions as linear functionals of path signatures of observation-action histories. This enables efficient learning in non-Markovian environments without requiring explicit knowledge of the transition dynamics. The algorithm is evaluated in two numerical experiments; this repository covers the financial execution experiment based on the ABIDES market simulator.
The path signature is a mathematical object from rough path theory, originally introduced by Chen, that encodes the sequential structure of a multidimensional path as an ordered collection of iterated integrals. Truncated (finite-depth) signatures are finite-dimensional and enjoy a universal approximation property: linear functionals of signatures can approximate any continuous function of paths arbitrarily well. This makes them a powerful and compact feature map for sequential data.
In the Signature-Q-Learning algorithm, the full observation-action history is treated as a path and its (truncated) signature serves as the feature vector for Q-function approximation. The signature captures temporal dependencies in a principled way, and Chen's identity enables incremental signature updates whenever a new observation arrives—avoiding full recomputation and keeping the method computationally efficient.
The environment models a simple algorithmic order execution problem implemented as a custom OpenAI Gym environment on top of the ABIDES market simulator.
Setup. The agent starts with a positive stock inventory (bounded by max_inventory = 1000) and must reduce it to (near) zero by the end of the trading session. The underlying market is generated by ABIDES using the rmsc04 configuration, which consists of 1 Exchange agent, 2 POV Market Maker agents, 102 Value agents, 12 Momentum agents, and 1000 Noise agents—providing rich, realistic market dynamics. Each episode covers 35 minutes of simulated trading. The agent first wakes up 5 minutes into the session (to allow the order book to populate), and then every 10 seconds for T = 180 steps until the session ends.
Observation Space. At each step the agent receives a 2-dimensional observation vector:
| Feature | Description | Range |
|---|---|---|
remaining_time_pct |
Normalised remaining time: 1 − t/T |
[0, 1] |
inventory_pct |
Current inventory as fraction of max: i_t / i_max |
[−1, 1] |
The environment is partially observable: mid-price movements and order flow drive both transitions and rewards but are hidden from the agent.
Action Space. Discrete, 3 actions:
| Action | Description |
|---|---|
0 |
Post limit buy order at best bid price (fixed size order_fixed_size) |
1 |
Post limit sell order at best ask price (fixed size order_fixed_size) |
2 |
Do nothing |
Reward. At each step t the running reward is
R_t = Δm_{t+1} * [ (ρ/2 / i_max)² − (i_{t+1} / i_max)² ]
where Δm_{t+1} is the absolute mid-price change and ρ is the fixed order size. This penalises large absolute inventory values and rewards keeping holdings near zero. An additional terminal reward at episode end reinforces ending with near-zero inventory.
Goal. Learn a policy that maximises cumulative reward by liquidating the initial inventory as quickly as possible and then refraining from further trading for the remainder of the session.
Baseline. An approximately optimal immediate liquidation baseline policy was implemented and evaluated over 2000 episodes. It achieved a mean episode reward of −0.0844 (95% CI: −0.0876 to −0.0812) with near-zero mean terminal inventory, providing a clear performance benchmark.
Training. Signature-Q-Learning was trained for 3000 episodes across 10 independent runs. All runs exhibited robust and stable convergence: episode rewards rose quickly during the first third of training and stabilised near the baseline level, losses converged to near zero, and average terminal inventory converged to near zero—indicating that the agent successfully learned to liquidate its position.
Testing. When the learned Q-functions were evaluated on held-out episodes, the first-observation values v₀* (an estimate of the true optimal value function) converged to the baseline mean reward of −0.0844, with 7 out of 10 runs falling within the 95% confidence interval of the baseline. This provides numerical evidence that Signature-Q-Learning successfully approximated the optimal Q-function in this non-Markovian environment.
.
├── src/ # Python source modules
│ ├── abides_gym_custom_execution_environment.py # Custom Gym environment
│ ├── train_execution.py # Training loop
│ ├── test_execution.py # Testing loop
│ ├── baseline_execution.py # Baseline policy runner
│ ├── qfunctions.py # Q-function architectures (Sig, RNN, LSTM)
│ ├── utils.py # Utilities
│ └── plotting_utils.py # Plotting helpers
├── notebooks/ # Jupyter notebooks for running and analysis
│ ├── execution_training_testing.ipynb # Train and test the agent
│ ├── execution_results_analysis.ipynb # Analyse and visualise results
│ └── decay_exploration.ipynb # Exploration decay schedule analysis
├── results/ # Saved result files (*.pkl.gz), named with date ID
├── figures/ # Saved figures (*.png), named with date ID
├── abides-jpmc-public/ # ABIDES submodule
├── requirements.txt
├── install.sh
└── README.md
To reproduce the results, run the notebooks in notebooks/ in order. Training and testing results are saved to results/ and figures to figures/, both with a unique date-based identifier (e.g. _20250127_A).
Python version. Python 3.8 is required. The signatory package used for path signature computations is only compatible with Python 3.8 (on macOS).
Submodule. This repository includes ABIDES as a Git submodule. Use --recursive when cloning, or initialise it afterwards:
git submodule update --init --recursiveSteps.
-
Clone the repository:
git clone --recursive https://github.com/phaelicks/SignatureQLearning-Execution.git cd SignatureQLearning-Execution -
Create and activate a Python 3.8 virtual environment:
python3.8 -m venv venv source venv/bin/activate -
Run the install script (installs all requirements and the ABIDES packages):
bash install.sh
Key dependencies: gym, numpy, torch==1.6.0, signatory==1.2.3.1.6.0, matplotlib, jupyter, ray[rllib]==1.8.0, scipy, pandas, and the ABIDES packages (abides-core, abides-markets, abides-gym).