liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-10 14:26:56 +08:00
Code Activity

Merge remote-tracking branch 'upstream/high-freq-execution' into high-freq-execution

Browse Source
This commit is contained in:
Yuchen Fang
2021-03-19 14:11:08 +08:00
parent 80aa08215f f9c30f9834
commit e3f463010b
27 changed files with 1401 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

87
examples/trade/README.md
View File

@@ -4,6 +4,91 @@ This is the experiment code for our AAAI 2021 paper "[Universal Trading for Orde
## Abstract
As a fundamental problem in algorithmic trading, order execution aims at fulfilling a specific trading order, either liquidation or acquirement, for a given instrument. Towards effective execution strategy, recent years have witnessed the shift from the analytical view with model-based market assumptions to model-free perspective, i.e., reinforcement learning, due to its nature of sequential decision optimization. However, the noisy and yet imperfect market information that can be leveraged by the policy has made it quite challenging to build up sample efficient reinforcement learning methods to achieve effective order execution. In this paper, we propose a novel universal trading policy optimization framework to bridge the gap between the noisy yet imperfect market states and the optimal action sequences for order execution. Particularly, this framework leverages a policy distillation method that can better guide the learning of the common policy towards practically optimal execution by an oracle teacher with perfect information to approximate the optimal trading strategy. The extensive experiments have shown significant improvements of our method over various strong baselines, with reasonable trading actions.
## Environment Dependencies
### Dependencies
```
gym==0.17.3
torch==1.6.0
numba==0.51.2
numpy==1.19.1
pandas==1.1.3
tqdm==4.50.2
tianshou==0.3.0.post1
env==0.1.0
PyYAML==5.4.1
redis==3.5.3
```
### Environment Variable
`EXP_PATH` Absolute path to your config folder, we give folder `exp` as an example.
`OUTPUT_DIR` Absolute path to your log folder.
## Data Processing
For Feature processing, we take Yahoo dataset as an example, which can be precessed in `qlib/examples/highfreq/workflow.py` file. If you have a need to change your data storage path, you can change the `data_path` in `workflow.py`, and then do the following.
```
python workflow.py
```
For order generation, if you have changed change the the `data_path` in `workflow.py`, change `data_path` in `order_gen.py` again, then do the following.
```
python order_gen.py
```
## Training and backtest
### Config file
Config file is need to start our project, we take `PPO`, `OPDS` and `OPD` as an example in folder `exp/example`. If you want to use our given config, make sure the `data_path` you set before matches the config file.
### Baseline method
To run a method, you can do the following.
```
python main.py --config={config_path}
```
Where `{config_path}` means the relative path from your config.yml to `EXP_PATH`.
If you need to run our given method such as PPO method, you can do the following.
```
python main.py --config=example/PPO/config.yml
```
### OPD method
OPD method is a multi step method, at first you should run OPDT as the teacher in OPD method.
```
python main.py --config=example/OPDT/config.yml
```
After training, find the `policy_best` file in your OPDT log file and copy it to `trade` file for backtest. Also you can change `policy_path` in the `example/OPDT_b/config.yml` to your `policy_best` file. Then run the backtest method.
```
python main.py --config=example/OPDT_b/config.yml
```
then processed feature from teacher. Remember to change `log_path` if you have changed `log_dir` in `OPDT_b/config.yml`.
```
python teacher_feature.py
```
and finally start our OPD method.
```
python main.py --config=example/OPD/config.yml
```
### Citation
You are more than welcome to cite our paper:
```
@@ -13,4 +98,4 @@ You are more than welcome to cite our paper:
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
year={2021}
}
```
```

76
examples/trade/exp/example/OPD/config.yml Normal file
View File

@@ -0,0 +1,76 @@
seed: 42
task: train
log_dir: example/OPD
buffer_size: 80000
io_conf:
test_sampler: TestSampler
train_sampler: Sampler
test_logger: DFLogger
resources:
num_cpus: 24
num_gpus: 1
device: cuda
train_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/train/
valid_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/valid/
test_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/test/
env_conf:
name: StockEnv_Acc
max_step_num: 237
limit: 10
time_interval: 30
interval_num: 8
features:
- name: raw
type: range
loc: ../data/normed_feature/
size: 180
- name: teacher_action
type: interval
size: 1
loc: ../data/feature/teacher/
obs:
name: RuleTeacher
config: {}
action:
name: Static_Action
config:
action_num: 5
action_map: [0, 0.25, 0.5, 0.75, 1]
reward:
VP_Penalty_small_vec:
penalty: 100
coefficient: 1
policy_conf:
name: PPO_sup
config:
discount_factor: 1.
max_grad_norm: 100.
reward_normalization: False
eps_clip: 0.3
value_clip: True
vf_coef: 1.
gae_lambda: 1.
vf_clip_para: 0.3
sup_coef: 0.01
network_conf:
name: OPD
config:
hidden_size: 64
out_shape: 5
fc_size: 32
cnn_shape: [30, 6]
optim:
lr: 1e-4
batch_size: 1024
max_epoch: 30
step_per_epoch: 20
collect_per_step: 10000
repeat_per_collect: 5
early_stopping: 5
weight_decay: 0.

71
examples/trade/exp/example/OPDS/config.yml Normal file
View File

@@ -0,0 +1,71 @@
seed: 42
task: train
log_dir: example/OPDS
buffer_size: 80000
io_conf:
test_sampler: TestSampler
train_sampler: Sampler
test_logger: DFLogger
resources:
num_cpus: 24
num_gpus: 1
device: cuda
train_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/train/
valid_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/valid/
test_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/test/
env_conf:
name: StockEnv_Acc
max_step_num: 237
limit: 10
time_interval: 30
interval_num: 8
features:
- name: raw
type: range
loc: ../data/normed_feature/
size: 180
obs:
name: TeacherObs
config: {}
action:
name: Static_Action
config:
action_num: 5
action_map: [0, 0.25, 0.5, 0.75, 1]
reward:
VP_Penalty_small_vec:
penalty: 100
coefficient: 1
policy_conf:
name: PPO
config:
discount_factor: 1.
max_grad_norm: 100.
reward_normalization: False
eps_clip: 0.3
value_clip: True
vf_coef: 1.
gae_lambda: 1.
vf_clip_para: 0.3
network_conf:
name: PPO
config:
hidden_size: 64
out_shape: 5
fc_size: 32
cnn_shape: [30, 6]
optim:
lr: 1e-4
batch_size: 1024
max_epoch: 30
step_per_epoch: 20
collect_per_step: 10000
repeat_per_collect: 5
early_stopping: 5
weight_decay: 0.

71
examples/trade/exp/example/OPDT/config.yml Normal file
View File

@@ -0,0 +1,71 @@
seed: 42
task: train
log_dir: example/OPDT
buffer_size: 80000
io_conf:
test_sampler: TestSampler
train_sampler: Sampler
test_logger: DFLogger
resources:
num_cpus: 24
num_gpus: 1
device: cuda
train_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/train/
valid_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/valid/
test_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/test/
env_conf:
name: StockEnv_Acc
max_step_num: 237
limit: 10
time_interval: 30
interval_num: 8
features:
- name: raw
type: range
loc: ../data/normed_feature/
size: 180
obs:
name: TeacherObs
config: {}
action:
name: Static_Action
config:
action_num: 5
action_map: [0, 0.25, 0.5, 0.75, 1]
reward:
VP_Penalty_small_vec:
penalty: 100
coefficient: 1
policy_conf:
name: PPO
config:
discount_factor: 1.
max_grad_norm: 100.
reward_normalization: False
eps_clip: 0.3
value_clip: True
vf_coef: 1.
gae_lambda: 1.
vf_clip_para: 0.3
network_conf:
name: Teacher
config:
hidden_size: 64
out_shape: 5
fc_size: 32
cnn_shape: [30, 6]
optim:
lr: 1e-4
batch_size: 1024
max_epoch: 30
step_per_epoch: 20
collect_per_step: 10000
repeat_per_collect: 5
early_stopping: 5
weight_decay: 0.

76
examples/trade/exp/example/OPDT_b/config.yml Normal file
View File

@@ -0,0 +1,76 @@
seed: 42
task: eval
log_dir: example/OPDT_b
buffer_size: 80000
io_conf:
test_sampler: TestSampler
train_sampler: Sampler
test_logger: DFLogger
resources:
num_cpus: 24
num_gpus: 1
device: cuda
train_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/train/
valid_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/valid/
test_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/all/
env_conf:
name: StockEnv_Acc
max_step_num: 237
limit: 10
time_interval: 30
interval_num: 8
features:
- name: raw
type: range
loc: ../data/normed_feature/
size: 180
obs:
name: TeacherObs
config: {}
action:
name: Static_Action
config:
action_num: 5
action_map: [0, 0.25, 0.5, 0.75, 1]
reward:
VP_Penalty_small_vec:
penalty: 100
coefficient: 1
policy_path: policy_best
policy_conf:
name: PPO
config:
discount_factor: 1.
max_grad_norm: 100.
reward_normalization: False
eps_clip: 0.3
value_clip: True
vf_coef: 1.
gae_lambda: 1.
vf_clip_para: 0.3
network_conf:
name: Teacher
config:
hidden_size: 64
out_shape: 5
fc_size: 32
cnn_shape: [30, 6]
optim:
lr: 1e-4
batch_size: 1024
max_epoch: 30
step_per_epoch: 20
collect_per_step: 10000
repeat_per_collect: 5
early_stopping: 5
weight_decay: 0.
search:
optim.weight_decay:
type: choice
value: [0.]

70
examples/trade/exp/example/PPO/config.yml Normal file
View File

@@ -0,0 +1,70 @@
seed: 42
task: train
log_dir: example/PPO
buffer_size: 80000
io_conf:
test_sampler: TestSampler
train_sampler: Sampler
test_logger: DFLogger
resources:
num_cpus: 24
num_gpus: 1
device: cuda
train_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/train/
valid_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/valid/
test_paths:
raw_dir: ../data/backtest/
order_dir: ../data/order/test/
env_conf:
name: StockEnv_Acc
max_step_num: 237
limit: 10
time_interval: 30
interval_num: 8
features:
- name: raw
type: range
loc: ../data/normed_feature/
size: 180
obs:
name: TeacherObs
config: {}
action:
name: Static_Action
config:
action_num: 5
action_map: [0, 0.25, 0.5, 0.75, 1]
reward:
PPO_Reward:
coefficient: 1
policy_conf:
name: PPO
config:
discount_factor: 1.
max_grad_norm: 100.
reward_normalization: False
eps_clip: 0.3
value_clip: True
vf_coef: 1.
gae_lambda: 1.
vf_clip_para: 0.3
network_conf:
name: PPO
config:
hidden_size: 64
out_shape: 5
fc_size: 32
cnn_shape: [30, 6]
optim:
lr: 1e-4
batch_size: 1024
max_epoch: 30
step_per_epoch: 20
collect_per_step: 10000
repeat_per_collect: 5
early_stopping: 5
weight_decay: 0.

2
examples/trade/logger/single_logger.py
View File

@@ -87,7 +87,7 @@ class DFLogger(object):
df_cache[ins] = (
[],
[],
len(pd.read_pickle(order_dir + ins + ".pkl.target")),
(pd.read_pickle(order_dir + ins + ".pkl.target")['amount'] != 0).sum(),
)
df_cache[ins][0].append(df)
df_cache[ins][1].append(res)

59
examples/trade/order_gen.py Normal file
View File

@@ -0,0 +1,59 @@
import numpy as np
import pandas as pd
import os
import time
import datetime
from joblib import Parallel, delayed
data_path = '../data/'
in_dir = os.path.join(data_path, 'backtest/')
### create order folders ####
def generate_order(df, start, end):
# df['date'] = df.index.map(lambda x: x[1].date())
# df.set_index('date', append=True, inplace=True)
df = df.groupby('date').take(range(start, end)).droplevel(level=0)
div = df['$volume0'].rolling((end - start)*60).mean().shift(1).groupby(level='date').transform('first')
order = df.groupby(level=(2, 0)).mean().dropna()
order = pd.DataFrame(order)
order['amount'] = np.random.lognormal(-3.28, 1.14) * order['$volume0']
order['order_type'] = 0
order = order.drop(columns=["$volume0", "$vwap0"])
return order
def w_order(f, start, end):
df = pd.read_pickle(in_dir + f)
#df['date'] = df.index.get_level_values(1).map(lambda x: x.date())
#df = df.set_index('date', append=True, drop=True)
# old_order = pd.read_pickle('../v-zeh/full-07-20/order/ratio_test/' + f)
order = generate_order(df, start, end)
# order = order[order.index.isin(old_order.index)]
order_train = order[order.index.get_level_values(0) < '2020-12-01']
order_test = order[order.index.get_level_values(0) >= '2020-12-01']
order_valid = order_test[order_test.index.get_level_values(0) < '2021-01-01']
order_test = order_test[order_test.index.get_level_values(0) >= '2021-01-01']
if len(order_train) > 0:
train_path = os.path.join(data_path, "order/train/")
if not os.path.exists(train_path):
os.makedirs(train_path)
order_train.to_pickle(train_path + f[:-9] + '.target')
if len(order_valid) > 0:
valid_path = os.path.join(data_path, "order/valid/")
if not os.path.exists(valid_path):
os.makedirs(valid_path)
order_valid.to_pickle(valid_path + f[:-9] + '.target')
if len(order_test) > 0:
test_path = os.path.join(data_path, "order/test/")
if not os.path.exists(test_path):
os.makedirs(test_path)
order_test.to_pickle(test_path + f[:-9] + '.target')
if len(order) > 0:
all_path = os.path.join(data_path, "order/all/")
if not os.path.exists(all_path):
os.makedirs(all_path)
order_test.to_pickle(all_path + f[:-9] + '.target')
return 0
res = Parallel(n_jobs=64)(delayed(w_order)(f, 0, 239) for f in os.listdir(in_dir))
print(sum(res))

24
examples/trade/teacher_feature.py Normal file
View File

@@ -0,0 +1,24 @@
import pandas as pd
import os
data_path = '../data/'
feature_path = os.path.join(data_path, 'feature/teacher/')
if not os.path.exists(feature_path):
os.makedirs(feature_path)
log_file = os.path.join(os.environ.get('OUTPUT_DIR'),'example/OPDT_b/0/test/')
files = os.listdir(log_file)
for f in files:
if f.endswith(".log"):
df = pd.read_pickle(log_file + f)
df['datetime'] = df.index.get_level_values(1).map(lambda x: x[1])
df.set_index('datetime', append=True, drop=True, inplace=True)
action = df['action']
action = action.reset_index(level=1, drop=True)
action.index = action.index.map(lambda x: (x[0], x[1], x[2].time()))
action = action.unstack().iloc[:, ::30] * 2
action = action.fillna(0)
train_action = action.astype("int")
final = train_action
final.to_pickle(feature_path + f[:-4] + '.pkl')