mirror of
https://github.com/microsoft/qlib.git
synced 2026-07-10 14:26:56 +08:00
init commit
This commit is contained in:
Young
327
docs/hidden/tuner.rst
Normal file
327
docs/hidden/tuner.rst
Normal file
@@ -0,0 +1,327 @@
|
||||
.. _tuner:
|
||||
|
||||
Tuner
|
||||
===================
|
||||
.. currentmodule:: qlib
|
||||
|
||||
Introduction
|
||||
-------------------
|
||||
|
||||
Welcome to use Tuner, this document is based on that you can use Estimator proficiently and correctly.
|
||||
|
||||
You can find the optimal hyper-parameters and combinations of models, trainers, strategies and data labels.
|
||||
|
||||
The usage of program `tuner` is similar with `estimator`, you need provide the URL of the configuration file.
|
||||
The `tuner` will do the following things:
|
||||
|
||||
- Construct tuner pipeline
|
||||
- Search and save best hyper-parameters of one tuner
|
||||
- Search next tuner in pipeline
|
||||
- Save the global best hyper-parameters and combination
|
||||
|
||||
Each tuner is consisted with a kind of combination of modules, and its goal is searching the optimal hyper-parameters of this combination.
|
||||
The pipeline is consisted with different tuners, it is aim at finding the optimal combination of modules.
|
||||
|
||||
The result will be printed on screen and saved in file, you can check the result in your experiment saving files.
|
||||
|
||||
Example
|
||||
~~~~~~~
|
||||
|
||||
Let's see an example,
|
||||
|
||||
First make sure you have the latest version of `qlib` installed.
|
||||
|
||||
Then, you need to privide a configuration to setup the experiment.
|
||||
We write a simple configuration example as following,
|
||||
|
||||
.. code-block:: YAML
|
||||
|
||||
experiment:
|
||||
name: tuner_experiment
|
||||
tuner_class: QLibTuner
|
||||
qlib_client:
|
||||
auto_mount: False
|
||||
logging_level: INFO
|
||||
optimization_criteria:
|
||||
report_type: model
|
||||
report_factor: model_score
|
||||
optim_type: max
|
||||
tuner_pipeline:
|
||||
-
|
||||
model:
|
||||
class: SomeModel
|
||||
space: SomeModelSpace
|
||||
trainer:
|
||||
class: RollingTrainer
|
||||
strategy:
|
||||
class: TopkAmountStrategy
|
||||
space: TopkAmountStrategySpace
|
||||
max_evals: 2
|
||||
|
||||
time_period:
|
||||
rolling_period: 360
|
||||
train_start_date: 2005-01-01
|
||||
train_end_date: 2014-12-31
|
||||
validate_start_date: 2015-01-01
|
||||
validate_end_date: 2016-06-30
|
||||
test_start_date: 2016-07-01
|
||||
test_end_date: 2018-04-30
|
||||
data:
|
||||
class: ALPHA360
|
||||
provider_uri: /data/qlib
|
||||
args:
|
||||
start_date: 2005-01-01
|
||||
end_date: 2018-04-30
|
||||
dropna_label: True
|
||||
dropna_feature: True
|
||||
filter:
|
||||
market: csi500
|
||||
filter_pipeline:
|
||||
-
|
||||
class: NameDFilter
|
||||
module_path: qlib.data.filter
|
||||
args:
|
||||
name_rule_re: S(?!Z3)
|
||||
fstart_time: 2018-01-01
|
||||
fend_time: 2018-12-11
|
||||
-
|
||||
class: ExpressionDFilter
|
||||
module_path: qlib.data.filter
|
||||
args:
|
||||
rule_expression: $open/$factor<=45
|
||||
fstart_time: 2018-01-01
|
||||
fend_time: 2018-12-11
|
||||
backtest:
|
||||
normal_backtest_args:
|
||||
verbose: False
|
||||
limit_threshold: 0.095
|
||||
account: 500000
|
||||
benchmark: SH000905
|
||||
deal_price: vwap
|
||||
long_short_backtest_args:
|
||||
topk: 50
|
||||
|
||||
Next, we run the following command, and you can see:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
~/v-yindzh/Qlib/cfg$ tuner -c tuner_config.yaml
|
||||
|
||||
Searching params: {'model_space': {'colsample_bytree': 0.8870905643607678, 'lambda_l1': 472.3188735122233, 'lambda_l2': 92.75390994877243, 'learning_rate': 0.09741751430635413, 'loss': 'mse', 'max_depth': 8, 'num_leaves': 160, 'num_threads': 20, 'subsample': 0.7536051584789751}, 'strategy_space': {'buffer_margin': 250, 'topk': 40}}
|
||||
...
|
||||
(Estimator experiment screen log)
|
||||
...
|
||||
Searching params: {'model_space': {'colsample_bytree': 0.6667379039007301, 'lambda_l1': 382.10698024977904, 'lambda_l2': 117.02506488151757, 'learning_rate': 0.18514539615228137, 'loss': 'mse', 'max_depth': 6, 'num_leaves': 200, 'num_threads': 12, 'subsample': 0.9449255686969292}, 'strategy_space': {'buffer_margin': 200, 'topk': 30}}
|
||||
...
|
||||
(Estimator experiment screen log)
|
||||
...
|
||||
Local best params: {'model_space': {'colsample_bytree': 0.6667379039007301, 'lambda_l1': 382.10698024977904, 'lambda_l2': 117.02506488151757, 'learning_rate': 0.18514539615228137, 'loss': 'mse', 'max_depth': 6, 'num_leaves': 200, 'num_threads': 12, 'subsample': 0.9449255686969292}, 'strategy_space': {'buffer_margin': 200, 'topk': 30}}
|
||||
Time cost: 489.87220 | Finished searching best parameters in Tuner 0.
|
||||
Time cost: 0.00069 | Finished saving local best tuner parameters to: tuner_experiment/estimator_experiment/estimator_experiment_0/local_best_params.json .
|
||||
Searching params: {'data_label_space': {'labels': ('Ref($vwap, -2)/Ref($vwap, -1) - 2',)}, 'model_space': {'input_dim': 158, 'lr': 0.001, 'lr_decay': 0.9100529502185579, 'lr_decay_steps': 162.48901403763966, 'optimizer': 'gd', 'output_dim': 1}, 'strategy_space': {'buffer_margin': 300, 'topk': 35}}
|
||||
...
|
||||
(Estimator experiment screen log)
|
||||
...
|
||||
Searching params: {'data_label_space': {'labels': ('Ref($vwap, -2)/Ref($vwap, -1) - 1',)}, 'model_space': {'input_dim': 158, 'lr': 0.1, 'lr_decay': 0.9882802970847494, 'lr_decay_steps': 164.76742865207729, 'optimizer': 'adam', 'output_dim': 1}, 'strategy_space': {'buffer_margin': 250, 'topk': 35}}
|
||||
...
|
||||
(Estimator experiment screen log)
|
||||
...
|
||||
Local best params: {'data_label_space': {'labels': ('Ref($vwap, -2)/Ref($vwap, -1) - 1',)}, 'model_space': {'input_dim': 158, 'lr': 0.1, 'lr_decay': 0.9882802970847494, 'lr_decay_steps': 164.76742865207729, 'optimizer': 'adam', 'output_dim': 1}, 'strategy_space': {'buffer_margin': 250, 'topk': 35}}
|
||||
Time cost: 550.74039 | Finished searching best parameters in Tuner 1.
|
||||
Time cost: 0.00023 | Finished saving local best tuner parameters to: tuner_experiment/estimator_experiment/estimator_experiment_1/local_best_params.json .
|
||||
Time cost: 1784.14691 | Finished tuner pipeline.
|
||||
Time cost: 0.00014 | Finished save global best tuner parameters.
|
||||
Best Tuner id: 0.
|
||||
You can check the best parameters at tuner_experiment/global_best_params.json.
|
||||
|
||||
|
||||
Finally, you can check the results of your experiment in the given path.
|
||||
|
||||
Configuration file
|
||||
------------------
|
||||
|
||||
Before using `tuner`, you need to prepare a configuration file. Next we will show you how to prepare each part of the configuration file.
|
||||
|
||||
About the experiment
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
First, your configuration file needs to have a field about the experiment, whose key is `experiment`, this field and its contents determine the saving path and tuner class.
|
||||
|
||||
Usually it should contain the following content:
|
||||
|
||||
.. code-block:: YAML
|
||||
|
||||
experiment:
|
||||
name: tuner_experiment
|
||||
tuner_class: QLibTuner
|
||||
|
||||
Also, there are some optional fields. The meaning of each field is as follows:
|
||||
|
||||
- `name`
|
||||
The experiment name, str type, the program will use this experiment name to construct a directory to save the process of the whole experiment and the results. The default value is `tuner_experiment`.
|
||||
|
||||
- `dir`
|
||||
The saving path, str type, the program will construct the experiment directory in this path. The default value is the path where configuration locate.
|
||||
|
||||
- `tuner_class`
|
||||
The class of tuner, str type, must be an already implemented model, such as `QLibTuner` in `qlib`, or a custom tuner, but it must be a subclass of `qlib.contrib.tuner.Tuner`, the default value is `QLibTuner`.
|
||||
|
||||
- `tuner_module_path`
|
||||
The module path, str type, absolute url is also supported, indicates the path of the implementation of tuner. The default value is `qlib.contrib.tuner.tuner`
|
||||
|
||||
About the optimization criteria
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You need to designate a factor to optimize, for tuner need a factor to decide which case is better than other cases.
|
||||
Usually, we use the result of `estimator`, such as backtest results and the score of model.
|
||||
|
||||
This part needs contain these fields:
|
||||
|
||||
.. code-block:: YAML
|
||||
|
||||
optimization_criteria:
|
||||
report_type: model
|
||||
report_factor: model_pearsonr
|
||||
optim_type: max
|
||||
|
||||
- `report_type`
|
||||
The type of the report, str type, determines which kind of report you want to use. If you want to use the backtest result type, you can choose `pred_long`, `pred_long_short`, `pred_short`, `sub_bench` and `sub_cost`. If you want to use the model result type, you can only choose `model`.
|
||||
|
||||
- `report_factor`
|
||||
The factor you want to use in the report, str type, determines which factor you want to optimize. If your `report_type` is backtest result type, you can choose `annual`, `sharpe`, `mdd`, `mean` and `std`. If your `report_type` is model result type, you can choose `model_score` and `model_pearsonr`.
|
||||
|
||||
- `optim_type`
|
||||
The optimization type, str type, determines what kind of optimization you want to do. you can minimize the factor or maximize the factor, so you can choose `max`, `min` or `correlation` at this field.
|
||||
Note: `correlation` means the factor's best value is 1, such as `model_pearsonr` (a corraltion coefficient).
|
||||
|
||||
If you want to process the factor or you want fetch other kinds of factor, you can override the `objective` method in your own tuner.
|
||||
|
||||
About the tuner pipeline
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The tuner pipeline contains different tuners, and the `tuner` program will process each tuner in pipeline. Each tuner will get an optimal hyper-parameters of its specific combination of modules. The pipeline will contrast the results of each tuner, and get the best combination and its optimal hyper-parameters. So, you need to configurate the pipeline and each tuner, here is an example:
|
||||
|
||||
.. code-block:: YAML
|
||||
|
||||
tuner_pipeline:
|
||||
-
|
||||
model:
|
||||
class: SomeModel
|
||||
space: SomeModelSpace
|
||||
trainer:
|
||||
class: RollingTrainer
|
||||
strategy:
|
||||
class: TopkAmountStrategy
|
||||
space: TopkAmountStrategySpace
|
||||
max_evals: 2
|
||||
|
||||
Each part represents a tuner, and its modules which are to be tuned. Space in each part is the hyper-parameters' space of a certain module, you need to create your searching space and modify it in `/qlib/contrib/tuner/space.py`. We use `hyperopt` package to help us to construct the space, you can see the detail of how to use it in https://github.com/hyperopt/hyperopt/wiki/FMin .
|
||||
|
||||
- model
|
||||
You need to provide the `class` and the `space` of the model. If the model is user's own implementation, you need to privide the `module_path`.
|
||||
|
||||
- trainer
|
||||
You need to proveide the `class` of the trainer. If the trainer is user's own implementation, you need to privide the `module_path`.
|
||||
|
||||
- strategy
|
||||
You need to provide the `class` and the `space` of the strategy. If the strategy is user's own implementation, you need to privide the `module_path`.
|
||||
|
||||
- data_label
|
||||
The label of the data, you can search which kinds of labels will lead to a better result. This part is optional, and you only need to provide `space`.
|
||||
|
||||
- max_evals
|
||||
Allow up to this many function evaluations in this tuner. The default value is 10.
|
||||
|
||||
If you don't want to search some modules, you can fix their spaces in `space.py`. We will not give the default module.
|
||||
|
||||
About the time period
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You need to use the same dataset to evaluate your different `estimator` experiments in `tuner` experiment. Two experiments using different dataset are uncomparable. You can specify `time_period` through the configuration file:
|
||||
|
||||
.. code-block:: YAML
|
||||
|
||||
time_period:
|
||||
rolling_period: 360
|
||||
train_start_date: 2005-01-01
|
||||
train_end_date: 2014-12-31
|
||||
validate_start_date: 2015-01-01
|
||||
validate_end_date: 2016-06-30
|
||||
test_start_date: 2016-07-01
|
||||
test_end_date: 2018-04-30
|
||||
|
||||
- `rolling_period`
|
||||
The rolling period, integer type, indicates how many time steps need rolling when rolling the data. The default value is `60`. If you use `RollingTrainer`, this config will be used, or it will be ignored.
|
||||
|
||||
- `train_start_date`
|
||||
Training start time, str type.
|
||||
|
||||
- `train_end_date`
|
||||
Training end time, str type.
|
||||
|
||||
- `validate_start_date`
|
||||
Validation start time, str type.
|
||||
|
||||
- `validate_end_date`
|
||||
Validation end time, str type.
|
||||
|
||||
- `test_start_date`
|
||||
Test start time, str type.
|
||||
|
||||
- `test_end_date`
|
||||
Test end time, str type. If `test_end_date` is `-1` or greater than the last date of the data, the last date of the data will be used as `test_end_date`.
|
||||
|
||||
About the data and backtest
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
`data` and `backtest` are all same in the whole `tuner` experiment. Different `estimator` experiments must use the same data and backtest method. So, these two parts of config are same with that in `estimator` configuration. You can see the precise defination of these parts in `estimator` introduction. We only provide an example here.
|
||||
|
||||
.. code-block:: YAML
|
||||
|
||||
data:
|
||||
class: ALPHA360
|
||||
provider_uri: /data/qlib
|
||||
args:
|
||||
start_date: 2005-01-01
|
||||
end_date: 2018-04-30
|
||||
dropna_label: True
|
||||
dropna_feature: True
|
||||
feature_label_config: /home/v-yindzh/v-yindzh/QLib/cfg/feature_config.yaml
|
||||
filter:
|
||||
market: csi500
|
||||
filter_pipeline:
|
||||
-
|
||||
class: NameDFilter
|
||||
module_path: qlib.filter
|
||||
args:
|
||||
name_rule_re: S(?!Z3)
|
||||
fstart_time: 2018-01-01
|
||||
fend_time: 2018-12-11
|
||||
-
|
||||
class: ExpressionDFilter
|
||||
module_path: qlib.filter
|
||||
args:
|
||||
rule_expression: $open/$factor<=45
|
||||
fstart_time: 2018-01-01
|
||||
fend_time: 2018-12-11
|
||||
backtest:
|
||||
normal_backtest_args:
|
||||
verbose: False
|
||||
limit_threshold: 0.095
|
||||
account: 500000
|
||||
benchmark: SH000905
|
||||
deal_price: vwap
|
||||
long_short_backtest_args:
|
||||
topk: 50
|
||||
|
||||
Experiment Result
|
||||
-----------------
|
||||
|
||||
All the results are stored in experiment file directly, you can check them directly in the corresponding files.
|
||||
What we save are as following:
|
||||
|
||||
- Global optimal parameters
|
||||
- Local optimal parameters of each tuner
|
||||
- Config file of this `tuner` experiment
|
||||
- Every `estimator` experiments result in the process
|
||||
|
||||
Reference in New Issue
Block a user