liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-03 11:00:57 +08:00
Code Activity

Update part of the docs

Browse Source
This commit is contained in:
Jactus
2020-11-24 23:56:16 +08:00
parent 5059bba51e
commit b4671746c2
30 changed files with 902 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/component/backtest.rst
View File

@@ -1,4 +1,5 @@
.. _backtest:
============================================
Intraday Trading: Model&Strategy Testing
============================================

1
docs/component/data.rst
View File

@@ -1,4 +1,5 @@
.. _data:
================================
Data Layer: Data Framework&Usage
================================

1
docs/component/model.rst
View File

@@ -1,4 +1,5 @@
.. _model:
============================================
Interday Model: Model Training & Prediction
============================================

409
docs/component/recorder.rst Normal file
View File

@@ -0,0 +1,409 @@
.. _recorder:
====================================
Qlib Recorder: Experiment Management
====================================
.. currentmodule:: qlib
Introduction
===================
``Qlib`` contains an experiment management system named ``QlibRecorder``, which is designed to help users handle experiment and analysis results in an efficient way.
There are three components of the system:
- `ExperimentManager`
a class that manages experiments.
- `Experiment`
a class of experiment, and each instance of it is responsible for a single experiment.
- `Recorder`
a class of recorder, and each instance of it is responsible for a single run.
Here is a general view of the structure of the system:
.. code-block::
ExperimentManager
- Experiment 1
- Recorder 1
- Recorder 2
- ...
- Experiment 2
- Recorder 1
- Recorder 2
- ...
- ...
Currently, the components of this experiment management system are implemented using the machine learning platform: ``MLFlow`` (`link <https://mlflow.org/>`_).
Qlib Recorder
===================
``QlibRecorder`` provides a high level API for users to use the experiment management system. The interfaces are wrapped in the variable ``R`` in ``Qlib``, and users can directly use ``R`` to interact with the system. The following command shows how to import ``R`` in Python:
.. code-block:: Python
from qlib.workflow import R
``QlibRecorder`` includes several common API for managing `experiments` and `recorders` within a workflow. For more available APIs, please refer to the following section about `Experiment Manager`, `Experiment` and `Recorder`.
Here are the available interfaces of ``QlibRecorder``:
- `__init__(exp_manager)`
- Initialization.
- It takes in an input: `exp_manager`, which is an `ExperimentManager` instance. The instance will be created during ``qlib.init``.
- `start(experiment_name=None, recorder_name=None)`
- High level API to start an experiment. This method can only be called within a Python's '`with`' statement.
- Parameters:
- `experiment_name` : str
name of the experiment one wants to start.
- `recorder_name` : str
name of the recorder under the experiment one wants to start.
- Use case:
.. code-block:: Python
with R.start('test', 'recorder_1'):
model.fit(dataset)
R.log...
... # further operations
- `start_exp(experiment_name=None, recorder_name=None, uri=None)`
- Lower level method for starting an experiment. When use this method, one should end the experiment manually and the status of the recorder may not be handled properly.
- Parameters:
- `experiment_name` : str
the name of the experiment to be started
- `recorder_name` : str
name of the recorder under the experiment one wants to start.
- `uri` : str
the tracking uri of the experiment, where all the artifacts/metrics etc. will be stored.
The default uri are set in the qlib.config.
- Returns:
- an experiment instance being started.
- Use case:
.. code-block:: Python
R.start_exp(experiment_name='test', recorder_name='recorder_1')
... # further operations
R.end_exp('FINISHED') or R.end_exp(Recorder.STATUS_S)
- `end_exp(recorder_status=Recorder.STATUS_FI)`
- Method for ending an experiment manually. It will end the current active experiment, as well as its active recorder with the specified `status` type.
- Parameters:
- `status` : str
The status of a recorder, which can be '`SCHEDULED`', '`RUNNING`', '`FINISHED`', '`FAILED`'.
- Use case:
.. code-block:: Python
R.start_exp(experiment_name='test')
... # further operations
R.end_exp('FINISHED') or R.end_exp(Recorder.STATUS_S)
- `search_records(experiment_ids, **kwargs)`
- Get a pandas DataFrame of all the records that have been stored with the given search criteria. This method is highly correlated with MLFlow's ``search_runs`` method (`link <https://www.mlflow.org/docs/latest/python_api/mlflow.html#mlflow.search_runs>`_).
- Parameters:
- `experiment_ids` : list
list of experiment IDs.
- `filter_string` : str
filter query string, defaults to searching all runs.
- `run_view_type` : int
one of enum values ACTIVE_ONLY (1), DELETED_ONLY (2), or ALL (3).
- `max_results` : int
the maximum number of runs to put in the dataframe.
- `order_by` : list
list of columns to order by (e.g., “metrics.rmse”).
- Returns:
- A pandas.DataFrame of records, where each metric, parameter, and tag are expanded into their own columns named metrics.*, params.*, and tags.* respectively. For records that don't have a particular metric, parameter, or tag, their value will be (NumPy) Nan, None, or None respectively.
- Use case:
.. code-block:: Python
R.log_metrics(m=2.50, step=0)
records = R.search_runs([experiment_id], order_by=["metrics.m DESC"])
- `list_experiments()`
- Method for listing all the existing experiments (except for those being deleted.)
- Returns:
- A dictionary (name -> experiment) of experiments information that being stored.
- Use case:
.. code-block:: Python
exps = R.list_experiments()
- `list_recorders(experiment_id=None, experiment_name=None)`
- Method for listing all the recorders of experiment with given id or name. If user doesn't provide the id or name of the experiment, this method will try to retrieve the default experiment and list all the recorders of the default experiment. If the default experiment doesn't exist, the method will first create the default experiment, and then create a new recorder under it.
- Parameters:
- `experiment_id` : str
id of the experiment.
- `experiment_name` : str
name of the experiment.
- Returns:
- A dictionary (id -> recorder) of recorder information that being stored.
- Use case:
.. code-block:: Python
recorders = R.list_recorders(experiment_name='test')
- `get_exp(experiment_id=None, experiment_name=None, create: bool = True)`
- Method for retrieving an experiment with given id or name. Once the '`create`' argument is set to True, if no valid experiment is found, this method will create one for the user. Otherwise, it will only retrieve a specific experiment or raise an Error.
- If '`create`' is True:
- If ``R``'s running:
- no id or name specified, return the active experiment.
- if id or name is specified, return the specified experiment. If no such exp found, create a new experiment with given id or name, and the experiment is set to be running.
- If ``R``'s not running:
- no id or name specified, create a default experiment, and the experiment is set to be running.
- if id or name is specified, return the specified experiment. If no such exp found, create a new experiment with given name or the default experiment, and the experiment is set to be running.
- Else If '`create`' is False:
- If ``R``'s running:
- no id or name specified, return the active experiment.
- if id or name is specified, return the specified experiment. If no such exp found, raise Error.
- If ``R``'s not running:
- no id or name specified. If the default experiment exists, return it, otherwise, raise Error.
- if id or name is specified, return the specified experiment. If no such exp found, raise Error.
- Parameters:
- `experiment_id` : str
id of the experiment.
- `experiment_name` : str
name of the experiment.
- `create` : boolean
an argument determines whether the method will automatically create a new experiment according to user's specification if the experiment hasn't been created before.
- Returns:
- An experiment instance with given id or name.
- Use case:
.. code-block:: Python
# Case 1
with R.start('test'):
exp = R.get_exp()
recorders = exp.list_recorders()
# Case 2
with R.start('test'):
exp = R.get_exp('test1')
# Case 3
exp = R.get_exp() -> a default experiment.
# Case 4
exp = R.get_exp(experiment_name='test')
# Case 5
exp = R.get_exp(create=False) -> the default experiment if exists.
- `delete_exp(experiment_id=None, experiment_name=None)`
- Method for deleting the experiment with given id or name. At least one of id or name must be given, otherwise, error will occur.
- Parameters:
- `experiment_id` : str
id of the experiment.
- `experiment_name` : str
name of the experiment.
- Use case:
.. code-block:: Python
R.delete_exp(experiment_name='test')
- `get_uri()`
- Method for retrieving the uri of current experiment manager.
- Returns:
- The uri of current experiment manager.
- Use case:
.. code-block:: Python
uri = R.get_uri()
- `get_recorder(recorder_id=None, recorder_name=None, experiment_name=None)`
- Method for retrieving a recorder. The recorder can be used for further process such as ``save_objects``, ``load_object``, ``log_params``, ``log_metrics``, etc.
- If ``R``'s running:
- no id or name specified, return the active recorder.
- if id or name is specified, return the specified recorder.
- If ``R``'s not running:
- no id or name specified, raise Error.
- if id or name is specified, and the corresponding experiment_name must be given, return the specified recorder. Otherwise, raise Error.
- Parameters:
- `recorder_id` : str
id of the recorder.
- `recorder_name` : str
name of the recorder.
- `experiment_name` : str
name of the experiment.
- Returns:
- A recorder instance.
- Use case:
.. code-block:: Python
# Case 1
with R.start('test'):
recorder = R.get_recorder()
# Case 2
with R.start('test'):
recorder = R.get_recorder(recorder_id='2e7a4efd66574fa49039e00ffaefa99d')
# Case 3
recorder = R.get_recorder() -> Error
# Case 4
recorder = R.get_recorder(recorder_id='2e7a4efd66574fa49039e00ffaefa99d') -> Error
# Case 5
recorder = R.get_recorder(recorder_id='2e7a4efd66574fa49039e00ffaefa99d', experiment_name='test')
- `delete_recorder(recorder_id=None, recorder_name=None)`
- Method for deleting the recorders with given id or name. At least one of id or name must be given, otherwise, error will occur.
- Parameters:
- `recorder_id` : str
id of the experiment.
- `recorder_name` : str
name of the experiment.
- Use case:
.. code-block:: Python
R.delete_recorder(recorder_id='2e7a4efd66574fa49039e00ffaefa99d')
- `save_objects(local_path=None, artifact_path=None, **kwargs)`
- Method for saving objects as artifacts in the experiment to the uri. It supports either saving from a local file/directory, or directly saving objects. User can use valid python's keywords arguments to specify the object to be saved as well as its name (name: value).
- If R's running: it will save the objects through the running recorder.
- If R's not running: the system will create a default experiment, and a new recorder and save objects under it.
.. note::
If one wants to save objects with a specific recorder. It is recommended to first get the specific recorder through `get_recorder` API and use the recorder the save objects. The supported arguments are the same as this method.
- Parameters:
- `local_path` : str
if provided, them save the file or directory to the artifact URI.
- `artifact_path` : str
the relative path for the artifact to be stored in the URI.
- Use case:
.. code-block:: Python
# Case 1
with R.start('test'):
pred = model.predict(dataset)
R.save_objects(**{"pred.pkl": pred}, artifact_path='prediction')
# Case 2
with R.start('test'):
R.save_objects(local_path='results/pred.pkl')
- `log_params(**kwargs)`
- Method for logging parameters during an experiment. In addition to using ``R``, one can also log to a specific recorder after getting it with `get_recorder` API.
- If R's running: it will log parameters through the running recorder.
- If R's not running: the system will create a default experiment as well as a new recorder, and log parameters under it.
- Parameters:
- `keyword argument`:
name1=value1, name2=value2, ...
- Use case:
.. code-block:: Python
# Case 1
with R.start('test'):
R.log_params(learning_rate=0.01)
# Case 2
R.log_params(learning_rate=0.01)
- `log_metrics(step=None, **kwargs)`
- Method for logging metrics during an experiment. In addition to using ``R``, one can also log to a specific recorder after getting it with `get_recorder` API.
- If R's running: it will log metrics through the running recorder.
- If R's not running: the system will create a default experiment as well as a new recorder, and log metrics under it.
- Parameters:
- `step`: int
a single integer step at which to log the specified Metrics. If unspecified, each metric is logged at step zero.
- `keyword argument`:
name1=value1, name2=value2, ...
- `set_tags(**kwargs)`
- Method for setting tags for a recorder. In addition to using ``R``, one can also set the tag to a specific recorder after getting it with `get_recorder` API.
- If R's running: it will set tags through the running recorder.
- If R's not running: the system will create a default experiment as well as a new recorder, and set the tags under it.
- Parameters:
- `keyword argument`:
name1=value1, name2=value2, ...
- Use case:
.. code-block:: Python
# Case 1
with R.start('test'):
R.set_tags(release_version="2.2.0")
# Case 2
R.set_tags(release_version="2.2.0")
Experiment Manager
===================
The ``ExpManager`` module in ``Qlib`` is responsible for managing different experiments. Most of the APIs of ``ExpManager`` are similar to ``QlibRecorder``, and the most important API will be the ``get_exp`` method. User can directly refer to the documents above for some detailed information about how to use the ``get_exp`` method.
For other interfaces such as `create_exp`, `delete_exp`, please refer to `Experiment Manager API <../reference/api.html#experiment-manager>`_.
Experiment
===================
The ``Experiment`` class is solely responsible for a single experiment, and it will handle any operations that are related to an experiment. Basic methods such as `start`, `end` an experiment are included. Besides, methods related to `recorders` are also available: such methods include `get_recorder` and `list_recorders`.
For other interfaces such as `search_records`, `delete_recorder`, please refer to `Experiment API <../reference/api.html#experiment>`_.
Recorder
===================
The ``Recorder`` class is responsible for a single recorder. It will handle some detailed operations such as ``log_metrics``, ``log_params`` of a single run. It is designed to help user to easily track results and things being generated during a run.
Here are some important APIs that are not included in the ``QlibRecorder``:
- `list_artifacts(artifact_path: str = None)`
- List all the artifacts of a recorder.
- Parameters:
- `artifact_path` : str
the relative path for the artifact to be stored in the URI.
- Returns:
- A list of artifacts information (name, path, etc.) that being stored.
- `list_metrics()`
- List all the metrics of a recorder.
- Returns:
- A dictionary of metrics that being stored.
- `list_params()`
- List all the params of a recorder.
- Returns:
- A dictionary of params that being stored.
- `list_tags()`
- List all the tags of a recorder.
- Returns:
- A dictionary of tags that being stored.
For other interfaces such as `save_objects`, `load_object`, please refer to `Recorder API <../reference/api.html#recorder>`_.
Record Template
===================
The ``RecordTemp`` class is a class that enables generate experiment results such as IC and backtest in a certain format. We have provided three different `Record Template` class:
- ``SignalRecord``: This class generates the `preidction` of the model.
- ``SigAnaRecord``: This class generates the `IC`, `ICIR`, `Rank IC` and `Rank ICIR`.
- ``PortAnaRecord``: This class generates the results of `backtest`. The detailed information about `backtest` as well as the available `strategy`, users can refer to `Strategy <../component/strategy.html>`_ and `Backtest <../component/backtest.html>`_.
For more information, please refer to `Record Template API <../reference/api.html#module-qlib.workflow.record_temp>`_.

1
docs/component/report.rst
View File

@@ -1,4 +1,5 @@
.. _report:
==========================================
Aanalysis: Evaluation & Results Analysis
==========================================

1
docs/component/strategy.rst
View File

@@ -1,4 +1,5 @@
.. _strategy:
========================================
Interday Strategy: Portfolio Management
========================================

279
docs/component/workflow.rst Normal file
View File

@@ -0,0 +1,279 @@
.. _workflow:
=================================
Workflow: Workflow Management
=================================
.. currentmodule:: qlib
Introduction
===================
The components in `Qlib Framework <../introduction/introduction.html#framework>`_ are designed in a loosely-coupled way. Users could build their own Quant research workflow with these components like `Example <https://github.com/microsoft/qlib/blob/main/examples/workflow_by_code.py>`_.
Besides, ``Qlib`` provides more user-friendly interfaces named ``qrun`` to automatically run the whole workflow defined by configuration. A concrete execution of the whole workflow is called an `experiment`.
With ``qrun``, user can easily run an `experiment`, which includes the following steps:
- Data
- Loading
- Processing
- Slicing
- Model
- Training and inference (static or rolling)
- Saving & loading
- Evaluation
- Backtest
For each `experiment`, ``Qlib`` has a complete system to tracking all the information as well as artifacts generated during training, inference and evaluation phase. For more information about how Qlib handles `experiment`, please refer to the related document: `Recorder: Experiment Management <../component/recorder.html>`_.
Complete Example
===================
Before getting into details, here is a complete example of ``qrun``, which defines the workflow in typical Quant research.
Below is a typical config file of ``qrun``.
.. code-block:: YAML
provider_uri: "~/.qlib/qlib_data/cn_data"
region: cn
market: &market csi300
benchmark: &benchmark SH000300
data_handler_config: &data_handler_config
start_time: 2008-01-01
end_time: 2020-08-01
fit_start_time: 2008-01-01
fit_end_time: 2014-12-31
instruments: *market
port_analysis_config: &port_analysis_config
strategy:
class: TopkDropoutStrategy
module_path: qlib.contrib.strategy.strategy
kwargs:
topk: 50
n_drop: 5
backtest:
verbose: False
limit_threshold: 0.095
account: 100000000
benchmark: *benchmark
deal_price: close
open_cost: 0.0005
close_cost: 0.0015
min_cost: 5
task:
model:
class: LGBModel
module_path: qlib.contrib.model.gbdt
kwargs:
loss: mse
colsample_bytree: 0.8879
learning_rate: 0.0421
subsample: 0.8789
lambda_l1: 205.6999
lambda_l2: 580.9768
max_depth: 8
num_leaves: 210
num_threads: 20
dataset:
class: DatasetH
module_path: qlib.data.dataset
kwargs:
handler:
class: Alpha158
module_path: qlib.contrib.data.handler
kwargs: *data_handler_config
segments:
train: [2008-01-01, 2014-12-31]
valid: [2015-01-01, 2016-12-31]
test: [2017-01-01, 2020-08-01]
record:
- class: SignalRecord
module_path: qlib.workflow.record_temp
kwargs: {}
- class: PortAnaRecord
module_path: qlib.workflow.record_temp
kwargs:
config: *port_analysis_config
After saving the config into `configuration.yaml`, users could start the workflow and test their ideas with a single command below.
.. code-block:: bash
qrun -c configuration.yaml
.. note::
`qrun` will be placed in your $PATH directory when installing ``Qlib``.
Configuration File
===================
Let's get into details of ``qrun`` in this section.
Before using ``qrun``, users need to prepare a configuration file. The following content shows how to prepare each part of the configuration file.
Qlib Data Section
--------------------
At first, the configuration file needs to contain several basic parameters about the data, which will be used for qlib initialization, data handling and backtest.
.. code-block:: YAML
provider_uri: "~/.qlib/qlib_data/cn_data"
region: cn
market: &market csi300
benchmark: &benchmark SH000300
The meaning of each field is as follows:
- `provider_uri`
Type: str. The URI of the Qlib data. For example, it could be the location where the data loaded by ``get_data.py`` are stored.
- `region`
- If `region` == "us", ``Qlib`` will be initialized in US-stock mode.
- If `region` == "cn", ``Qlib`` will be initialized in china-stock mode.
.. note::
The value of `region` should be aligned with the data stored in `provider_uri`.
- `market`
Type: str. Index name, the default value is `csi500`.
- `benchmark`
Type: str, list or pandas.Series. Stock index symbol, the default value is `SH000905`.
.. note::
* If `benchmark` is str, it will use the daily change as the 'bench'.
* If `benchmark` is list, it will use the daily average change of the stock pool in the list as the 'bench'.
* If `benchmark` is pandas.Series, whose `index` is trading date and the value T is the change from T-1 to T, it will be directly used as the 'bench'. An example is as following:
.. code-block:: python
print(D.features(D.instruments('csi500'), ['$close/Ref($close, 1)-1'])['$close/Ref($close, 1)-1'].head())
2017-01-04 0.011693
2017-01-05 0.000721
2017-01-06 -0.004322
2017-01-09 0.006874
2017-01-10 -0.003350
.. note::
The symbol `&` in `yaml` file stands for an anchor of a field, which is useful when another fields include this parameter as part of the value. Taking the configuration file above as an example, users can directly change the value of `market` and `benchmark` without traversing the entire configuration file.
Model Section
--------------------
In the `task` field, the `model` section describes the parameters of the model to be used for training and inference. For more information about the base ``Model`` class, please refer to `Qlib Model <../component/model.html>`_.
.. code-block:: YAML
model:
class: LGBModel
module_path: qlib.contrib.model.gbdt
kwargs:
loss: mse
colsample_bytree: 0.8879
learning_rate: 0.0421
subsample: 0.8789
lambda_l1: 205.6999
lambda_l2: 580.9768
max_depth: 8
num_leaves: 210
num_threads: 20
The meaning of each field is as follows:
- `class`
Type: str. The name for the model class.
- `module_path`
Type: str. The path for the model in qlib.
- `kwargs`
The keywords arguments for the model. Please refer to the specific model implementation for more information: `models <https://github.com/microsoft/qlib/blob/main/qlib/contrib/model>`_.
.. note::
``Qlib`` provides a util named: ``init_instance_by_config`` to initialize any class inside ``Qlib`` with the configuration includes the fields: `class`, `module_path` and `kwargs`.
Dataset Section
--------------------
The `dataset` field describes the parameters for the ``Dataset`` module in ``Qlib`` as well those for the module ``DataHandler``. For more information about the ``Dataset`` module, please refer to `Qlib Model <../component/data.html#dataset>`_.
The keywords arguments configuration of the ``DataHandler`` is as follows:
.. code-block:: YAML
data_handler_config: &data_handler_config
start_time: 2008-01-01
end_time: 2020-08-01
fit_start_time: 2008-01-01
fit_end_time: 2014-12-31
instruments: *market
Users can refer to the document of `DataHandler <../component/data.html#datahandler>`_ for more information about the meaning of each field in the configuration.
Here is the configuration for the ``Dataset`` module which will take care of data preprossing and slicing during the training and testing phase.
.. code-block:: YAML
dataset:
class: DatasetH
module_path: qlib.data.dataset
kwargs:
handler:
class: Alpha158
module_path: qlib.contrib.data.handler
kwargs: *data_handler_config
segments:
train: [2008-01-01, 2014-12-31]
valid: [2015-01-01, 2016-12-31]
test: [2017-01-01, 2020-08-01]
Record Section
--------------------
The `record` field is about the parameters the ``Record`` module in ``Qlib``. ``Record`` is responsible for generating certain analysis and evaluation results such as `prediction`, `information Coefficient (IC)` and `backtest`.
The following script is the configuration of `backtest` and the `strategy` used in `backtest`:
.. code-block:: YAML
port_analysis_config: &port_analysis_config
strategy:
class: TopkDropoutStrategy
module_path: qlib.contrib.strategy.strategy
kwargs:
topk: 50
n_drop: 5
backtest:
verbose: False
limit_threshold: 0.095
account: 100000000
benchmark: *benchmark
deal_price: close
open_cost: 0.0005
close_cost: 0.0015
min_cost: 5
For more information about the meaning of each field in configuration of `strategy` and `backtest`, users can look up the documents: `Strategy <../component/strategy.html>`_ and `Backtest <../component/backtest.html>`_.
Here is the configuration details of different `Record Template` such as ``SignalRecord`` and ``PortAnaRecord``:
.. code-block:: YAML
record:
- class: SignalRecord
module_path: qlib.workflow.record_temp
kwargs: {}
- class: PortAnaRecord
module_path: qlib.workflow.record_temp
kwargs:
config: *port_analysis_config
For more information about the ``Record`` module in ``Qlib``, user can refer to the related document: `Record <../component/recorder.html#record-template>`_.