liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-10 22:36:55 +08:00
Code Activity

Update docs and fix tabnet

Browse Source
This commit is contained in:
Jactus
2020-11-26 00:55:26 +08:00
parent 5be847909f
commit 87cee85cea
27 changed files with 624 additions and 495 deletions
Download Patch File Download Diff File Expand all files Collapse all files

234
docs/component/data.rst
View File

@@ -1,7 +1,7 @@
.. _data: .. _data:
================================ ================================
Data Layer: Data Framework&Usage Data Layer: Data Framework & Usage
================================ ================================
Introduction Introduction
@@ -15,7 +15,9 @@ The introduction of ``Data Layer`` includes the following parts.
- Data Preparation - Data Preparation
- Data API - Data API
- Data Loader
- Data Handler - Data Handler
- Dataset
- Cache - Cache
- Data and Cache File Structure - Data and Cache File Structure
@@ -146,43 +148,161 @@ Filter
To know more about ``Filter``, please refer to `Filter API <../reference/api.html#module-qlib.data.filter>`_. To know more about ``Filter``, please refer to `Filter API <../reference/api.html#module-qlib.data.filter>`_.
Reference Reference
------------- -------------
To know more about ``Data API``, please refer to `Data API <../reference/api.html#data>`_. To know more about ``Data API``, please refer to `Data API <../reference/api.html#data>`_.
Data Loader
=================
``Data Loader`` in ``Qlib`` is designed to load raw data from the original data source. It will be loaded and used in the ``Data Handler`` module.
The ``QlibDataLoader`` class in ``Qlib`` is such an interface that allows users to load raw data from the data source.
Interface
------------
Here are some interfaces of the ``QlibDataLoader`` class:
- `load(instruments, start_time=None, end_time=None)`
- This method loads the data as pd.DataFrame
- Parameters:
- `instruments` : str or dict
it can either be the market name or the config file of instruments generated by InstrumentProvider.
- `start_time` : str
start of the time range.
- `end_time` : str
end of the time range.
- Returns:
- The data being loaded with type `pd.DataFrame`
- `load_group_df(instruments, exprs: list, names: list, start_time=None, end_time=None)`
- This method loads the dataframe for specific group.
- Parameters:
- `instruments` : str or dict
it can either be the market name or the config file of instruments generated by InstrumentProvider.
- `exprs` : list
the expressions to describe the content of the data.
- `names` : list
the name of the data.
- `start_time` : str
start of the time range.
- `end_time` : str
end of the time range.
- Returns:
- The queried data in type `pd.DataFrame`.
API
-----------
To know more about ``Data Loader``, please refer to `Data Loader API <../reference/api.html#module-qlib.data.dataset.loader>`_.
Data Handler Data Handler
================= =================
Users can use ``Data Handler`` in an automatic workflow by ``Estimator``, refer to `Estimator: Workflow Management <estimator.html>`_ for more details. The ``Data Handler`` module in ``Qlib`` is designed to handler those common data processing methods which will be used by most of the models.
Users can use ``Data Handler`` in an automatic workflow by ``qrun``, refer to `Workflow: Workflow Management <workflow.html>`_ for more details.
Also, ``Data Handler`` can be used as an independent module, by which users can easily preprocess data(standardization, remove NaN, etc.) and build datasets. It is a subclass of ``qlib.data.dataset.handler.DataHandlerLP``, which provides some interfaces as follows.
Base Class & Interface Base Class & Interface
---------------------- ----------------------
Qlib provides a base class `qlib.data.dataset.DataHandlerLP <../reference/api.html#qlib.data.dataset.handler.DataHandlerLP>`_, which provides the following interfaces: In addition to use ``Data Handler`` in an automatic workflow with ``qrun``, ``Data Handler`` can be used as an independent module, by which users can easily preprocess data (standardization, remove NaN, etc.) and build datasets.
- `load_feature` In order to achieve so, ``Qlib`` provides a base class `qlib.data.dataset.DataHandlerLP <../reference/api.html#qlib.data.dataset.handler.DataHandlerLP>`_. The core idea of this class is that: we will have some leanable ``Processors`` which can learn the parameters of data processing. When new data comes in, these `trained` ``Processors`` can then infer on the new data and thus processing real-time data in an efficient way. More information about ``Processors`` will be listed in the next subsection.
Implement the interface to load the data features.
- `load_label` Here are some important interfaces that ``DataHandlerLP`` provides:
Implement the interface to load the data labels and calculate the users' labels.
- `setup_processed_data` - `__init__(instruments=None, start_time=None, end_time=None, data_loader: Tuple[dict, str, DataLoader] = None, infer_processors=[], learn_processors=[], process_type=PTYPE_A, **kwargs)`
Implement the interface for data preprocessing, such as preparing feature columns, discarding blank lines, and so on. - Initialization of the class.
- Parameters:
- `infer_processors` : list
- list of <description info> of processors to generate data for inference
- example of <description info>:
Qlib also provides two functions to help users init the data handler, users can override them for users' needs. .. code-block::
1) classname & kwargs:
{
"class": "MinMaxNorm",
"kwargs": {
"fit_start_time": "20080101",
"fit_end_time": "20121231"
}
}
2) Only classname:
"DropnaFeature"
3) object instance of Processor
- `_init_raw_data` - `learn_processors` : list
Users can init the raw df, feature names, and label names of data handler in this function. similar to infer_processors, but for generating data for learning models
If the index of feature df and label df are not the same, users need to override this method to merge them (e.g. inner, left, right merge).
- `process_type`: str
- PTYPE_I = 'independent'
- self._infer will processed by infer_processors
- self._learn will be processed by learn_processors
- PTYPE_A = 'append'
- self._infer will processed by infer_processors
- self._learn will be processed by infer_processors + learn_processors
- (e.g. self._infer processed by learn_processors )
- `fetch(selector: Union[pd.Timestamp, slice, str] = slice(None, None), level: Union[str, int] = "datetime", col_set=DataHandler.CS_ALL, data_key: str = DK_I)`
- This method fetches data from underlying data source
- Parameters:
- `selector` : Union[pd.Timestamp, slice, str]
describe how to select data by index.
- `level` : Union[str, int]
which index level to select the data.
- `col_set` : str
select a set of meaningful columns.(e.g. features, columns).
- `data_key` : str
The data to fetch: DK_*.
- Returns:
- The retrieved results in the type: `pd.DataFrame`.
- `get_cols(col_set=DataHandler.CS_ALL, data_key: str = DK_I)`
- This method gets the column names.
- Parameters:
- `col_set` : str
select a set of meaningful columns.(e.g. features, columns).
- `data_key` : str
the data to fetch: DK_*.
- Returns:
- A list of column names.
If users want to load features and labels by config, users can inherit ``qlib.data.dataset.handler.ConfigDataHandler``, ``Qlib`` also provides some preprocess method in this subclass. If users want to load features and labels by config, users can inherit ``qlib.data.dataset.handler.ConfigDataHandler``, ``Qlib`` also provides some preprocess method in this subclass.
If users want to use qlib data, `QLibDataHandler` is recommended. Users can inherit their custom class from `QLibDataHandler`, which is also a subclass of `ConfigDataHandler`. If users want to use qlib data, `QLibDataHandler` is recommended. Users can inherit their custom class from `QLibDataHandler`, which is also a subclass of `ConfigDataHandler`.
Processor
----------
The ``Processor`` module in ``Qlib`` is designed to be learnable and it is responsible for handling data processing such as `normalization` and `drop none/nan features/labels`.
``Qlib`` provides the following ``Processors``:
- ``DropnaProcessor``: `processor` that drops N/A features.
- ``DropnaLabel``: `processor` that drops N/A labels.
- ``TanhProcess``: `processor` that uses `tanh` to process noise data.
- ``ProcessInf``: `processor` that handles infinity values, it will be replaces by the mean of the column.
- ``Fillna``: `processor` that handles N/A values, which will fill the N/A value by 0 or other given number.
- ``MinMaxNorm``: `processor` that applies min-max normalization.
- ``ZscoreNorm``: `processor` that applies z-score normalization.
- ``CSZScoreNorm``: `processor` that applies cross sectional z-score normalization.
- ``CSRankNorm``: `processor` that applies cross sectional rank normalization.
Users can also create their own `processor` by inheriting the base class of ``Processor``. Please refer to the implementation of all the processors for more information (`Processor Link <https://github.com/microsoft/qlib/blob/main/qlib/data/dataset/processor.py>`_).
API
---------
To know more about ``Processor``, please refer to `Processor API <../reference/api.html#module-qlib.data.dataset.processor>`_.
Usage Usage
-------------- --------------
@@ -194,15 +314,12 @@ Usage
- `get_rolling_data` - `get_rolling_data`
- According to the start and end dates, and `rolling_period`, an iterator is returned, which can be used to traverse the features and labels used for rolling. - According to the start and end dates, and `rolling_period`, an iterator is returned, which can be used to traverse the features and labels used for rolling.
Example Example
-------------- --------------
``Data Handler`` can be run with ``estimator`` by modifying the configuration file, and can also be used as a single module. ``Data Handler`` can be run with ``qrun`` by modifying the configuration file, and can also be used as a single module.
Know more about how to run ``Data Handler`` with ``Estimator``, please refer to `Estimator: Workflow Management <estimator.html>`_ Know more about how to run ``Data Handler`` with ``qrun``, please refer to `Workflow: Workflow Management <workflow.html>`_
Qlib provides implemented data handler `Alpha158`. The following example shows how to run `Alpha158` as a single module. Qlib provides implemented data handler `Alpha158`. The following example shows how to run `Alpha158` as a single module.
@@ -211,45 +328,70 @@ Qlib provides implemented data handler `Alpha158`. The following example shows h
.. code-block:: Python .. code-block:: Python
import qlib
from qlib.contrib.data.handler import Alpha158 from qlib.contrib.data.handler import Alpha158
from qlib.contrib.model.gbdt import LGBModel
DATA_HANDLER_CONFIG = { data_handler_config = {
"dropna_label": True, "start_time": "2008-01-01",
"start_date": "2007-01-01", "end_time": "2020-08-01",
"end_date": "2020-08-01", "fit_start_time": "2008-01-01",
"market": "csi300", "fit_end_time": "2014-12-31",
"instruments": "csi300",
} }
TRAINER_CONFIG = { if __name__ == "__main__":
"train_start_date": "2007-01-01", qlib.init()
"train_end_date": "2014-12-31", h = Alpha158(**data_handler_config)
"validate_start_date": "2015-01-01",
"validate_end_date": "2016-12-31",
"test_start_date": "2017-01-01",
"test_end_date": "2020-08-01",
}
exampleDataHandler = Alpha158(**DATA_HANDLER_CONFIG) # get all the columns of the data
print(h.get_cols())
# example of 'get_split_data' # fetch all the labels
x_train, y_train, x_validate, y_validate, x_test, y_test = exampleDataHandler.get_split_data(**TRAINER_CONFIG) print(h.fetch(col_set="label"))
# example of 'get_rolling_data' # fetch all the features
print(h.fetch(col_set="feature"))
for (x_train, y_train, x_validate, y_validate, x_test, y_test) in exampleDataHandler.get_rolling_data(**TRAINER_CONFIG):
print(x_train, y_train, x_validate, y_validate, x_test, y_test)
.. note:: (x_train, y_train, x_validate, y_validate, x_test, y_test) can be used as arguments for the `fit`, `predic``, and `score` methods of the ``Interday Model`` , please refer to `Model <model.html#base-class-interface>`_.
Also, the above example has been given in ``examples.estimator.train_backtest_analyze.ipynb``.
API API
--------- ---------
To know more about ``Data Handler``, please refer to `Data Handler API <../reference/api.html#module-qlib.data.dataset.handler>`_. To know more about ``Data Handler``, please refer to `Data Handler API <../reference/api.html#module-qlib.data.dataset.handler>`_.
Dataset
=================
The ``Dataset`` module in ``Qlib`` aims to prepare data for model training and inferencing.
The motivation of this module is that we want to maximize the flexibility of of different models to handle data that are suitable for themselves. This module gives the model the rights to process their data in an unique way. For instance, models such as ``GBDT`` may work well on data that contains `nan` or `None` value, while neural networks such as ``DNN`` will break down on such data.
The ``DatasetH`` class is the `dataset` with `Data Handler`. Here is the most important interface of the class:
- `prepare(segments: Union[List[str], Tuple[str], str, slice], col_set=DataHandler.CS_ALL, data_key=DataHandlerLP.DK_I, **kwargs)`
- This method prepares the data for learning and inference.
- Parameters:
- `segments` : Union[List[str], Tuple[str], str, slice]
Describe the scope of the data to be prepared
Here are some examples:
- 'train'
- ['train', 'valid']
- `col_set` : str
The col_set will be passed to self._handler when fetching data.
- `data_key` : str
The data to fetch: DK_*
Default is DK_I, which indicate fetching data for **inference**.
API
---------
To know more about ``Dataset``, please refer to `Dataset API <../reference/api.html#module-qlib.data.dataset.__init__>`_.
Cache Cache
========== ==========

214
docs/component/model.rst
View File

@@ -7,7 +7,7 @@ Interday Model: Model Training & Prediction
Introduction Introduction
=================== ===================
``Interday Model`` is designed to make the `prediction score` about stocks. Users can use the ``Interday Model`` in an automatic workflow by ``Estimator``, please refer to `Estimator: Workflow Management <estimator.html>`_. ``Interday Model`` is designed to make the `prediction score` about stocks. Users can use the ``Interday Model`` in an automatic workflow by ``qrun``, please refer to `Workflow: Workflow Management <workflow.html>`_.
Because the components in ``Qlib`` are designed in a loosely-coupled way, ``Interday Model`` can be used as an independent module also. Because the components in ``Qlib`` are designed in a loosely-coupled way, ``Interday Model`` can be used as an independent module also.
@@ -20,151 +20,125 @@ The base class provides the following interfaces:
- `__init__(**kwargs)` - `__init__(**kwargs)`
- Initialization. - Initialization.
- If users use ``Estimator`` to start an `experiment`, the parameter of `__init__` method shoule be consistent with the hyperparameters in the configuration file.
- `fit(self, x_train, y_train, x_valid, y_valid, w_train=None, w_valid=None, **kwargs)` - `fit(self, dataset, **kwargs)`
- Train model. - Train model.
- Parameter: - Parameter:
- `x_train`, pd.DataFrame type, train feature - `dataset`, ``Qlib``'s ``DatasetH`` type. For more information about ``DatasetH``, users can refer to the related document: `Qlib Dataset <../component/data.html#dataset>`_.
The following example explains the value of `x_train`: The `dataset` is passed into the `model`'s method because there are some unique data preprocessing procedures for each, we want to give each model maximum flexibility to handle the data that is suitable for their own.
The following code example shows how to retrieve `x_train`, `y_train` and `w_train` from the `dataset`:
.. code-block:: YAML .. code-block:: Python
KMID KLEN KMID2 KUP KUP2
instrument datetime
SH600004 2012-01-04 0.000000 0.017685 0.000000 0.012862 0.727275
2012-01-05 -0.006473 0.025890 -0.250001 0.012945 0.499998
2012-01-06 0.008117 0.019481 0.416666 0.008117 0.416666
2012-01-09 0.016051 0.025682 0.624998 0.006421 0.250001
2012-01-10 0.017323 0.026772 0.647057 0.003150 0.117648
... ... ... ... ... ...
SZ300273 2014-12-25 -0.005295 0.038697 -0.136843 0.016293 0.421052
2014-12-26 -0.022486 0.041701 -0.539215 0.002453 0.058824
2014-12-29 -0.031526 0.039092 -0.806451 0.000000 0.000000
2014-12-30 -0.010000 0.032174 -0.310811 0.013913 0.432433
2014-12-31 0.010917 0.020087 0.543479 0.001310 0.065216
# get features and labels
`x_train` is a pandas DataFrame, whose index is MultiIndex <instrument(str), datetime(pd.Timestamp)>. Each column of `x_train` corresponds to a feature, and the column name is the feature name. df_train, df_valid = dataset.prepare(
["train", "valid"], col_set=["feature", "label"], data_key=DataHandlerLP.DK_L
.. note:: )
x_train, y_train = df_train["feature"], df_train["label"]
The number and names of the columns are determined by the data handler, please refer to `Data Handler <data.html#data-handler>`_ and `Estimator Data Section <estimator.html#data-section>`_. x_valid, y_valid = df_valid["feature"], df_valid["label"]
- `y_train`, pd.DataFrame type, train label
The following example explains the value of `y_train`:
.. code-block:: YAML # get weights
try:
LABEL wdf_train, wdf_valid = dataset.prepare(["train", "valid"], col_set=["weight"], data_key=DataHandlerLP.DK_L)
instrument datetime w_train, w_valid = wdf_train["weight"], wdf_valid["weight"]
SH600004 2012-01-04 -0.798456 except KeyError as e:
2012-01-05 -1.366716 w_train = pd.DataFrame(np.ones_like(y_train.values), index=y_train.index)
2012-01-06 -0.491026 w_valid = pd.DataFrame(np.ones_like(y_valid.values), index=y_valid.index)
2012-01-09 0.296900
2012-01-10 0.501426
... ...
SZ300273 2014-12-25 -0.465540
2014-12-26 0.233864
2014-12-29 0.471368
2014-12-30 0.411914
2014-12-31 1.342723
`y_train` is a pandas DataFrame, whose index is MultiIndex <instrument(str), datetime(pd.Timestamp)>. The `LABEL` column represents the value of train label.
.. note::
The number and names of the columns are determined by the ``Data Handler``, please refer to `Data Handler <data.html#data-handler>`_.
- `x_valid`, pd.DataFrame type, validation feature
The format of `x_valid` is same as `x_train`
- `y_valid`, pd.DataFrame type, validation label
The format of `y_valid` is same as `y_train`
- `w_train`(Optional args, default is None), pd.DataFrame type, train weight
`w_train` is a pandas DataFrame, whose shape and index is same as `x_train`. The float value in `w_train` represents the weight of the feature at the same position in `x_train`.
- `w_train`(Optional args, default is None), pd.DataFrame type, validation weight
`w_train` is a pandas DataFrame, whose shape and index is the same as `x_valid`. The float value in `w_train` represents the weight of the feature at the same position in `x_train`.
- `predict(self, x_test, **kwargs)`
- Predict test data 'x_test'
- Parameter:
- `x_test`, pd.DataFrame type, test features
The form of `x_test` is same as `x_train` in 'fit' method.
- Return:
- `label`, np.ndarray type, test label
The label of `x_test` that predicted by model.
- `score(self, x_test, y_test, w_test=None, **kwargs)`
- Evaluate model with test feature/label
- Parameter:
- `x_test`, pd.DataFrame type, test feature
The format of `x_test` is same as `x_train` in `fit` method.
- `x_test`, pd.DataFrame type, test label - `predict(self, dataset, **kwargs)`
The format of `y_test` is same as `y_train` in `fit` method. - Predict test data.
- Parameter:
- `dataset`, ``Qlib``'s ``DatasetH`` type. The usage is similar to the example above.
- Returns:
- Predic results with type: `pandas.Series`.
- `w_test`, pd.DataFrame type, test weight - `finetune(self, dataset, **kwargs)`
The format of `w_test` is same as `w_train` in `fit` method. - Finetune the model.
- Return: float type, evaluation score - Parameter:
- `dataset`, ``Qlib``'s ``DatasetH`` type. The usage is similar to the example above.
For other interfaces such as `save`, `load`, `finetune`, please refer to `Model API <../reference/api.html#module-qlib.model.base>`_.
For other interfaces such as `finetune`, please refer to `Model API <../reference/api.html#module-qlib.model.base>`_.
Example Example
================== ==================
``Qlib`` provides ``LightGBM`` and ``DNN`` models as the baseline, the following steps show how to run`` LightGBM`` as an independent module. ``Qlib``'s `Model Zoo` includes models such as ``LightGBM``, ``DNN``, ``LSTM``, etc.. These models are treated as the baselines of ``Interday Model``. The following steps show how to run`` LightGBM`` as an independent module.
- Initialize ``Qlib`` with `qlib.init` first, please refer to `Initialization <../start/initialization.html>`_. - Initialize ``Qlib`` with `qlib.init` first, please refer to `Initialization <../start/initialization.html>`_.
- Run the following code to get the `prediction score` `pred_score` - Run the following code to get the `prediction score` `pred_score`
.. code-block:: Python .. code-block:: Python
from qlib.contrib.data.handler import Alpha158
from qlib.contrib.model.gbdt import LGBModel from qlib.contrib.model.gbdt import LGBModel
from qlib.contrib.data.handler import Alpha158
from qlib.utils import init_instance_by_config, flatten_dict
from qlib.workflow import R
from qlib.workflow.record_temp import SignalRecord, PortAnaRecord
DATA_HANDLER_CONFIG = { market = "csi300"
"dropna_label": True, benchmark = "SH000300"
"start_date": "2007-01-01",
"end_date": "2020-08-01", data_handler_config = {
"market": MARKET, "start_time": "2008-01-01",
"end_time": "2020-08-01",
"fit_start_time": "2008-01-01",
"fit_end_time": "2014-12-31",
"instruments": market,
} }
TRAINER_CONFIG = { task = {
"train_start_date": "2007-01-01", "model": {
"train_end_date": "2014-12-31", "class": "LGBModel",
"validate_start_date": "2015-01-01", "module_path": "qlib.contrib.model.gbdt",
"validate_end_date": "2016-12-31", "kwargs": {
"test_start_date": "2017-01-01", "loss": "mse",
"test_end_date": "2020-08-01", "colsample_bytree": 0.8879,
"learning_rate": 0.0421,
"subsample": 0.8789,
"lambda_l1": 205.6999,
"lambda_l2": 580.9768,
"max_depth": 8,
"num_leaves": 210,
"num_threads": 20,
},
},
"dataset": {
"class": "DatasetH",
"module_path": "qlib.data.dataset",
"kwargs": {
"handler": {
"class": "Alpha158",
"module_path": "qlib.contrib.data.handler",
"kwargs": data_handler_config,
},
"segments": {
"train": ("2008-01-01", "2014-12-31"),
"valid": ("2015-01-01", "2016-12-31"),
"test": ("2017-01-01", "2020-08-01"),
},
},
},
} }
# model initiaiton
model = init_instance_by_config(task["model"])
dataset = init_instance_by_config(task["dataset"])
x_train, y_train, x_validate, y_validate, x_test, y_test = Alpha158( # start exp
**DATA_HANDLER_CONFIG with R.start(experiment_name="workflow"):
).get_split_data(**TRAINER_CONFIG) # train
R.log_params(**flatten_dict(task))
model.fit(dataset)
# prediction
recorder = R.get_recorder()
sr = SignalRecord(model, dataset, recorder)
sr.generate()
MODEL_CONFIG = { .. note::
"loss": "mse",
"colsample_bytree": 0.8879, `Alpha158` is the data handler provided by ``Qlib``, please refer to `Data Handler <data.html#data-handler>`_.
"learning_rate": 0.0421, `SignalRecord` is the `Record Template` in ``Qlib``, please refer to `Workflow <recorder.html#record-template>`_.
"subsample": 0.8789,
"lambda_l1": 205.6999,
"lambda_l2": 580.9768,
"max_depth": 8,
"num_leaves": 210,
"num_threads": 20,
}
# use default model
model = LGBModel(**MODEL_CONFIG)
model.fit(x_train, y_train, x_validate, y_validate)
_pred = model.predict(x_test)
pred_score = pd.DataFrame(index=_pred.index)
pred_score["score"] = _pred.iloc(axis=1)[0]
.. note:: `Alpha158` is the data handler provided by ``Qlib``, please refer to `Data Handler <data.html#data-handler>`_.
Also, the above example has been given in ``examples/train_backtest_analyze.ipynb``. Also, the above example has been given in ``examples/train_backtest_analyze.ipynb``.

6
docs/component/recorder.rst
View File

@@ -402,8 +402,8 @@ Record Template
The ``RecordTemp`` class is a class that enables generate experiment results such as IC and backtest in a certain format. We have provided three different `Record Template` class: The ``RecordTemp`` class is a class that enables generate experiment results such as IC and backtest in a certain format. We have provided three different `Record Template` class:
- ``SignalRecord``: This class generates the `preidction` of the model. - ``SignalRecord``: This class generates the `preidction` results of the model.
- ``SigAnaRecord``: This class generates the `IC`, `ICIR`, `Rank IC` and `Rank ICIR`. - ``SigAnaRecord``: This class generates the `IC`, `ICIR`, `Rank IC` and `Rank ICIR` of the model.
- ``PortAnaRecord``: This class generates the results of `backtest`. The detailed information about `backtest` as well as the available `strategy`, users can refer to `Strategy <../component/strategy.html>`_ and `Backtest <../component/backtest.html>`_. - ``PortAnaRecord``: This class generates the results of `backtest`. The detailed information about `backtest` as well as the available `strategy`, users can refer to `Strategy <../component/strategy.html>`_ and `Backtest <../component/backtest.html>`_.
For more information, please refer to `Record Template API <../reference/api.html#module-qlib.workflow.record_temp>`_. For more information about the APIs, please refer to `Record Template API <../reference/api.html#module-qlib.workflow.record_temp>`_.

14
docs/reference/api.rst
View File

@@ -60,12 +60,26 @@ Cache
Contrib Contrib
==================== ====================
Data Loader
---------------
.. automodule:: qlib.data.dataset.loader
:members:
Data Handler Data Handler
--------------- ---------------
.. automodule:: qlib.data.dataset.handler .. automodule:: qlib.data.dataset.handler
:members: :members:
Processor
---------------
.. automodule:: qlib.data.dataset.processor
:members:
Dataset
---------------
.. automodule:: qlib.data.dataset.__init__
:members:
Model Model
-------------------- --------------------
.. automodule:: qlib.model.base .. automodule:: qlib.model.base

103
docs/start/integration.rst
View File

@@ -5,7 +5,7 @@ Custom Model Integration
Introduction Introduction
=================== ===================
``Qlib`` provides ``lightGBM`` and ``Dnn`` model as the baseline of ``Interday Model``. In addition to the default model, users can integrate their own custom models into ``Qlib``. ``Qlib``'s `Model Zoo` includes models such as ``LightGBM``, ``DNN``, ``LSTM``, etc.. These models are treated as the baselines of ``Interday Model``. In addition to the default models ``Qlib`` provide, users can integrate their own custom models into ``Qlib``.
Users can integrate their own custom models according to the following steps. Users can integrate their own custom models according to the following steps.
@@ -32,79 +32,76 @@ The Custom models need to inherit `qlib.model.base.Model <../reference/api.html#
- Override the `fit` method - Override the `fit` method
- ``Qlib`` calls the fit method to train the model - ``Qlib`` calls the fit method to train the model
- The parameters must include training feature `x_train`, training label `y_train`, test feature `x_valid`, test label `y_valid` at least. - The parameters must include training feature `dataset`.
- The parameters could include some optional parameters with default values, such as train weight `w_train`, test weight `w_valid` and `num_boost_round = 1000`. - The parameters could include some optional parameters with default values, such as `num_boost_round = 1000` for `GBDT`.
- Code Example: In the following example, `num_boost_round = 1000` is an optional parameter. - Code Example: In the following example, `num_boost_round = 1000` is an optional parameter.
.. code-block:: Python .. code-block:: Python
def fit(self, x_train:pd.DataFrame, y_train:pd.DataFrame, x_valid:pd.DataFrame, y_valid:pd.DataFrame, def fit(self, dataset: DatasetH, num_boost_round = 1000, **kwargs):
w_train:pd.DataFrame = None, w_valid:pd.DataFrame = None, num_boost_round = 1000, **kwargs):
# prepare dataset for lgb training and evaluation
df_train, df_valid = dataset.prepare(
["train", "valid"], col_set=["feature", "label"], data_key=DataHandlerLP.DK_L
)
x_train, y_train = df_train["feature"], df_train["label"]
x_valid, y_valid = df_valid["feature"], df_valid["label"]
# Lightgbm need 1D array as its label # Lightgbm need 1D array as its label
if y_train.values.ndim == 2 and y_train.values.shape[1] == 1: if y_train.values.ndim == 2 and y_train.values.shape[1] == 1:
y_train_1d, y_valid_1d = np.squeeze(y_train.values), np.squeeze(y_valid.values) y_train, y_valid = np.squeeze(y_train.values), np.squeeze(y_valid.values)
else: else:
raise ValueError('LightGBM doesn\'t support multi-label training') raise ValueError("LightGBM doesn't support multi-label training")
w_train_weight = None if w_train is None else w_train.values dtrain = lgb.Dataset(x_train.values, label=y_train)
w_valid_weight = None if w_valid is None else w_valid.values dvalid = lgb.Dataset(x_valid.values, label=y_valid)
dtrain = lgb.Dataset(x_train.values, label=y_train_1d, weight=w_train_weight) # fit the model
dvalid = lgb.Dataset(x_valid.values, label=y_valid_1d, weight=w_valid_weight) self.model = lgb.train(
self._model = lgb.train( self.params,
self._params, dtrain,
dtrain,
num_boost_round=num_boost_round, num_boost_round=num_boost_round,
valid_sets=[dtrain, dvalid], valid_sets=[dtrain, dvalid],
valid_names=['train', 'valid'], valid_names=["train", "valid"],
early_stopping_rounds=early_stopping_rounds,
verbose_eval=verbose_eval,
evals_result=evals_result,
**kwargs **kwargs
) )
- Override the `predict` method - Override the `predict` method
- The parameters include the test features. - The parameters must include training feature `dataset`, which will be userd to get the test dataset.
- Return the `prediction score`. - Return the `prediction score`.
- Please refer to `Model API <../reference/api.html#module-qlib.model.base>`_ for the parameter types of the fit method. - Please refer to `Model API <../reference/api.html#module-qlib.model.base>`_ for the parameter types of the fit method.
- Code Example: In the following example, users need to use dnn to predict the label(such as `preds`) of test data `x_test` and return it. - Code Example: In the following example, users need to use `LightGBM` to predict the label(such as `preds`) of test data `x_test` and return it.
.. code-block:: Python .. code-block:: Python
def predict(self, x_test:pd.DataFrame, **kwargs)-> numpy.ndarray: def predict(self, dataset: DatasetH, **kwargs)-> pandas.Series:
if self._model is None: if self.model is None:
raise ValueError('model is not fitted yet!') raise ValueError("model is not fitted yet!")
return self._model.predict(x_test.values) x_test = dataset.prepare("test", col_set="feature", data_key=DataHandlerLP.DK_I)
return pd.Series(self.model.predict(x_test.values), index=x_test.index)
- Override the `save` method & `load` method - Override the `finetune` method
- The `save` method parameter includes the a `filename` that represents an absolute path, user need to save model into the path. - The parameters must include training feature `dataset`.
- The `load` method parameter includes the a `buffer` read from the `filename` passed in the `save` method, users need to load model from the `buffer`. - Code Example: In the following example, users will use `LightGBM` as the model and finetune it.
- Code Example:
.. code-block:: Python .. code-block:: Python
def save(self, filename): def finetune(self, dataset: DatasetH, num_boost_round=10, verbose_eval=20):
if self._model is None: dtrain, _ = self._prepare_data(dataset)
raise ValueError('model is not fitted yet!') self.model = lgb.train(
self._model.save_model(filename) self.params,
dtrain,
def load(self, buffer): num_boost_round=num_boost_round,
self._model = lgb.Booster(params={'model_str': buffer.decode('utf-8')}) init_model=self.model,
valid_sets=[dtrain],
.. Without tuner, this part will not be used valid_names=["train"],
.. - Override the `score` method(This step is optional) verbose_eval=verbose_eval,
.. - The parameters include the test features and test labels. )
.. - Return the evaluation score of the model. It's recommended to adopt the loss between labels and `prediction score`.
.. - Code Example: In the following example, users need to calculate the weighted loss with test data `x_test`, test label `y_test` and the weight `w_test`.
.. .. code-block:: Python
..
.. def score(self, x_test:pd.Dataframe, y_test:pd.Dataframe, w_test:pd.DataFrame = None) -> float:
.. # Remove rows from x, y and w, which contain Nan in any columns in y_test.
.. x_test, y_test, w_test = drop_nan_by_y_index(x_test, y_test, w_test)
.. preds = self.predict(x_test)
.. w_test_weight = None if w_test is None else w_test.values
.. scorer = mean_squared_error if self.loss_type == 'mse' else roc_auc_score
.. return scorer(y_test.values, preds, sample_weight=w_test_weight)
Configuration File Configuration File
======================= =======================
The configuration file is described in detail in the `estimator <../component/estimator.html#complete-example>`_ document. In order to integrate the custom model into ``Qlib``, users need to modify the "model" field in the configuration file. The configuration file is described in detail in the `Workflow <../component/workflow.html#complete-example>`_ document. In order to integrate the custom model into ``Qlib``, users need to modify the "model" field in the configuration file.
- Example: The following example describes the `model` field of configuration file about the custom lightgbm model mentioned above, where `module_path` is the module path, `class` is the class name, and `args` is the hyperparameter passed into the __init__ method. All parameters in the field is passed to `self._params` by `\*\*kwargs` in `__init__` except `loss = mse`. - Example: The following example describes the `model` field of configuration file about the custom lightgbm model mentioned above, where `module_path` is the module path, `class` is the class name, and `args` is the hyperparameter passed into the __init__ method. All parameters in the field is passed to `self._params` by `\*\*kwargs` in `__init__` except `loss = mse`.
@@ -124,20 +121,20 @@ The configuration file is described in detail in the `estimator <../component/es
num_leaves: 210 num_leaves: 210
num_threads: 20 num_threads: 20
Users could find configuration file of the baseline of the ``Model`` in ``qlib/examples/estimator/estimator_config.yaml`` and ``qlib/examples/estimator/estimator_config_dnn.yaml`` Users could find configuration file of the baselines of the ``Model`` in ``examples/benchmarks``. All the configurations of different models are listed under the corresponding model folder.
Model Testing Model Testing
===================== =====================
Assuming that the configuration file is ``examples/estimator/estimator_config.yaml``, users can run the following command to test the custom model: Assuming that the configuration file is ``examples/benchmarks/LightGBM/workflow_config_lightgbm.yaml``, users can run the following command to test the custom model:
.. code-block:: bash .. code-block:: bash
cd examples # Avoid running program under the directory contains `qlib` cd examples # Avoid running program under the directory contains `qlib`
estimator -c estimator/estimator_config.yaml qrun benchmarks/LightGBM/workflow_config_lightgbm.yaml
.. note:: ``estimator`` is a built-in command of ``Qlib``. .. note:: ``qrun`` is a built-in command of ``Qlib``.
Also, ``Model`` can also be tested as a single module. An example has been given in ``examples/train_backtest_analyze.ipynb``. Also, ``Model`` can also be tested as a single module. An example has been given in ``examples/workflow_by_code.ipynb``.
Reference Reference

4
examples/benchmarks/HATS/README.md
View File

@@ -1,11 +1,11 @@
##Requirement ## Requirement
* pandas==1.1.2 * pandas==1.1.2
* numpy==1.17.4 * numpy==1.17.4
* scikit_learn==0.23.2 * scikit_learn==0.23.2
* torch==1.7.0 * torch==1.7.0
##HATS ## HATS
* HATS is a a hierarchical attention network for stock prediction which uses relational data for stock market prediction. HATS selectively aggregates information * HATS is a a hierarchical attention network for stock prediction which uses relational data for stock market prediction. HATS selectively aggregates information
on different relation types and adds the information to the representations of each company. HATS is used as a relational modeling module with initialized node representations.Furthermore, HATS on different relation types and adds the information to the representations of each company. HATS is used as a relational modeling module with initialized node representations.Furthermore, HATS

2
examples/benchmarks/TFT/README.md
View File

@@ -5,7 +5,7 @@
**GitHub**: https://github.com/google-research/google-research/tree/master/tft **GitHub**: https://github.com/google-research/google-research/tree/master/tft
## Run the Workflow ## Run the Workflow
Users can follow the ``workflow_by_code_tft.py`` to run the benchmark. Users can follow the ``workflow_by_code_tft.py`` to run the benchmark. Please be **aware** that this script can only support Python 3.5 - 3.8.
### Notes ### Notes
1. The model must run in GPU, or an error will be raised. 1. The model must run in GPU, or an error will be raised.

2
examples/benchmarks/TabNet/workflow_config_tabnet.yaml
View File

@@ -44,7 +44,7 @@ task:
module_path: qlib.data.dataset module_path: qlib.data.dataset
kwargs: kwargs:
handler: handler:
class: Alpha158 class: ALPHA360_Denoise
module_path: qlib.contrib.data.handler module_path: qlib.contrib.data.handler
kwargs: *data_handler_config kwargs: *data_handler_config
segments: segments:

84
qlib/contrib/evaluate.py
View File

@@ -26,9 +26,9 @@ def risk_analysis(r, N=252):
Parameters Parameters
---------- ----------
r : pandas.Series r : pandas.Series
daily return series daily return series.
N: int N: int
scaler for annualizing information_ratio (day: 250, week: 50, month: 12) scaler for annualizing information_ratio (day: 250, week: 50, month: 12).
""" """
mean = r.mean() mean = r.mean()
std = r.std(ddof=1) std = r.std(ddof=1)
@@ -61,7 +61,7 @@ def get_strategy(
---------- ----------
strategy : Strategy() strategy : Strategy()
strategy used in backtest strategy used in backtest.
topk : int (Default value: 50) topk : int (Default value: 50)
top-N stocks to buy. top-N stocks to buy.
margin : int or float(Default value: 0.5) margin : int or float(Default value: 0.5)
@@ -73,14 +73,14 @@ def get_strategy(
sell_limit = pred_in_a_day.count() * margin sell_limit = pred_in_a_day.count() * margin
buffer margin, in single score_mode, continue holding stock if it is in nlargest(sell_limit) buffer margin, in single score_mode, continue holding stock if it is in nlargest(sell_limit).
sell_limit should be no less than topk sell_limit should be no less than topk.
n_drop : int n_drop : int
number of stocks to be replaced in each trading date number of stocks to be replaced in each trading date.
risk_degree: float risk_degree: float
0-1, 0.95 for example, use 95% money to trade 0-1, 0.95 for example, use 95% money to trade.
str_type: 'amount', 'weight' or 'dropout' str_type: 'amount', 'weight' or 'dropout'
strategy type: TopkAmountStrategy ,TopkWeightStrategy or TopkDropoutStrategy strategy type: TopkAmountStrategy ,TopkWeightStrategy or TopkDropoutStrategy.
Returns Returns
------- -------
@@ -126,21 +126,21 @@ def get_exchange(
---------- ----------
# exchange related arguments # exchange related arguments
exchange: Exchange() exchange: Exchange().
subscribe_fields: list subscribe_fields: list
subscribe fields subscribe fields.
open_cost : float open_cost : float
open transaction cost open transaction cost.
close_cost : float close_cost : float
close transaction cost close transaction cost.
min_cost : float min_cost : float
min transaction cost min transaction cost.
trade_unit : int trade_unit : int
100 for China A 100 for China A.
deal_price: str deal_price: str
dealing price type: 'close', 'open', 'vwap' dealing price type: 'close', 'open', 'vwap'.
limit_threshold : float limit_threshold : float
limit move 0.1 (10%) for example, long and short with same limit limit move 0.1 (10%) for example, long and short with same limit.
extract_codes: bool extract_codes: bool
will we pass the codes extracted from the pred to the exchange. will we pass the codes extracted from the pred to the exchange.
NOTE: This will be faster with offline qlib. NOTE: This will be faster with offline qlib.
@@ -193,20 +193,20 @@ def backtest(pred, account=1e9, shift=1, benchmark="SH000905", verbose=True, **k
- **backtest workflow related or commmon arguments** - **backtest workflow related or commmon arguments**
pred : pandas.DataFrame pred : pandas.DataFrame
predict should has <datetime, instrument> index and one `score` column predict should has <datetime, instrument> index and one `score` column.
account : float account : float
init account value init account value.
shift : int shift : int
whether to shift prediction by one day whether to shift prediction by one day.
benchmark : str benchmark : str
benchmark code, default is SH000905 CSI 500 benchmark code, default is SH000905 CSI 500.
verbose : bool verbose : bool
whether to print log whether to print log.
- **strategy related arguments** - **strategy related arguments**
strategy : Strategy() strategy : Strategy()
strategy used in backtest strategy used in backtest.
topk : int (Default value: 50) topk : int (Default value: 50)
top-N stocks to buy. top-N stocks to buy.
margin : int or float(Default value: 0.5) margin : int or float(Default value: 0.5)
@@ -218,33 +218,33 @@ def backtest(pred, account=1e9, shift=1, benchmark="SH000905", verbose=True, **k
sell_limit = pred_in_a_day.count() * margin sell_limit = pred_in_a_day.count() * margin
buffer margin, in single score_mode, continue holding stock if it is in nlargest(sell_limit) buffer margin, in single score_mode, continue holding stock if it is in nlargest(sell_limit).
sell_limit should be no less than topk sell_limit should be no less than topk.
n_drop : int n_drop : int
number of stocks to be replaced in each trading date number of stocks to be replaced in each trading date.
risk_degree: float risk_degree: float
0-1, 0.95 for example, use 95% money to trade 0-1, 0.95 for example, use 95% money to trade.
str_type: 'amount', 'weight' or 'dropout' str_type: 'amount', 'weight' or 'dropout'
strategy type: TopkAmountStrategy ,TopkWeightStrategy or TopkDropoutStrategy strategy type: TopkAmountStrategy ,TopkWeightStrategy or TopkDropoutStrategy.
- **exchange related arguments** - **exchange related arguments**
exchange: Exchange() exchange: Exchange()
pass the exchange for speeding up. pass the exchange for speeding up.
subscribe_fields: list subscribe_fields: list
subscribe fields subscribe fields.
open_cost : float open_cost : float
open transaction cost. The default value is 0.002(0.2%). open transaction cost. The default value is 0.002(0.2%).
close_cost : float close_cost : float
close transaction cost. The default value is 0.002(0.2%). close transaction cost. The default value is 0.002(0.2%).
min_cost : float min_cost : float
min transaction cost min transaction cost.
trade_unit : int trade_unit : int
100 for China A 100 for China A.
deal_price: str deal_price: str
dealing price type: 'close', 'open', 'vwap' dealing price type: 'close', 'open', 'vwap'.
limit_threshold : float limit_threshold : float
limit move 0.1 (10%) for example, long and short with same limit limit move 0.1 (10%) for example, long and short with same limit.
extract_codes: bool extract_codes: bool
will we pass the codes extracted from the pred to the exchange. will we pass the codes extracted from the pred to the exchange.
@@ -291,17 +291,17 @@ def long_short_backtest(
""" """
A backtest for long-short strategy A backtest for long-short strategy
:param pred: The trading signal produced on day `T` :param pred: The trading signal produced on day `T`.
:param topk: The short topk securities and long topk securities :param topk: The short topk securities and long topk securities.
:param deal_price: The price to deal the trading :param deal_price: The price to deal the trading.
:param shift: Whether to shift prediction by one day. The trading day will be T+1 if shift==1. :param shift: Whether to shift prediction by one day. The trading day will be T+1 if shift==1.
:param open_cost: open transaction cost :param open_cost: open transaction cost.
:param close_cost: close transaction cost :param close_cost: close transaction cost.
:param trade_unit: 100 for China A :param trade_unit: 100 for China A.
:param limit_threshold: limit move 0.1 (10%) for example, long and short with same limit :param limit_threshold: limit move 0.1 (10%) for example, long and short with same limit.
:param min_cost: min transaction cost :param min_cost: min transaction cost.
:param subscribe_fields: subscribe fields :param subscribe_fields: subscribe fields.
:param extract_codes: bool :param extract_codes: bool.
will we pass the codes extracted from the pred to the exchange. will we pass the codes extracted from the pred to the exchange.
NOTE: This will be faster with offline qlib. NOTE: This will be faster with offline qlib.
:return: The result of backtest, it is represented by a dict. :return: The result of backtest, it is represented by a dict.

16
qlib/contrib/report/analysis_model/analysis_model_performance.py
View File

@@ -252,7 +252,7 @@ def model_performance_graph(
"""Model performance """Model performance
:param pred_label: index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[score, :param pred_label: index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[score,
label]**. It is usually same as the label of model training(e.g. "Ref($close, -2)/Ref($close, -1) - 1") label]**. It is usually same as the label of model training(e.g. "Ref($close, -2)/Ref($close, -1) - 1").
.. code-block:: python .. code-block:: python
@@ -266,13 +266,13 @@ def model_performance_graph(
:param lag: `pred.groupby(level='instrument')['score'].shift(lag)`. It will be only used in the auto-correlation computing. :param lag: `pred.groupby(level='instrument')['score'].shift(lag)`. It will be only used in the auto-correlation computing.
:param N: group number, default 5 :param N: group number, default 5.
:param reverse: if `True`, `pred['score'] *= -1` :param reverse: if `True`, `pred['score'] *= -1`.
:param rank: if **True**, calculate rank ic :param rank: if **True**, calculate rank ic.
:param graph_names: graph names; default ['cumulative_return', 'pred_ic', 'pred_autocorr', 'pred_turnover'] :param graph_names: graph names; default ['cumulative_return', 'pred_ic', 'pred_autocorr', 'pred_turnover'].
:param show_notebook: whether to display graphics in notebook, the default is `True` :param show_notebook: whether to display graphics in notebook, the default is `True`.
:param show_nature_day: whether to display the abscissa of non-trading day :param show_nature_day: whether to display the abscissa of non-trading day.
:return: if show_notebook is True, display in notebook; else return `plotly.graph_objs.Figure` list :return: if show_notebook is True, display in notebook; else return `plotly.graph_objs.Figure` list.
""" """
figure_list = [] figure_list = []
for graph_name in graph_names: for graph_name in graph_names:

6
qlib/contrib/report/analysis_position/cumulative_return.py
View File

@@ -218,10 +218,10 @@ def cumulative_return_graph(
Graph desc: Graph desc:
- Axis X: Trading day - Axis X: Trading day.
- Axis Y: - Axis Y:
- Above axis Y: `(((Ref($close, -1)/$close - 1) * weight).sum() / weight.sum()).cumsum()` - Above axis Y: `(((Ref($close, -1)/$close - 1) * weight).sum() / weight.sum()).cumsum()`.
- Below axis Y: Daily weight sum - Below axis Y: Daily weight sum.
- In the **sell** graph, `y < 0` stands for profit; in other cases, `y > 0` stands for profit. - In the **sell** graph, `y < 0` stands for profit; in other cases, `y > 0` stands for profit.
- In the **buy_minus_sell** graph, the **y** value of the **weight** graph at the bottom is `buy_weight + sell_weight`. - In the **buy_minus_sell** graph, the **y** value of the **weight** graph at the bottom is `buy_weight + sell_weight`.
- In each graph, the **red line** in the histogram on the right represents the average. - In each graph, the **red line** in the histogram on the right represents the average.

6
qlib/contrib/report/analysis_position/rank_label.py
View File

@@ -97,9 +97,9 @@ def rank_label_graph(
qcr.rank_label_graph(positions, features_df, pred_df_dates.min(), pred_df_dates.max()) qcr.rank_label_graph(positions, features_df, pred_df_dates.min(), pred_df_dates.max())
:param position: position data; **qlib.contrib.backtest.backtest.backtest** result :param position: position data; **qlib.contrib.backtest.backtest.backtest** result.
:param label_data: **D.features** result; index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[label]**. :param label_data: **D.features** result; index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[label]**.
**The label T is the change from T to T+1**, it is recommended to use ``close``, example: `D.features(D.instruments('csi500'), ['Ref($close, -1)/$close-1'])` **The label T is the change from T to T+1**, it is recommended to use ``close``, example: `D.features(D.instruments('csi500'), ['Ref($close, -1)/$close-1'])`.
.. code-block:: python .. code-block:: python
@@ -115,7 +115,7 @@ def rank_label_graph(
:param start_date: start date :param start_date: start date
:param end_date: end_date :param end_date: end_date
:param show_notebook: **True** or **False**. If True, show graph in notebook, else return figures :param show_notebook: **True** or **False**. If True, show graph in notebook, else return figures.
:return: :return:
""" """
position = copy.deepcopy(position) position = copy.deepcopy(position)

6
qlib/contrib/report/analysis_position/report.py
View File

@@ -186,7 +186,7 @@ def report_graph(report_df: pd.DataFrame, show_notebook: bool = True) -> [list,
qcr.report_graph(report_normal_df) qcr.report_graph(report_normal_df)
:param report_df: **df.index.name** must be **date**, **df.columns** must contain **return**, **turnover**, **cost**, **bench** :param report_df: **df.index.name** must be **date**, **df.columns** must contain **return**, **turnover**, **cost**, **bench**.
.. code-block:: python .. code-block:: python
@@ -200,8 +200,8 @@ def report_graph(report_df: pd.DataFrame, show_notebook: bool = True) -> [list,
2017-01-10 -0.000416 0.000440 -0.003350 0.208396 2017-01-10 -0.000416 0.000440 -0.003350 0.208396
:param show_notebook: whether to display graphics in notebook, the default is **True** :param show_notebook: whether to display graphics in notebook, the default is **True**.
:return: if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure** list :return: if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure** list.
""" """
report_df = report_df.copy() report_df = report_df.copy()
fig_list = _report_figure(report_df) fig_list = _report_figure(report_df)

6
qlib/contrib/report/analysis_position/risk_analysis.py
View File

@@ -218,7 +218,7 @@ def risk_analysis_graph(
max_drawdown -0.088263 max_drawdown -0.088263
:param report_normal_df: **df.index.name** must be **date**, df.columns must contain **return**, **turnover**, **cost**, **bench** :param report_normal_df: **df.index.name** must be **date**, df.columns must contain **return**, **turnover**, **cost**, **bench**.
.. code-block:: python .. code-block:: python
@@ -232,7 +232,7 @@ def risk_analysis_graph(
2017-01-10 -0.000416 0.000440 -0.003350 0.208396 2017-01-10 -0.000416 0.000440 -0.003350 0.208396
:param report_long_short_df: **df.index.name** must be **date**, df.columns contain **long**, **short**, **long_short** :param report_long_short_df: **df.index.name** must be **date**, df.columns contain **long**, **short**, **long_short**.
.. code-block:: python .. code-block:: python
@@ -246,7 +246,7 @@ def risk_analysis_graph(
2017-01-10 0.000824 -0.001944 -0.001120 2017-01-10 0.000824 -0.001944 -0.001120
:param show_notebook: Whether to display graphics in a notebook, default **True** :param show_notebook: Whether to display graphics in a notebook, default **True**.
If True, show graph in notebook If True, show graph in notebook
If False, return graph figure If False, return graph figure
:return: :return:

6
qlib/contrib/report/analysis_position/score_ic.py
View File

@@ -36,7 +36,7 @@ def score_ic_graph(pred_label: pd.DataFrame, show_notebook: bool = True) -> [lis
analysis_position.score_ic_graph(pred_label) analysis_position.score_ic_graph(pred_label)
:param pred_label: index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[score, label]** :param pred_label: index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[score, label]**.
.. code-block:: python .. code-block:: python
@@ -49,8 +49,8 @@ def score_ic_graph(pred_label: pd.DataFrame, show_notebook: bool = True) -> [lis
2017-12-15 -0.102778 -0.102778 2017-12-15 -0.102778 -0.102778
:param show_notebook: whether to display graphics in notebook, the default is **True** :param show_notebook: whether to display graphics in notebook, the default is **True**.
:return: if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure** list :return: if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure** list.
""" """
_ic_df = _get_score_ic(pred_label) _ic_df = _get_score_ic(pred_label)
# FIXME: support HIGH-FREQ # FIXME: support HIGH-FREQ

70
qlib/contrib/strategy/strategy.py
View File

@@ -31,16 +31,16 @@ class BaseStrategy:
Parameters Parameters
----------- -----------
score_series : pd.Seires score_series : pd.Seires
stock_id , score stock_id , score.
current : Position() current : Position()
current state of position current state of position.
DO NOT directly change the state of current DO NOT directly change the state of current.
trade_exchange : Exchange() trade_exchange : Exchange()
trade exchange trade exchange.
pred_date : pd.Timestamp pred_date : pd.Timestamp
predict date predict date.
trade_date : pd.Timestamp trade_date : pd.Timestamp
trade date trade date.
""" """
pass pass
@@ -49,11 +49,11 @@ class BaseStrategy:
Parameters Parameters
----------- -----------
score_series : pd.Series score_series : pd.Series
stock_id , score stock_id , score.
pred_date : pd.Timestamp pred_date : pd.Timestamp
oredict date oredict date.
trade_date : pd.Timestamp trade_date : pd.Timestamp
trade date trade date.
""" """
pass pass
@@ -67,7 +67,7 @@ class BaseStrategy:
""" """
This method only be used in 'online' module, it will generate the *args to initial the strategy. This method only be used in 'online' module, it will generate the *args to initial the strategy.
:param :param
mode : model used in 'online' module mode : model used in 'online' module.
""" """
return {} return {}
@@ -82,7 +82,7 @@ class StrategyWrapper:
def __init__(self, inner_strategy): def __init__(self, inner_strategy):
"""__init__ """__init__
:param inner_strategy: set the inner strategy :param inner_strategy: set the inner strategy.
""" """
self.inner_strategy = inner_strategy self.inner_strategy = inner_strategy
@@ -99,9 +99,9 @@ class AdjustTimer:
Responsible for timing of position adjusting Responsible for timing of position adjusting
This is designed as multiple inheritance mechanism due to: This is designed as multiple inheritance mechanism due to:
- the is_adjust may need access to the internel state of a strategy - the is_adjust may need access to the internel state of a strategy.
- it can be reguard as a enhancement to the existing strategy - it can be reguard as a enhancement to the existing strategy.
""" """
# adjust position in each trade date # adjust position in each trade date
@@ -146,12 +146,12 @@ class WeightStrategyBase(BaseStrategy, AdjustTimer):
Parameters Parameters
----------- -----------
score : pd.Series score : pd.Series
pred score for this trade date, index is stock_id, contain 'score' column pred score for this trade date, index is stock_id, contain 'score' column.
current : Position() current : Position()
current position current position.
trade_exchange : Exchange() trade_exchange : Exchange()
trade_date : pd.Timestamp trade_date : pd.Timestamp
trade date trade date.
""" """
raise NotImplementedError() raise NotImplementedError()
@@ -160,13 +160,13 @@ class WeightStrategyBase(BaseStrategy, AdjustTimer):
Parameters Parameters
----------- -----------
score_series : pd.Seires score_series : pd.Seires
stock_id , score stock_id , score.
current : Position() current : Position()
current of account current of account.
trade_exchange : Exchange() trade_exchange : Exchange()
exchange exchange.
trade_date : pd.Timestamp trade_date : pd.Timestamp
date date.
""" """
# judge if to adjust # judge if to adjust
if not self.is_adjust(trade_date): if not self.is_adjust(trade_date):
@@ -206,26 +206,26 @@ class TopkDropoutStrategy(BaseStrategy, ListAdjustTimer):
Parameters Parameters
----------- -----------
topk : int topk : int
The number of stocks in the portfolio the number of stocks in the portfolio.
n_drop : int n_drop : int
number of stocks to be replaced in each trading date number of stocks to be replaced in each trading date.
method_sell : str method_sell : str
dropout method_sell, random/bottom dropout method_sell, random/bottom.
method_buy : str method_buy : str
dropout method_buy, random/top dropout method_buy, random/top.
risk_degree : float risk_degree : float
position percentage of total value position percentage of total value.
thresh : int thresh : int
minimun holding days since last buy singal of the stock minimun holding days since last buy singal of the stock.
hold_thresh : int hold_thresh : int
minimum holding days minimum holding days
before sell stock , will check current.get_stock_count(order.stock_id) >= self.thresh before sell stock , will check current.get_stock_count(order.stock_id) >= self.thresh.
only_tradable : bool only_tradable : bool
will the strategy only consider the tradable stock when buying and selling. will the strategy only consider the tradable stock when buying and selling.
if only_tradable: if only_tradable:
strategy will make buy sell decision without checking the tradable state of the stock strategy will make buy sell decision without checking the tradable state of the stock.
else: else:
strategy will make decision with the tradable state of the stock info and avoid buy and sell them strategy will make decision with the tradable state of the stock info and avoid buy and sell them.
""" """
super(TopkDropoutStrategy, self).__init__() super(TopkDropoutStrategy, self).__init__()
ListAdjustTimer.__init__(self, kwargs.get("adjust_dates", None)) ListAdjustTimer.__init__(self, kwargs.get("adjust_dates", None))
@@ -245,7 +245,7 @@ class TopkDropoutStrategy(BaseStrategy, ListAdjustTimer):
def get_risk_degree(self, date): def get_risk_degree(self, date):
"""get_risk_degree """get_risk_degree
Return the proportion of your total value you will used in investment. Return the proportion of your total value you will used in investment.
Dynamically risk_degree will result in Market timing Dynamically risk_degree will result in Market timing.
""" """
# It will use 95% amoutn of your total value by default # It will use 95% amoutn of your total value by default
return self.risk_degree return self.risk_degree
@@ -257,15 +257,15 @@ class TopkDropoutStrategy(BaseStrategy, ListAdjustTimer):
Parameters Parameters
----------- -----------
score_series : pd.Series score_series : pd.Series
stock_id , score stock_id , score.
current : Position() current : Position()
current of account current of account.
trade_exchange : Exchange() trade_exchange : Exchange()
exchange exchange.
pred_date : pd.Timestamp pred_date : pd.Timestamp
predict date predict date.
trade_date : pd.Timestamp trade_date : pd.Timestamp
trade date trade date.
""" """
if not self.is_adjust(trade_date): if not self.is_adjust(trade_date):
return [] return []

8
qlib/data/base.py
View File

@@ -129,13 +129,13 @@ class Expression(abc.ABC):
Parameters Parameters
---------- ----------
instrument : str instrument : str
instrument code instrument code.
start_index : str start_index : str
feature start index [in calendar] feature start index [in calendar].
end_index : str end_index : str
feature end index [in calendar] feature end index [in calendar].
freq : str freq : str
feature frequency feature frequency.
Returns Returns
---------- ----------

40
qlib/data/cache.py
View File

@@ -76,8 +76,8 @@ class MemCache(object):
Parameters Parameters
---------- ----------
mem_cache_size_limit: cache max size mem_cache_size_limit: cache max size.
limit_type: length or sizeof; length(call fun: len), size(call fun: sys.getsizeof) limit_type: length or sizeof; length(call fun: len), size(call fun: sys.getsizeof).
""" """
if limit_type not in ["length", "sizeof"]: if limit_type not in ["length", "sizeof"]:
raise ValueError(f"limit_type must be length or sizeof, your limit_type is {limit_type}") raise ValueError(f"limit_type must be length or sizeof, your limit_type is {limit_type}")
@@ -118,9 +118,9 @@ class MemCacheExpire:
def set_cache(mem_cache, key, value): def set_cache(mem_cache, key, value):
"""set cache """set cache
:param mem_cache: MemCache attribute('c'/'i'/'f') :param mem_cache: MemCache attribute('c'/'i'/'f').
:param key: cache key :param key: cache key.
:param value: cache value :param value: cache value.
""" """
mem_cache[key] = value, time.time() mem_cache[key] = value, time.time()
@@ -128,9 +128,9 @@ class MemCacheExpire:
def get_cache(mem_cache, key): def get_cache(mem_cache, key):
"""get mem cache """get mem cache
:param mem_cache: MemCache attribute('c'/'i'/'f') :param mem_cache: MemCache attribute('c'/'i'/'f').
:param key: cache key :param key: cache key.
:return: cache value; if cache not exist, return None :return: cache value; if cache not exist, return None.
""" """
value = None value = None
expire = False expire = False
@@ -275,12 +275,12 @@ class ExpressionCache(BaseProviderCache):
Parameters Parameters
---------- ----------
cache_uri : str cache_uri : str
the complete uri of expression cache file (include dir path) the complete uri of expression cache file (include dir path).
Returns Returns
------- -------
int int
0(successful update)/ 1(no need to update)/ 2(update failure) 0(successful update)/ 1(no need to update)/ 2(update failure).
""" """
raise NotImplementedError("Implement this method if you want to make expression cache up to date") raise NotImplementedError("Implement this method if you want to make expression cache up to date")
@@ -348,7 +348,7 @@ class DatasetCache(BaseProviderCache):
Parameters Parameters
---------- ----------
cache_uri : str cache_uri : str
the complete uri of dataset cache file (include dir path) the complete uri of dataset cache file (include dir path).
Returns Returns
------- -------
@@ -361,9 +361,9 @@ class DatasetCache(BaseProviderCache):
def cache_to_origin_data(data, fields): def cache_to_origin_data(data, fields):
"""cache data to origin data """cache data to origin data
:param data: pd.DataFrame, cache data :param data: pd.DataFrame, cache data.
:param fields: feature fields :param fields: feature fields.
:return: pd.DataFrame :return: pd.DataFrame.
""" """
not_space_fields = remove_fields_space(fields) not_space_fields = remove_fields_space(fields)
data = data.loc[:, not_space_fields] data = data.loc[:, not_space_fields]
@@ -583,7 +583,7 @@ class DiskDatasetCache(DatasetCache):
:param cache_path: :param cache_path:
:param start_time: :param start_time:
:param end_time: :param end_time:
:param fields: The fields order of the dataset cache is sorted. So rearrange the columns to make it consistent :param fields: The fields order of the dataset cache is sorted. So rearrange the columns to make it consistent.
:return: :return:
""" """
@@ -771,12 +771,12 @@ class DiskDatasetCache(DatasetCache):
- This is a hdf file sorted by datetime - This is a hdf file sorted by datetime
:param cache_path: The path to store the cache :param cache_path: The path to store the cache.
:param instruments: The instruments to store the cache :param instruments: The instruments to store the cache.
:param fields: The fields to store the cache :param fields: The fields to store the cache.
:param freq: The freq to store the cache :param freq: The freq to store the cache.
:return type pd.DataFrame; The fields of the returned DataFrame are consistent with the parameters of the function :return type pd.DataFrame; The fields of the returned DataFrame are consistent with the parameters of the function.
""" """
# get calendar # get calendar
from .data import Cal from .data import Cal

8
qlib/data/client.py
View File

@@ -51,13 +51,13 @@ class Client(object):
Parameters Parameters
---------- ----------
request_type : str request_type : str
type of proposed request, 'calendar'/'instrument'/'feature' type of proposed request, 'calendar'/'instrument'/'feature'.
request_content : dict request_content : dict
records the information of the request records the information of the request.
msg_proc_func : func msg_proc_func : func
the function to process the message when receiving response, should have arg `*args` the function to process the message when receiving response, should have arg `*args`.
msg_queue: Queue msg_queue: Queue
The queue to pass the messsage after callback The queue to pass the messsage after callback.
""" """
head_info = {"version": qlib.__version__} head_info = {"version": qlib.__version__}

92
qlib/data/data.py
View File

@@ -41,13 +41,13 @@ class CalendarProvider(abc.ABC):
Parameters Parameters
---------- ----------
start_time : str start_time : str
start of the time range start of the time range.
end_time : str end_time : str
end of the time range end of the time range.
freq : str freq : str
time frequency, available: year/quarter/month/week/day time frequency, available: year/quarter/month/week/day.
future : bool future : bool
whether including future trading day whether including future trading day.
Returns Returns
---------- ----------
@@ -62,24 +62,24 @@ class CalendarProvider(abc.ABC):
Parameters Parameters
---------- ----------
start_time : str start_time : str
start of the time range start of the time range.
end_time : str end_time : str
end of the time range end of the time range.
freq : str freq : str
time frequency, available: year/quarter/month/week/day time frequency, available: year/quarter/month/week/day.
future : bool future : bool
whether including future trading day whether including future trading day.
Returns Returns
------- -------
pd.Timestamp pd.Timestamp
the real start time the real start time.
pd.Timestamp pd.Timestamp
the real end time the real end time.
int int
the index of start time the index of start time.
int int
the index of end time the index of end time.
""" """
start_time = pd.Timestamp(start_time) start_time = pd.Timestamp(start_time)
end_time = pd.Timestamp(end_time) end_time = pd.Timestamp(end_time)
@@ -103,16 +103,16 @@ class CalendarProvider(abc.ABC):
Parameters Parameters
---------- ----------
freq : str freq : str
frequency of read calendar file frequency of read calendar file.
future : bool future : bool
whether including future trading day whether including future trading day.
Returns Returns
------- -------
list list
list of timestamps list of timestamps.
dict dict
dict composed by timestamp as key and index as value for fast search dict composed by timestamp as key and index as value for fast search.
""" """
flag = f"{freq}_future_{future}" flag = f"{freq}_future_{future}"
if flag in H["c"]: if flag in H["c"]:
@@ -141,14 +141,14 @@ class InstrumentProvider(abc.ABC):
Parameters Parameters
---------- ----------
market : str market : str
market/industry/index shortname, e.g. all/sse/szse/sse50/csi300/csi500 market/industry/index shortname, e.g. all/sse/szse/sse50/csi300/csi500.
filter_pipe : list filter_pipe : list
the list of dynamic filters the list of dynamic filters.
Returns Returns
---------- ----------
dict dict
dict of stockpool config dict of stockpool config.
{`market`=>base market name, `filter_pipe`=>list of filters} {`market`=>base market name, `filter_pipe`=>list of filters}
example : example :
@@ -182,13 +182,13 @@ class InstrumentProvider(abc.ABC):
Parameters Parameters
---------- ----------
instruments : dict instruments : dict
stockpool config stockpool config.
start_time : str start_time : str
start of the time range start of the time range.
end_time : str end_time : str
end of the time range end of the time range.
as_list : bool as_list : bool
return instruments as list or dict return instruments as list or dict.
Returns Returns
------- -------
@@ -243,15 +243,15 @@ class FeatureProvider(abc.ABC):
Parameters Parameters
---------- ----------
instrument : str instrument : str
a certain instrument a certain instrument.
field : str field : str
a certain field of feature a certain field of feature.
start_time : str start_time : str
start of the time range start of the time range.
end_time : str end_time : str
end of the time range end of the time range.
freq : str freq : str
time frequency, available: year/quarter/month/week/day time frequency, available: year/quarter/month/week/day.
Returns Returns
------- -------
@@ -294,15 +294,15 @@ class ExpressionProvider(abc.ABC):
Parameters Parameters
---------- ----------
instrument : str instrument : str
a certain instrument a certain instrument.
field : str field : str
a certain field of feature a certain field of feature.
start_time : str start_time : str
start of the time range start of the time range.
end_time : str end_time : str
end of the time range end of the time range.
freq : str freq : str
time frequency, available: year/quarter/month/week/day time frequency, available: year/quarter/month/week/day.
Returns Returns
------- -------
@@ -325,20 +325,20 @@ class DatasetProvider(abc.ABC):
Parameters Parameters
---------- ----------
instruments : list or dict instruments : list or dict
list/dict of instruments or dict of stockpool config list/dict of instruments or dict of stockpool config.
fields : list fields : list
list of feature instances list of feature instances.
start_time : str start_time : str
start of the time range start of the time range.
end_time : str end_time : str
end of the time range end of the time range.
freq : str freq : str
time frequency time frequency.
Returns Returns
---------- ----------
pd.DataFrame pd.DataFrame
a pandas dataframe with <instrument, datetime> index a pandas dataframe with <instrument, datetime> index.
""" """
raise NotImplementedError("Subclass of DatasetProvider must implement `Dataset` method") raise NotImplementedError("Subclass of DatasetProvider must implement `Dataset` method")
@@ -357,17 +357,17 @@ class DatasetProvider(abc.ABC):
Parameters Parameters
---------- ----------
instruments : list or dict instruments : list or dict
list/dict of instruments or dict of stockpool config list/dict of instruments or dict of stockpool config.
fields : list fields : list
list of feature instances list of feature instances.
start_time : str start_time : str
start of the time range start of the time range.
end_time : str end_time : str
end of the time range end of the time range.
freq : str freq : str
time frequency time frequency.
disk_cache : int disk_cache : int
whether to skip(0)/use(1)/replace(2) disk_cache whether to skip(0)/use(1)/replace(2) disk_cache.
""" """
return DiskDatasetCache._uri(instruments, fields, start_time, end_time, freq, disk_cache) return DiskDatasetCache._uri(instruments, fields, start_time, end_time, freq, disk_cache)
@@ -526,7 +526,7 @@ class LocalCalendarProvider(CalendarProvider):
Parameters Parameters
---------- ----------
freq : str freq : str
frequency of read calendar file frequency of read calendar file.
Returns Returns
---------- ----------

30
qlib/data/dataset/__init__.py
View File

@@ -17,7 +17,7 @@ class Dataset(Serializable):
init is designed to finish following steps: init is designed to finish following steps:
- setup data - setup data
- The data related attributes' names should start with '_' so that it will not be saved on disk when serializing - The data related attributes' names should start with '_' so that it will not be saved on disk when serializing.
- initialize the state of the dataset(info to prepare the data) - initialize the state of the dataset(info to prepare the data)
- The name of essential state for preparing data should not start with '_' so that it could be serialized on disk when serializing. - The name of essential state for preparing data should not start with '_' so that it could be serialized on disk when serializing.
@@ -29,17 +29,17 @@ class Dataset(Serializable):
def setup_data(self, *args, **kwargs): def setup_data(self, *args, **kwargs):
""" """
setup the data Setup the data.
We split the setup_data function for following situation: We split the setup_data function for following situation:
- User have a Dataset object with learned status on disk - User have a Dataset object with learned status on disk.
- User load the Dataset object from the disk(Note the init function is skiped) - User load the Dataset object from the disk(Note the init function is skiped).
- User call `setup_data` to load new data - User call `setup_data` to load new data.
- User prepare data for model based on previous status - User prepare data for model based on previous status.
""" """
pass pass
@@ -66,9 +66,10 @@ class DatasetH(Dataset):
User should try to put the data preprocessing functions into handler. User should try to put the data preprocessing functions into handler.
Only following data processing functions should be placed in Dataset: Only following data processing functions should be placed in Dataset:
- The processing is related to specific model. - The processing is related to specific model.
- The processing is related to data split - The processing is related to data split.
""" """
def __init__(self, handler: Union[dict, DataHandler], segments: list): def __init__(self, handler: Union[dict, DataHandler], segments: list):
@@ -76,15 +77,15 @@ class DatasetH(Dataset):
Parameters Parameters
---------- ----------
handler : Union[dict, DataHandler] handler : Union[dict, DataHandler]
handler will be passed into setup_data handler will be passed into setup_data.
segments : list segments : list
handler will be passed into setup_data handler will be passed into setup_data.
""" """
super().__init__(handler, segments) super().__init__(handler, segments)
def setup_data(self, handler: Union[dict, DataHandler], segments: list): def setup_data(self, handler: Union[dict, DataHandler], segments: list):
""" """
setup the underlying data Setup the underlying data.
Parameters Parameters
---------- ----------
@@ -121,7 +122,7 @@ class DatasetH(Dataset):
**kwargs, **kwargs,
) -> Union[List[pd.DataFrame], pd.DataFrame]: ) -> Union[List[pd.DataFrame], pd.DataFrame]:
""" """
prepare the data for learning and inference Prepare the data for learning and inference.
Parameters Parameters
---------- ----------
@@ -132,11 +133,12 @@ class DatasetH(Dataset):
- 'train' - 'train'
- ['train', 'valid'] - ['train', 'valid']
col_set : str col_set : str
The col_set will be passed to self._handler when fetching data The col_set will be passed to self._handler when fetching data.
data_key: str data_key : str
The data to fetch: DK_* The data to fetch: DK_*
Default is DK_I, which indicate fetching data for **inference** Default is DK_I, which indicate fetching data for **inference**.
Returns Returns
------- -------

38
qlib/data/dataset/handler.py
View File

@@ -29,7 +29,7 @@ class DataHandler(Serializable):
""" """
The steps to using a handler The steps to using a handler
1. initialized data handler (call by `init`). 1. initialized data handler (call by `init`).
2. use the data 2. use the data.
The data handler try to maintain a handler with 2 level. The data handler try to maintain a handler with 2 level.
@@ -65,17 +65,17 @@ class DataHandler(Serializable):
Parameters Parameters
---------- ----------
instruments : instruments :
The stock list to retrive The stock list to retrive.
start_time : start_time :
start_time of the original data start_time of the original data.
end_time : end_time :
end_time of the original data end_time of the original data.
data_loader : Tuple[dict, str, DataLoader] data_loader : Tuple[dict, str, DataLoader]
data loader to load the data data loader to load the data.
init_data : init_data :
intialize the original data in the constructor intialize the original data in the constructor.
fetch_orig : bool fetch_orig : bool
Return the original data instead of copy if possible Return the original data instead of copy if possible.
""" """
# Set logger # Set logger
self.logger = get_module_logger("DataHandler") self.logger = get_module_logger("DataHandler")
@@ -219,9 +219,9 @@ class DataHandler(Serializable):
get a iterator of sliced data with given periods get a iterator of sliced data with given periods
Args: Args:
periods (int): number of periods periods (int): number of periods.
min_periods (int): minimum periods for sliced dataframe min_periods (int): minimum periods for sliced dataframe.
kwargs (dict): will be passed to `self.fetch` kwargs (dict): will be passed to `self.fetch`.
""" """
trading_dates = self._data.index.unique(level="datetime") trading_dates = self._data.index.unique(level="datetime")
if min_periods is None: if min_periods is None:
@@ -377,7 +377,7 @@ class DataHandlerLP(DataHandler):
Parameters Parameters
---------- ----------
init_type : str init_type : str
The type `IT_*` listed above The type `IT_*` listed above.
enable_cache : bool enable_cache : bool
default value is false: default value is false:
@@ -419,13 +419,13 @@ class DataHandlerLP(DataHandler):
Parameters Parameters
---------- ----------
selector : Union[pd.Timestamp, slice, str] selector : Union[pd.Timestamp, slice, str]
describe how to select data by index describe how to select data by index.
level : Union[str, int] level : Union[str, int]
which index level to select the data which index level to select the data.
col_set : str col_set : str
select a set of meaningful columns.(e.g. features, columns) select a set of meaningful columns.(e.g. features, columns).
data_key: str data_key : str
The data to fetch: DK_* the data to fetch: DK_*.
Returns Returns
------- -------
@@ -443,9 +443,9 @@ class DataHandlerLP(DataHandler):
Parameters Parameters
---------- ----------
col_set : str col_set : str
select a set of meaningful columns.(e.g. features, columns) select a set of meaningful columns.(e.g. features, columns).
data_key: str data_key : str
The data to fetch: DK_* the data to fetch: DK_*.
Returns Returns
------- -------

8
qlib/data/dataset/loader.py
View File

@@ -100,16 +100,16 @@ class DLWParser(DataLoader):
Parameters Parameters
---------- ----------
instruments : instruments :
the instruments the instruments.
exprs : list exprs : list
The expressions to describe the content of the data the expressions to describe the content of the data.
names : list names : list
The name of the data the name of the data.
Returns Returns
------- -------
pd.DataFrame: pd.DataFrame:
the queried dataframe the queried dataframe.
""" """
pass pass

6
qlib/data/dataset/processor.py
View File

@@ -21,7 +21,7 @@ def get_group_columns(df: pd.DataFrame, group: str):
Parameters Parameters
---------- ----------
df : pd.DataFrame df : pd.DataFrame
with multi of columns with multi of columns.
group : str group : str
the name of the feature group, i.e. the first level value of the group index. the name of the feature group, i.e. the first level value of the group index.
""" """
@@ -56,7 +56,7 @@ class Processor(Serializable):
Parameters Parameters
---------- ----------
df : pd.DataFrame df : pd.DataFrame
The raw_df of handler or result from previous processor The raw_df of handler or result from previous processor.
""" """
pass pass
@@ -68,7 +68,7 @@ class Processor(Serializable):
Returns Returns
------- -------
bool: bool:
if it is usable for infenrece if it is usable for infenrece.
""" """
return True return True

60
qlib/data/filter.py
View File

@@ -32,7 +32,7 @@ class BaseDFilter(abc.ABC):
Parameters Parameters
---------- ----------
config : dict config : dict
dict of config parameters dict of config parameters.
""" """
raise NotImplementedError("Subclass of BaseDFilter must reimplement `from_config` method") raise NotImplementedError("Subclass of BaseDFilter must reimplement `from_config` method")
@@ -43,7 +43,7 @@ class BaseDFilter(abc.ABC):
Returns Returns
---------- ----------
dict dict
return the dict of config parameters return the dict of config parameters.
""" """
raise NotImplementedError("Subclass of BaseDFilter must reimplement `to_config` method") raise NotImplementedError("Subclass of BaseDFilter must reimplement `to_config` method")
@@ -69,9 +69,9 @@ class SeriesDFilter(BaseDFilter):
Parameters Parameters
---------- ----------
fstart_time: str fstart_time: str
the time for the filter rule to start filter the instruments the time for the filter rule to start filter the instruments.
fend_time: str fend_time: str
the time for the filter rule to stop filter the instruments the time for the filter rule to stop filter the instruments.
""" """
super(SeriesDFilter, self).__init__() super(SeriesDFilter, self).__init__()
self.filter_start_time = pd.Timestamp(fstart_time) if fstart_time else None self.filter_start_time = pd.Timestamp(fstart_time) if fstart_time else None
@@ -83,12 +83,12 @@ class SeriesDFilter(BaseDFilter):
Parameters Parameters
---------- ----------
instruments: dict instruments: dict
the dict of instruments in the form {instrument_name => list of timestamp tuple} the dict of instruments in the form {instrument_name => list of timestamp tuple}.
Returns Returns
---------- ----------
pd.Timestamp, pd.Timestamp pd.Timestamp, pd.Timestamp
the lower time bound and upper time bound of all the instruments the lower time bound and upper time bound of all the instruments.
""" """
trange = Cal.calendar(freq=self.filter_freq) trange = Cal.calendar(freq=self.filter_freq)
ubound, lbound = trange[0], trange[-1] ubound, lbound = trange[0], trange[-1]
@@ -105,14 +105,14 @@ class SeriesDFilter(BaseDFilter):
Parameters Parameters
---------- ----------
time_range : D.calendar time_range : D.calendar
the time range of the instruments the time range of the instruments.
target_timestamp : list target_timestamp : list
the list of tuple (timestamp, timestamp) the list of tuple (timestamp, timestamp).
Returns Returns
---------- ----------
pd.Series pd.Series
the series of bool value for an instrument the series of bool value for an instrument.
""" """
# Construct a whole dict of {date => bool} # Construct a whole dict of {date => bool}
timestamp_series = {timestamp: False for timestamp in time_range} timestamp_series = {timestamp: False for timestamp in time_range}
@@ -124,19 +124,19 @@ class SeriesDFilter(BaseDFilter):
return timestamp_series return timestamp_series
def _filterSeries(self, timestamp_series, filter_series): def _filterSeries(self, timestamp_series, filter_series):
"""Filter the timestamp series with filter series by using element-wise AND operation of the two series """Filter the timestamp series with filter series by using element-wise AND operation of the two series.
Parameters Parameters
---------- ----------
timestamp_series : pd.Series timestamp_series : pd.Series
the series of bool value indicating existing time the series of bool value indicating existing time.
filter_series : pd.Series filter_series : pd.Series
the series of bool value indicating filter feature the series of bool value indicating filter feature.
Returns Returns
---------- ----------
pd.Series pd.Series
the series of bool value indicating whether the date satisfies the filter condition and exists in target timestamp the series of bool value indicating whether the date satisfies the filter condition and exists in target timestamp.
""" """
fstart, fend = list(filter_series.keys())[0], list(filter_series.keys())[-1] fstart, fend = list(filter_series.keys())[0], list(filter_series.keys())[-1]
filter_series = filter_series.astype("bool") # Make sure the filter_series is boolean filter_series = filter_series.astype("bool") # Make sure the filter_series is boolean
@@ -144,17 +144,17 @@ class SeriesDFilter(BaseDFilter):
return timestamp_series return timestamp_series
def _toTimestamp(self, timestamp_series): def _toTimestamp(self, timestamp_series):
"""Convert the timestamp series to a list of tuple (timestamp, timestamp) indicating a continuous range of TRUE """Convert the timestamp series to a list of tuple (timestamp, timestamp) indicating a continuous range of TRUE.
Parameters Parameters
---------- ----------
timestamp_series: pd.Series timestamp_series: pd.Series
the series of bool value after being filtered the series of bool value after being filtered.
Returns Returns
---------- ----------
list list
the list of tuple (timestamp, timestamp) the list of tuple (timestamp, timestamp).
""" """
# sort the timestamp_series according to the timestamps # sort the timestamp_series according to the timestamps
timestamp_series.sort_index() timestamp_series.sort_index()
@@ -194,18 +194,18 @@ class SeriesDFilter(BaseDFilter):
Parameters Parameters
---------- ----------
instruments : dict instruments : dict
the dict of instruments to be filtered the dict of instruments to be filtered.
fstart : pd.Timestamp fstart : pd.Timestamp
start time of filter start time of filter.
fend : pd.Timestamp fend : pd.Timestamp
end time of filter end time of filter.
.. note:: fstart/fend indicates the intersection of instruments start/end time and filter start/end time .. note:: fstart/fend indicates the intersection of instruments start/end time and filter start/end time.
Returns Returns
---------- ----------
pd.Dataframe pd.Dataframe
a series of {pd.Timestamp => bool} a series of {pd.Timestamp => bool}.
""" """
raise NotImplementedError("Subclass of SeriesDFilter must reimplement `getFilterSeries` method") raise NotImplementedError("Subclass of SeriesDFilter must reimplement `getFilterSeries` method")
@@ -215,16 +215,16 @@ class SeriesDFilter(BaseDFilter):
Parameters Parameters
---------- ----------
instruments: dict instruments: dict
input instruments to be filtered input instruments to be filtered.
start_time: str start_time: str
start of the time range start of the time range.
end_time: str end_time: str
end of the time range end of the time range.
Returns Returns
---------- ----------
dict dict
filtered instruments, same structure as input instruments filtered instruments, same structure as input instruments.
""" """
lbound, ubound = self._getTimeBound(instruments) lbound, ubound = self._getTimeBound(instruments)
start_time = pd.Timestamp(start_time or lbound) start_time = pd.Timestamp(start_time or lbound)
@@ -272,7 +272,7 @@ class NameDFilter(SeriesDFilter):
params: params:
------ ------
name_rule_re: str name_rule_re: str
regular expression for the name rule regular expression for the name rule.
""" """
super(NameDFilter, self).__init__(fstart_time, fend_time) super(NameDFilter, self).__init__(fstart_time, fend_time)
self.name_rule_re = name_rule_re self.name_rule_re = name_rule_re
@@ -325,13 +325,13 @@ class ExpressionDFilter(SeriesDFilter):
params: params:
------ ------
fstart_time: str fstart_time: str
filter the feature starting from this time filter the feature starting from this time.
fend_time: str fend_time: str
filter the feature ending by this time filter the feature ending by this time.
rule_expression: str rule_expression: str
an input expression for the rule an input expression for the rule.
keep: bool keep: bool
whether to keep the instruments of which features don't exist in the filter time span whether to keep the instruments of which features don't exist in the filter time span.
""" """
super(ExpressionDFilter, self).__init__(fstart_time, fend_time) super(ExpressionDFilter, self).__init__(fstart_time, fend_time)
self.rule_expression = rule_expression self.rule_expression = rule_expression

6
qlib/model/base.py
View File

@@ -33,7 +33,7 @@ class Model(BaseModel):
Parameters Parameters
---------- ----------
dataset : Dataset dataset : Dataset
dataset will generate the processed data from model training dataset will generate the processed data from model training.
""" """
raise NotImplementedError() raise NotImplementedError()
@@ -44,7 +44,7 @@ class Model(BaseModel):
Parameters Parameters
---------- ----------
dataset : Dataset dataset : Dataset
dataset will generate the processed dataset from model training dataset will generate the processed dataset from model training.
""" """
raise NotImplementedError() raise NotImplementedError()
@@ -59,6 +59,6 @@ class ModelFT(Model):
Parameters Parameters
---------- ----------
dataset : Dataset dataset : Dataset
dataset will generate the processed dataset from model training dataset will generate the processed dataset from model training.
""" """
raise NotImplementedError() raise NotImplementedError()

44
qlib/model/riskmodel.py
View File

@@ -23,9 +23,9 @@ class RiskModel(BaseModel):
def __init__(self, nan_option: str = "ignore", assume_centered: bool = False, scale_return: bool = True): def __init__(self, nan_option: str = "ignore", assume_centered: bool = False, scale_return: bool = True):
""" """
Args: Args:
nan_option (str): nan handling option (`ignore`/`mask`/`fill`) nan_option (str): nan handling option (`ignore`/`mask`/`fill`).
assume_centered (bool): whether the data is assumed to be centered assume_centered (bool): whether the data is assumed to be centered.
scale_return (bool): whether scale returns as percentage scale_return (bool): whether scale returns as percentage.
""" """
# nan # nan
assert nan_option in [ assert nan_option in [
@@ -45,11 +45,11 @@ class RiskModel(BaseModel):
Args: Args:
X (pd.Series, pd.DataFrame or np.ndarray): data from which to estimate the covariance, X (pd.Series, pd.DataFrame or np.ndarray): data from which to estimate the covariance,
with variables as columns and observations as rows. with variables as columns and observations as rows.
return_corr (bool): whether return the correlation matrix return_corr (bool): whether return the correlation matrix.
is_price (bool): whether `X` contains price (if not assume stock returns) is_price (bool): whether `X` contains price (if not assume stock returns).
Returns: Returns:
pd.DataFrame or np.ndarray: estimated covariance (or correlation) pd.DataFrame or np.ndarray: estimated covariance (or correlation).
""" """
# transform input into 2D array # transform input into 2D array
if not isinstance(X, (pd.Series, pd.DataFrame)): if not isinstance(X, (pd.Series, pd.DataFrame)):
@@ -101,10 +101,10 @@ class RiskModel(BaseModel):
By default, this method implements the empirical covariance estimation. By default, this method implements the empirical covariance estimation.
Args: Args:
X (np.ndarray): data matrix containing multiple variables (columns) and observations (rows) X (np.ndarray): data matrix containing multiple variables (columns) and observations (rows).
Returns: Returns:
np.ndarray: covariance matrix np.ndarray: covariance matrix.
""" """
xTx = np.asarray(X.T.dot(X)) xTx = np.asarray(X.T.dot(X))
N = len(X) N = len(X)
@@ -117,7 +117,7 @@ class RiskModel(BaseModel):
"""handle nan and centerize data """handle nan and centerize data
Note: Note:
if `nan_option='mask'` then the returned array will be `np.ma.MaskedArray` if `nan_option='mask'` then the returned array will be `np.ma.MaskedArray`.
""" """
# handle nan # handle nan
if self.nan_option == self.FILL_NAN: if self.nan_option == self.FILL_NAN:
@@ -139,15 +139,15 @@ class ShrinkCovEstimator(RiskModel):
where `alpha` is the shrink parameter and `F` is the shrinking target. where `alpha` is the shrink parameter and `F` is the shrinking target.
The following shrinking parameters (`alpha`) are supported: The following shrinking parameters (`alpha`) are supported:
- `lw` [1][2][3]: use Ledoit-Wolf shrinking parameter - `lw` [1][2][3]: use Ledoit-Wolf shrinking parameter.
- `oas` [4]: use Oracle Approximating Shrinkage shrinking parameter - `oas` [4]: use Oracle Approximating Shrinkage shrinking parameter.
- float: directly specify the shrink parameter, should be between [0, 1] - float: directly specify the shrink parameter, should be between [0, 1].
The following shrinking targets (`F`) are supported: The following shrinking targets (`F`) are supported:
- `const_var` [1][4][5]: assume stocks have the same constant variance and zero correlation - `const_var` [1][4][5]: assume stocks have the same constant variance and zero correlation.
- `const_corr` [2][6]: assume stocks have different variance but equal correlation - `const_corr` [2][6]: assume stocks have different variance but equal correlation.
- `single_factor` [3][7]: assume single factor model as the shrinking target - `single_factor` [3][7]: assume single factor model as the shrinking target.
- np.ndarray: provide the shrinking targets directly - np.ndarray: provide the shrinking targets directly.
Note: Note:
- The optimal shrinking parameter depends on the selection of the shrinking target. - The optimal shrinking parameter depends on the selection of the shrinking target.
@@ -402,13 +402,13 @@ class POETCovEstimator(RiskModel):
def __init__(self, num_factors: int = 0, thresh: float = 1.0, thresh_method: str = "soft", **kwargs): def __init__(self, num_factors: int = 0, thresh: float = 1.0, thresh_method: str = "soft", **kwargs):
""" """
Args: Args:
num_factors (int): number of factors (if set to zero, no factor model will be used) num_factors (int): number of factors (if set to zero, no factor model will be used).
thresh (float): the positive constant for thresholding thresh (float): the positive constant for thresholding.
thresh_method (str): thresholding method, which can be thresh_method (str): thresholding method, which can be
- 'soft': soft thresholding - 'soft': soft thresholding.
- 'hard': hard thresholding - 'hard': hard thresholding.
- 'scad': scad thresholding - 'scad': scad thresholding.
kwargs: see `RiskModel` for more information kwargs: see `RiskModel` for more information.
""" """
super().__init__(**kwargs) super().__init__(**kwargs)