liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-04 03:21:00 +08:00
Code Activity

release-0.5.0 (#1)

Browse Source 
* init commit

* change the version number

* rich the docs&fix cache docs

* update index readme

* Modify cache class name

* Modify sharpe to information_ratio

* Modify Group- to Group

* add the description of graphical results & fix the backtest docs

* fix docs in details

* update docs

* Update introduction.rst

* Update README.md

* Update introduction.rst

* Update introduction.rst

* Update introduction.rst

* Update installation.rst

* Update installation.rst

* Update initialization.rst

* Update getdata.rst

* Update integration.rst

* Update initialization.rst

* Update getdata.rst

* Update estimator.rst

Modify some typos.

* Update README.md

Modify the typos.

* Update initialization.rst

* Update data.rst

* Update report.rst

* Update estimator.rst

* Update cumulative_return.py

* Update model.rst

* Update rank_label.py

* Update cumulative_return.py

* Update strategy.rst

* Update getdata.rst

* Update backtest.rst

* Update integration.rst

* Update getdata.rst

* Update introduction.rst

* Update introduction.rst

* Update README.md

* Update report.rst

* Update integration.rst

Fix typos

* Update installation.rst

Fix typos

* Update getdata.rst

* Update initialization.rst

Fix typos.

* add quick start docs&fix detials

* fix estimator docs & fix strategy docs

* fix the cahce in data.rst

* update documents

* Fix Corr && Rsquare

* fix data retrival example to csi300 & fix a data bug

* fix filter bug

* Fix data collector

* Modift model args

* add the log & fix README.md\quick.rst

* add enviroment depend & add intoduction of qlib-server online mode

* fix image center fomat & set log_only of docs is True

* fix README.md format

* update data preparation & readme logo image

* get_data support version

* Modify analysis names

* Modify analysis graph

* update report.rst & data.rst

* commmit estimator for merge

* minimal requirements

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update READEME.md

* Update READEME.md

* update estimator

* Fix doc urls

* fix get_data.py docstring

* update test_get_data.py

* Upate docs

* Upate docs

* Upate docs

Co-authored-by: bxdd <bxddream@gmail.com>
Co-authored-by: zhupr <zhu.pengrong@foxmail.com>
Co-authored-by: Wendi Li <wendili.academic@qq.com>
Co-authored-by: Dingsu Wang <dingsu.wang@gmail.com>
Co-authored-by: bxdd <45119470+bxdd@users.noreply.github.com>
Co-authored-by: cslwqxx <cslwqxx@users.noreply.github.com>
This commit is contained in:
you-n-g
2020-09-23 23:01:39 -05:00
committed by GitHub
parent 99ebd87cba
commit de9e13b171
82 changed files with 1580 additions and 1145 deletions
Download Patch File Download Diff File Expand all files Collapse all files

251
docs/component/data.rst
View File

@@ -6,79 +6,106 @@ Data Layer: Data Framework&Usage
Introduction
============================
``Data Layer`` is designed to download raw data, retrieve data, construct datasets and get frequently-used data.
``Data Layer`` provides user-friendly APIs to manage and retrieve data. It provides high-performance data infrastructure.
Also, users can building formulaic alphas with ``Data Layer`` easliy. If users are interesting formulaic alphas, please refer to `Building Formulaic Alphas <../advanced/alpha.html>`_.
It is designed for quantitative investment. For example, users could build formulaic alphas with ``Data Layer`` easily. Please refer to `Building Formulaic Alphas <../advanced/alpha.html>`_ for more details.
The ``Data Layer`` framework includes four components as follows.
The introduction of ``Data Layer`` includes the following parts.
- Raw Data
- Data Preparation
- Data API
- Data Handler
- Cache
- Data and Cache File Structure
Raw Data
Data Preparation
============================
``Qlib`` provides the script ``scripts/get_data.py`` to download the raw data that will be used to initialize the qlib package, please refer to `Initialization <../start/initialization.rst>`_.
Qlib Format Data
------------------
When ``Qlib`` is initialized, users can choose china-stock mode or US-stock mode, please refer to `Initialization <../start/initialization.rst>`_.
We've specially designed a data structure to manage financial data, please refer to the `File storage design section in Qlib paper <https://arxiv.org/abs/2009.11189>`_ for detailed information.
Such data will be stored with filename suffix `.bin` (We'll call them `.bin` file, `.bin` format or qlib format). `.bin` file is designed for scientific computing on finance data
China-Stock Market Mode
Qlib Format Dataset
--------------------
``Qlib`` has provided an off-the-shelf dataset in `.bin` format, users could use the script ``scripts/get_data.py`` to download the dataset as follows.
.. code-block:: bash
python scripts/get_data.py qlib_data_cn --target_dir ~/.qlib/qlib_data/cn_data
After running the above command, users can find china-stock data in Qlib format in the ``~/.qlib/csv_data/cn_data`` directory.
``Qlib`` also provides the scripts in ``scripts/data_collector`` to help users crawl the latest data on the Internet and convert it to qlib format.
When ``Qlib`` is initialized with this dataset, users could build and evaluate their own models with it. Please refer to `Initialization <../start/initialization.html>`_ for more details.
Converting CSV Format into Qlib Format
-------------------------------------------
``Qlib`` has provided the script ``scripts/dump_bin.py`` to convert data in CSV format into `.bin` files(Qlib format).
Users can download the china-stock data in CSV format as follows for reference to the CSV format.
.. code-block:: bash
python scripts/get_data.py csv_data_cn --target_dir ~/.qlib/csv_data/cn_data
Supposed that users prepare their CSV format data in the directory ``~/.qlib/csv_data/my_data``, they can run the following command to start the conversion.
.. code-block:: bash
python scripts/dump_bin.py dump --csv_path ~/.qlib/csv_data/my_data --qlib_dir ~/.qlib/qlib_data/my_data --include_fields open,close,high,low,volume,factor
After conversion, users can find their Qlib format data in the directory `~/.qlib/qlib_data/my_data`.
.. note::
The arguments of `--include_fields` should correspond with the columns names of CSV files. The columns names of dataset provided by ``Qlib`` includes open,close,high,low,volume,factor.
- `open`
The opening price
- `close`
The closing price
- `high`
The highest price
- `low`
The lowest price
- `volume`
The trading volume
- `factor`
The Restoration factor
China-Stock Mode & US-Stock Mode
--------------------------------
If users use ``Qlib`` in china-stock mode, china-stock data is required. The script ``scripts/get_data.py`` can be used to download china-stock data. If users want to use ``Qlib`` in china-stock mode, they need to do as follows.
- If users use ``Qlib`` in china-stock mode, china-stock data is required. Users can use ``Qlib`` in china-stock mode according to the following steps:
- Download china-stock in qlib format, please refer to section `Qlib Format Dataset <#qlib-format-dataset>`_.
- Initialize ``Qlib`` in china-stock mode
Supposed that users download their Qlib format data in the directory ``~/.qlib/csv_data/cn_data``. Users only need to initialize ``Qlib`` as follows.
.. code-block:: python
- Download data in qlib format
Run the following command to download china-stock data in csv format.
.. code-block:: bash
python scripts/get_data.py qlib_data_cn --target_dir ~/.qlib/qlib_data/cn_data
Users can find china-stock data in qlib format in the'~/.qlib/csv_data/cn_data' directory.
- Initialize ``Qlib`` in china-stock mode
Users only need to initialize ``Qlib`` as follows.
.. code-block:: python
from qlib.config import REG_CN
qlib.init(provider_uri='~/.qlib/qlib_data/cn_data', region=REG_CN)
from qlib.config import REG_CN
qlib.init(provider_uri='~/.qlib/qlib_data/cn_data', region=REG_CN)
US-Stock Market Mode
-------------------------
If users use ``Qlib`` in US-stock mode, US-stock data is required. ``Qlib`` does not provide script to download US-stock data. If users want to use ``Qlib`` in US-stock market mode, they need to do as follows.
- Prepare data in csv format
Users need to prepare US-stock data in csv format by themselves, which is in the same format as the china-stock data in csv format. Please download the china-stock data in csv format as follows for reference of format.
.. code-block:: bash
python scripts/get_data.py csv_data_cn --target_dir ~/.qlib/csv_data/cn_data
- Convert data from csv format to ``Qlib`` format
``Qlib`` provides the script ``scripts/dump_bin.py`` to convert data from csv format to qlib format.
Assuming that the users store the US-stock data in csv format in path '~/.qlib/csv_data/us_data', they need to execute the following command to convert the data from csv format to ``Qlib`` format:
.. code-block:: bash
python scripts/dump_bin.py dump --csv_path ~/.qlib/csv_data/us_data --qlib_dir ~/.qlib/qlib_data/us_data --include_fields open,close,high,low,volume,factor
- Initialize ``Qlib`` in US-stock mode
Users only need to initialize ``Qlib`` as follows.
.. code-block:: python
from qlib.config import REG_US
qlib.init(provider_uri='~/.qlib/qlib_data/us_data', region=REG_US)
- If users use ``Qlib`` in US-stock mode, US-stock data is required. ``Qlib`` does not provide a script to download US-stock data. Users can use ``Qlib`` in US-stock mode according to the following steps:
- Prepare data in CSV format
- Convert data from CSV format to Qlib format, please refer to section `Converting CSV Format into Qlib Format <#converting-csv-format-into-qlib-format>`_.
- Initialize ``Qlib`` in US-stock mode
Supposed that users prepare their Qlib format data in the directory ``~/.qlib/csv_data/us_data``. Users only need to initialize ``Qlib`` as follows.
.. code-block:: python
Please refer to `Script API <../reference/api.html>`_ for more details.
from qlib.config import REG_US
qlib.init(provider_uri='~/.qlib/qlib_data/us_data', region=REG_US)
Data API
========================
@@ -90,10 +117,10 @@ Users can use APIs in ``qlib.data`` to retrieve data, please refer to `Data Retr
Feature
------------------
``Qlib`` provides `Feature` and `ExpressionOps` to fetch the features according to users' need.
``Qlib`` provides `Feature` and `ExpressionOps` to fetch the features according to users' needs.
- `Feature`
Load data from data provider.
Load data from the data provider. User can get the features like `$high`, `$low`, `$open`, `$close`, .etc, which should correspond with the arguments of `--include_fields`, please refer to section `Converting CSV Format into Qlib Format <#converting-csv-format-into-qlib-format>`_.
- `ExpressionOps`
`ExpressionOps` will use operator for feature construction.
@@ -103,7 +130,7 @@ To know more about ``Feature``, please refer to `Feature API <../reference/api.
Filter
-------------------
``Qlib`` provides `NameDFilter` and `ExpressionDFilter` to filter the instruments according to users' need.
``Qlib`` provides `NameDFilter` and `ExpressionDFilter` to filter the instruments according to users' needs.
- `NameDFilter`
Name dynamic instrument filter. Filter the instruments based on a regulated name format. A name rule regular expression is required.
@@ -121,14 +148,14 @@ To know more about ``Filter``, please refer to `Filter API <../reference/api.htm
API
-------------
To know more about ``Data Api``, please refer to `Data Api <../reference/api.html>`_.
To know more about ``Data API``, please refer to `Data API <../reference/api.html>`_.
Data Handler
=================
``Data Handler`` is a part of ``estimator`` and can also be used as a single module.
Users can use ``Data Handler`` in an automatic workflow by ``Estimator``, refer to `Estimator <estimator.html>`_ for more details.
``Data Handler`` can be used to load raw data, prepare features and label columns, preprocess data(standardization, remove NaN, etc.), split training, validation, and test sets. It is a subclass of ``qlib.contrib.estimator.handler.BaseDataHandler``, which provides some interfaces, for example:
Also, ``Data Handler`` can be used as an independent module, by which users can easily preprocess data(standardization, remove NaN, etc.) and build datasets. It is a subclass of ``qlib.contrib.estimator.handler.BaseDataHandler``, which provides some interfaces as follows.
Base Class & Interface
----------------------
@@ -139,20 +166,20 @@ Qlib provides a base class `qlib.contrib.estimator.BaseDataHandler <../reference
Implement the interface to load the data features.
- `setup_label`
Implement the interface to load the data labels and calculate user's labels.
Implement the interface to load the data labels and calculate the users' labels.
- `setup_processed_data`
Implement the interface for data preprocessing, such as preparing feature columns, discarding blank lines, and so on.
Qlib also provides two functions to help user init the data handler, user can override them for user's need.
Qlib also provides two functions to help users init the data handler, users can override them for users' needs.
- `_init_kwargs`
User can init the kwargs of the data handler in this function, some kwargs may be used when init the raw df.
Users can init the kwargs of the data handler in this function, some kwargs may be used when init the raw df.
Kwargs are the other attributes in data.args, like dropna_label, dropna_feature
- `_init_raw_df`
User can init the raw df, feature names and label names of data handler in this function.
If the index of feature df and label df are not same, user need to override this method to merge them (e.g. inner, left, right merge).
Users can init the raw df, feature names, and label names of data handler in this function.
If the index of feature df and label df are not same, users need to override this method to merge them (e.g. inner, left, right merge).
If users want to load features and labels by config, users can inherit ``qlib.contrib.estimator.handler.ConfigDataHandler``, ``Qlib`` also have provided some preprocess method in this subclass.
If users want to use qlib data, `QLibDataHandler` is recommended. Users can inherit their custom class from `QLibDataHandler`, which is also a subclass of `ConfigDataHandler`.
@@ -160,7 +187,8 @@ If users want to use qlib data, `QLibDataHandler` is recommended. Users can inhe
Usage
--------------
'Data Handler' can be used as a single module, which provides the following mehtod:
``Data Handler`` can be used as a single module, which provides the following mehtods:
- `get_split_data`
- According to the start and end dates, return features and labels of the pandas DataFrame type used for the 'Model'
@@ -178,21 +206,21 @@ Example
Know more about how to run ``Data Handler`` with ``estimator``, please refer to `Estimator <estimator.html#about-data>`_.
Qlib provides implemented data handler `QLibDataHandlerV1`. The following example shows how to run 'QLibDataHandlerV1' as a single module.
Qlib provides implemented data handler `QLibDataHandlerClose`. The following example shows how to run `QLibDataHandlerV1` as a single module.
.. note:: User needs to initialize ``Qlib`` with `qlib.init` first, please refer to `initialization <initialization.rst>`_.
.. note:: Users need to initialize ``Qlib`` with `qlib.init` first, please refer to `initialization <../start/initialization.html>`_.
.. code-block:: Python
from qlib.contrib.estimator.handler import QLibDataHandlerV1
from qlib.contrib.estimator.handler import QLibDataHandlerClose
from qlib.contrib.model.gbdt import LGBModel
DATA_HANDLER_CONFIG = {
"dropna_label": True,
"start_date": "2007-01-01",
"end_date": "2020-08-01",
"market": "csi500",
"market": "csi300",
}
TRAINER_CONFIG = {
@@ -204,7 +232,7 @@ Qlib provides implemented data handler `QLibDataHandlerV1`. The following exampl
"test_end_date": "2020-08-01",
}
exampleDataHandler = QLibDataHandlerV1(**DATA_HANDLER_CONFIG)
exampleDataHandler = QLibDataHandlerClose(**DATA_HANDLER_CONFIG)
# example of 'get_split_data'
x_train, y_train, x_validate, y_validate, x_test, y_test = exampleDataHandler.get_split_data(**TRAINER_CONFIG)
@@ -222,22 +250,17 @@ Also, the above example has been given in ``examples.estimator.train_backtest_an
API
---------
To know more abot ``Data Handler``, please refer to `Data Handler API <../reference/api.html#handler>`_.
To know more about ``Data Handler``, please refer to `Data Handler API <../reference/api.html#handler>`_.
Cache
==========
``Cache`` is an optional module that helps accelerate providing data by saving some frequently-used data as cache file.
``Cache`` is an optional module that helps accelerate providing data by saving some frequently-used data as cache file. ``Qlib`` provides a `Memcache` class to cache the most-frequently-used data in memory, an inheritable `ExpressionCache` class and an inheritable `DatasetCache` class.
Memory Cache
--------------
Global Memory Cache
---------------------
Base Class & Interface
~~~~~~~~~~~~~~~~~~~~~~~
``Qlib`` provides a `Memcache` class to cache the most-frequently-used data in memory, an inheritable `ExpressionCache` class, and an inheritable `DatasetCache` class.
`Memcache` is a memory cache mechanism that composes of three `MemCacheUnit` instances to cache **Calendar**, **Instruments**, and **Features**. The MemCache is defined globally in `cache.py` as `H`. User can use `H['c'], H['i'], H['f']` to get/set memcache.
`Memcache` is a global memory cache mechanism that composes of three `MemCacheUnit` instances to cache **Calendar**, **Instruments**, and **Features**. The `MemCache` is defined globally in `cache.py` as `H`. Users can use `H['c'], H['i'], H['f']` to get/set `memcache`.
.. autoclass:: qlib.data.cache.MemCacheUnit
:members:
@@ -246,60 +269,42 @@ Base Class & Interface
:members:
Disk Cache
--------------
Base Class & Interface
~~~~~~~~~~~~~~~~~~~~~~~
`ExpressionCache` is a disk cache mechanism that saves expressions such as **Mean($close, 5)**. Users can inherit this base class to define their own cache mechanism. Users need to override `self._uri` method to define how their cache file path is generated, `self._expression` method to define what data they want to cache and how to cache it.
`DatasetCache` is a disk cache mechanism that saves datasets. A certain dataset is regulated by a stockpool configuration (or a series of instruments, though not recommended), a list of expressions or static feature fields, the start time and end time for the collected features and the frequency. Users need to override `self._uri` method to define how their cache file path is generated, `self._expression` method to define what data they want to cache and how to cache it.
`ExpressionCache` and `DatasetCache` actually provides the same interfaces with `ExpressionProvider` and `DatasetProvider` so that the disk cache layer is transparent to users and will only be used if they want to define their own cache mechanism. The users can plug the cache mechanism into the server system by assigning the cache class they want to use in `config.py`:
.. code-block:: python
'ExpressionCache': 'ServerExpressionCache',
'DatasetCache': 'ServerDatasetCache',
Users can find the cache interface here.
ExpressionCache
^^^^^^^^^^^^^^^^^^^^
-----------------
`ExpressionCache` is a cache mechanism that saves expressions such as **Mean($close, 5)**. Users can inherit this base class to define their own cache mechanism that saves expressions according to the following steps.
- Override `self._uri` method to define how the cache file path is generated
- Override `self._expression` method to define what data will be cached and how to cache it.
The following shows the details about the interfaces:
.. autoclass:: qlib.data.cache.ExpressionCache
:members:
``Qlib`` has currently provided implemented disk cache `DiskExpressionCache` which inherits from `ExpressionCache` . The expressions data will be stored in the disk.
DatasetCache
^^^^^^^^^^^^^^^^^^^^
-----------------
`DatasetCache` is a cache mechanism that saves datasets. A certain dataset is regulated by a stock pool configuration (or a series of instruments, though not recommended), a list of expressions or static feature fields, the start time, and end time for the collected features and the frequency. Users can inherit this base class to define their own cache mechanism that saves datasets according to the following steps.
- Override `self._uri` method to define how their cache file path is generated
- Override `self._expression` method to define what data will be cached and how to cache it.
The following shows the details about the interfaces:
.. autoclass:: qlib.data.cache.DatasetCache
:members:
``Qlib`` has currently provided implemented disk cache `DiskDatasetCache` which inherits from `DatasetCache` . The datasets data will be stored in the disk.
Implemented Disk Cache
~~~~~~~~~~~~~~~~~~~~~~~
.. note::
If the user does not use QlibServer, please ignore the content of this section
Qlib has currently provided `ServerExpressionCache` class and `ServerDatasetCache` class as the cache mechanisms used for QlibServer. The class interface and file structure designed for server cache mechanism is listed below.
DiskExpressionCache
^^^^^^^^^^^^^^^^^^^^
.. autoclass:: qlib.data.cache.ServerExpressionCache
DiskDatasetCache
^^^^^^^^^^^^^^^^^^^^
.. autoclass:: qlib.data.cache.ServerDatasetCache
Data and Cache File Structure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
==================================
We've specially designed a file structure to manage data and cache, please refer to the `File storage design section in Qlib paper <https://arxiv.org/abs/2009.11189>`_ for detailed information.The file structure of data and cache is listed as follows.
.. code-block:: json
@@ -317,7 +322,7 @@ Data and Cache File Structure
- close.day.bin
- ...
- ...
[cached data] updated by server when raw data is updated
[cached data] updated when raw data is updated
- calculated features/
- sh600000/
- [hash(instrtument, field_expression, freq)]
@@ -331,3 +336,5 @@ Data and Cache File Structure
- .index : an assorted index file recording the line index of all calendars
- ...
.. TODO: refer to paper