liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-06-06 05:51:17 +08:00
Code Activity

Improve the style of documentation (#1132)

Browse Source 
This commit improves the documentation (rst files) only in the
following three ways:

* Aligned section headers with their underline/overline punctuation characters

* Deleted all trailling whitespaces in rst files

* Deleted a few trailling newlines at the end of the rst files

Co-authored-by: Bingyao Liu <Bingyao.Liu@sofund.com>
This commit is contained in:
YaOzI
2022-07-07 19:42:27 +08:00
committed by GitHub
parent e62684eddf
commit 1dededa33f
29 changed files with 400 additions and 411 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
docs/FAQ/FAQ.rst
View File

@@ -3,7 +3,7 @@ Qlib FAQ
############
Qlib Frequently Asked Questions
================================
===============================
.. contents::
:depth: 1
:local:
@@ -13,7 +13,7 @@ Qlib Frequently Asked Questions
1. RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase...
------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------
.. code-block:: console
@@ -52,7 +52,7 @@ This is caused by the limitation of multiprocessing under windows OS. Please ref
2. qlib.data.cache.QlibCacheException: It sees the key(...) of the redis lock has existed in your redis db now.
-----------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------
It sees the key of the redis lock has existed in your redis db now. You can use the following command to clear your redis keys and rerun your commands
@@ -72,7 +72,7 @@ If the issue is not resolved, use ``keys *`` to find if multiple keys exist. If
Also, feel free to post a new issue in our GitHub repository. We always check each issue carefully and try our best to solve them.
3. ModuleNotFoundError: No module named 'qlib.data._libs.rolling'
------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------
.. code-block:: python
@@ -101,7 +101,7 @@ Also, feel free to post a new issue in our GitHub repository. We always check ea
4. BadNamespaceError: / is not a connected namespace
------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------
.. code-block:: python
@@ -125,7 +125,7 @@ Also, feel free to post a new issue in our GitHub repository. We always check ea
5. TypeError: send() got an unexpected keyword argument 'binary'
------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------
.. code-block:: python

6
docs/advanced/PIT.rst
View File

@@ -1,14 +1,14 @@
.. _pit:
===========================
============================
(P)oint-(I)n-(T)ime Database
===========================
============================
.. currentmodule:: qlib
Introduction
------------
Point-in-time data is a very important consideration when performing any sort of historical market analysis.
Point-in-time data is a very important consideration when performing any sort of historical market analysis.
For example, let’s say we are backtesting a trading strategy and we are using the past five years of historical data as our input.
Our model is assumed to trade once a day, at the market close, and we’ll say we are calculating the trading signal for 1 January 2020 in our backtest. At that point, we should only have data for 1 January 2020, 31 December 2019, 30 December 2019 etc.

22
docs/advanced/alpha.rst
View File

@@ -1,12 +1,12 @@
.. _alpha:
===========================
Building Formulaic Alphas
===========================
=========================
Building Formulaic Alphas
=========================
.. currentmodule:: qlib
Introduction
===================
============
In quantitative trading practice, designing novel factors that can explain and predict future asset returns are of vital importance to the profitability of a strategy. Such factors are usually called alpha factors, or alphas in short.
@@ -15,28 +15,28 @@ A formulaic alpha, as the name suggests, is a kind of alpha that can be presente
Building Formulaic Alphas in ``Qlib``
======================================
=====================================
In ``Qlib``, users can easily build formulaic alphas.
Example
-----------------
-------
`MACD`, short for moving average convergence/divergence, is a formulaic alpha used in technical analysis of stock prices. It is designed to reveal changes in the strength, direction, momentum, and duration of a trend in a stock's price.
`MACD` can be presented as the following formula:
.. math::
.. math::
MACD = 2\times (DIF-DEA)
.. note::
`DIF` means Differential value, which is 12-period EMA minus 26-period EMA.
.. math::
DIF = \frac{EMA(CLOSE, 12) - EMA(CLOSE, 26)}{CLOSE}
DIF = \frac{EMA(CLOSE, 12) - EMA(CLOSE, 26)}{CLOSE}
`DEA`means a 9-period EMA of the DIF.
@@ -65,7 +65,7 @@ Users can use ``Data Handler`` to build formulaic alphas `MACD` in qlib:
>> print(df)
feature label
MACD LABEL
datetime instrument
datetime instrument
2010-01-04 SH600000 -0.011547 -0.019672
SH600004 0.002745 -0.014721
SH600006 0.010133 0.002911
@@ -79,7 +79,7 @@ Users can use ``Data Handler`` to build formulaic alphas `MACD` in qlib:
SZ300315 -0.030557 0.012455
Reference
===========
=========
To learn more about ``Data Loader``, please refer to `Data Loader <../component/data.html#data-loader>`_

20
docs/advanced/serial.rst
View File

@@ -1,26 +1,26 @@
.. _serial:
=================================
=============
Serialization
=================================
=============
.. currentmodule:: qlib
Introduction
===================
``Qlib`` supports dumping the state of ``DataHandler``, ``DataSet``, ``Processor`` and ``Model``, etc. into a disk and reloading them.
============
``Qlib`` supports dumping the state of ``DataHandler``, ``DataSet``, ``Processor`` and ``Model``, etc. into a disk and reloading them.
Serializable Class
========================
==================
``Qlib`` provides a base class ``qlib.utils.serial.Serializable``, whose state can be dumped into or loaded from disk in `pickle` format.
``Qlib`` provides a base class ``qlib.utils.serial.Serializable``, whose state can be dumped into or loaded from disk in `pickle` format.
When users dump the state of a ``Serializable`` instance, the attributes of the instance whose name **does not** start with `_` will be saved on the disk.
However, users can use ``config`` method or override ``default_dump_all`` attribute to prevent this feature.
Users can also override ``pickle_backend`` attribute to choose a pickle backend. The supported value is "pickle" (default and common) and "dill" (dump more things such as function, more information in `here <https://pypi.org/project/dill/>`_).
Example
==========================
``Qlib``'s serializable class includes ``DataHandler``, ``DataSet``, ``Processor`` and ``Model``, etc., which are subclass of ``qlib.utils.serial.Serializable``.
=======
``Qlib``'s serializable class includes ``DataHandler``, ``DataSet``, ``Processor`` and ``Model``, etc., which are subclass of ``qlib.utils.serial.Serializable``.
Specifically, ``qlib.data.dataset.DatasetH`` is one of them. Users can serialize ``DatasetH`` as follows.
.. code-block:: Python
@@ -33,7 +33,7 @@ Specifically, ``qlib.data.dataset.DatasetH`` is one of them. Users can serialize
dataset = pickle.load(file_dataset)
.. note::
Only state of ``DatasetH`` should be saved on the disk, such as some `mean` and `variance` used for data normalization, etc.
Only state of ``DatasetH`` should be saved on the disk, such as some `mean` and `variance` used for data normalization, etc.
After reloading the ``DatasetH``, users need to reinitialize it. It means that users can reset some states of ``DatasetH`` or ``QlibDataHandler`` such as `instruments`, `start_time`, `end_time` and `segments`, etc., and generate new data according to the states (data is not state and should not be saved on the disk).
@@ -41,5 +41,5 @@ A more detailed example is in this `link <https://github.com/microsoft/qlib/tree
API
===================
===
Please refer to `Serializable API <../reference/api.html#module-qlib.utils.serial.Serializable>`_.

16
docs/advanced/server.rst
View File

@@ -1,15 +1,15 @@
.. _server:
=================================
=============================
``Online`` & ``Offline`` mode
=================================
=============================
.. currentmodule:: qlib
Introduction
=============
============
``Qlib`` supports ``Online`` mode and ``Offline`` mode. Only the ``Offline`` mode is introduced in this document.
``Qlib`` supports ``Online`` mode and ``Offline`` mode. Only the ``Offline`` mode is introduced in this document.
The ``Online`` mode is designed to solve the following problems:
@@ -18,12 +18,12 @@ The ``Online`` mode is designed to solve the following problems:
- Make the data can be accessed in a remote way.
Qlib-Server
===============
===========
``Qlib-Server`` is the assorted server system for ``Qlib``, which utilizes ``Qlib`` for basic calculations and provides extensive server system and cache mechanism. With QLibServer, the data provided for ``Qlib`` can be managed in a centralized manner. With ``Qlib-Server``, users can use ``Qlib`` in ``Online`` mode.
``Qlib-Server`` is the assorted server system for ``Qlib``, which utilizes ``Qlib`` for basic calculations and provides extensive server system and cache mechanism. With QLibServer, the data provided for ``Qlib`` can be managed in a centralized manner. With ``Qlib-Server``, users can use ``Qlib`` in ``Online`` mode.
Reference
=================
If users are interested in ``Qlib-Server`` and ``Online`` mode, please refer to `Qlib-Server Project <https://github.com/microsoft/qlib-server>`_ and `Qlib-Server Document <https://qlib-server.readthedocs.io/en/latest/>`_.
=========
If users are interested in ``Qlib-Server`` and ``Online`` mode, please refer to `Qlib-Server Project <https://github.com/microsoft/qlib-server>`_ and `Qlib-Server Document <https://qlib-server.readthedocs.io/en/latest/>`_.

10
docs/advanced/task_management.rst
View File

@@ -1,13 +1,13 @@
.. _task_management:
=================================
===============
Task Management
=================================
===============
.. currentmodule:: qlib
Introduction
=============
============
The `Workflow <../component/introduction.html>`_ part introduces how to run research workflow in a loosely-coupled way. But it can only execute one ``task`` when you use ``qrun``.
To automatically generate and execute different tasks, ``Task Management`` provides a whole process including `Task Generating`_, `Task Storing`_, `Task Training`_ and `Task Collecting`_.
@@ -36,7 +36,7 @@ Here is the base class of ``TaskGen``:
This class allows users to verify the effect of data from different periods on the model in one experiment. More information is `here <../reference/api.html#TaskGen>`_.
Task Storing
===============
============
To achieve higher efficiency and the possibility of cluster operation, ``Task Manager`` will store all tasks in `MongoDB <https://www.mongodb.com/>`_.
``TaskManager`` can fetch undone tasks automatically and manage the lifecycle of a set of tasks with error handling.
Users **MUST** finish the configuration of `MongoDB <https://www.mongodb.com/>`_ when using this module.
@@ -57,7 +57,7 @@ Users need to provide the MongoDB URL and database name for using ``TaskManager`
More information of ``Task Manager`` can be found in `here <../reference/api.html#TaskManager>`_.
Task Training
===============
=============
After generating and storing those ``task``, it's time to run the ``task`` which is in the *WAITING* status.
``Qlib`` provides a method called ``run_task`` to run those ``task`` in task pool, however, users can also customize how tasks are executed.
An easy way to get the ``task_func`` is using ``qlib.model.trainer.task_train`` directly.

1
docs/changelog/changelog.rst
View File

@@ -1,2 +1 @@
.. include:: ../../CHANGES.rst

105
docs/component/data.rst
View File

@@ -1,13 +1,13 @@
.. _data:
================================
==================================
Data Layer: Data Framework & Usage
================================
==================================
Introduction
============================
============
``Data Layer`` provides user-friendly APIs to manage and retrieve data. It provides high-performance data infrastructure.
``Data Layer`` provides user-friendly APIs to manage and retrieve data. It provides high-performance data infrastructure.
It is designed for quantitative investment. For example, users could build formulaic alphas with ``Data Layer`` easily. Please refer to `Building Formulaic Alphas <../advanced/alpha.html>`_ for more details.
@@ -23,16 +23,16 @@ The introduction of ``Data Layer`` includes the following parts.
Here is a typical example of Qlib data workflow
- Users download data and converting data into Qlib format(with filename suffix `.bin`). In this step, typically only some basic data are stored on disk(such as OHLCV).
- Users download data and converting data into Qlib format(with filename suffix `.bin`). In this step, typically only some basic data are stored on disk(such as OHLCV).
- Creating some basic features based on Qlib's expression Engine(e.g. "Ref($close, 60) / $close", the return of last 60 trading days). Supported operators in the expression engine can be found `here <https://github.com/microsoft/qlib/blob/main/qlib/data/ops.py>`_. This step is typically implemented in Qlib's `Data Loader <https://qlib.readthedocs.io/en/latest/component/data.html#data-loader>`_ which is a component of `Data Handler <https://qlib.readthedocs.io/en/latest/component/data.html#data-handler>`_ .
- If users require more complicated data processing (e.g. data normalization), `Data Handler <https://qlib.readthedocs.io/en/latest/component/data.html#data-handler>`_ support user-customized processors to process data(some predefined processors can be found `here <https://github.com/microsoft/qlib/blob/main/qlib/data/dataset/processor.py>`_). The processors are different from operators in expression engine. It is designed for some complicated data processing methods which is hard to supported in operators in expression engine.
- At last, `Dataset <https://qlib.readthedocs.io/en/latest/component/data.html#dataset>`_ is responsible to prepare model-specific dataset from the processed data of Data Handler
Data Preparation
============================
================
Qlib Format Data
------------------
----------------
We've specially designed a data structure to manage financial data, please refer to the `File storage design section in Qlib paper <https://arxiv.org/abs/2009.11189>`_ for detailed information.
Such data will be stored with filename suffix `.bin` (We'll call them `.bin` file, `.bin` format, or qlib format). `.bin` file is designed for scientific computing on finance data.
@@ -50,9 +50,9 @@ Alpha158 √ √
Also, ``Qlib`` provides a high-frequency dataset. Users can run a high-frequency dataset example through this `link <https://github.com/microsoft/qlib/tree/main/examples/highfreq>`_.
Qlib Format Dataset
--------------------
-------------------
``Qlib`` has provided an off-the-shelf dataset in `.bin` format, users could use the script ``scripts/get_data.py`` to download the China-Stock dataset as follows.
The price volume data look different from the actual dealling price because of they are **adjusted** (`adjusted price <https://www.investopedia.com/terms/a/adjusted_closing_price.asp>`_). And then you may find that the adjusted price may be different from different data sources. This is because different data sources may vary in the way of adjusting prices. Qlib normalize the price on first trading day of each stock to 1 when adjusting them.
The price volume data look different from the actual dealling price because of they are **adjusted** (`adjusted price <https://www.investopedia.com/terms/a/adjusted_closing_price.asp>`_). And then you may find that the adjusted price may be different from different data sources. This is because different data sources may vary in the way of adjusting prices. Qlib normalize the price on first trading day of each stock to 1 when adjusting them.
Users can leverage `$factor` to get the original trading price (e.g. `$close / $factor` to get the original close price).
.. code-block:: bash
@@ -104,7 +104,7 @@ Automatic update of daily frequency data
Converting CSV Format into Qlib Format
-------------------------------------------
--------------------------------------
``Qlib`` has provided the script ``scripts/dump_bin.py`` to convert **any** data in CSV format into `.bin` files (``Qlib`` format) as long as they are in the correct format.
@@ -126,16 +126,16 @@ Users can also provide their own data in CSV format. However, the CSV data **mus
- CSV file is named after a specific stock *or* the CSV file includes a column of the stock name
- Name the CSV file after a stock: `SH600000.csv`, `AAPL.csv` (not case sensitive).
- CSV file includes a column of the stock name. User **must** specify the column name when dumping the data. Here is an example:
.. code-block:: bash
python scripts/dump_bin.py dump_all ... --symbol_field_name symbol
where the data are in the following format:
.. code-block::
.. code-block::
symbol,close
SH600000,120
@@ -145,10 +145,10 @@ Users can also provide their own data in CSV format. However, the CSV data **mus
.. code-block:: bash
python scripts/dump_bin.py dump_all ... --date_field_name date
where the data are in the following format:
.. code-block::
.. code-block::
symbol,date,close,open,volume
SH600000,2020-11-01,120,121,12300000
@@ -172,7 +172,7 @@ After conversion, users can find their Qlib format data in the directory `~/.qli
.. note::
The arguments of `--include_fields` should correspond with the column names of CSV files. The columns names of dataset provided by ``Qlib`` should include open, close, high, low, volume and factor at least.
- `open`
The adjusted opening price
- `close`
@@ -186,11 +186,11 @@ After conversion, users can find their Qlib format data in the directory `~/.qli
- `factor`
The Restoration factor. Normally, ``factor = adjusted_price / original_price``, `adjusted price` reference: `split adjusted <https://www.investopedia.com/terms/s/splitadjusted.asp>`_
In the convention of `Qlib` data processing, `open, close, high, low, volume, money and factor` will be set to NaN if the stock is suspended.
In the convention of `Qlib` data processing, `open, close, high, low, volume, money and factor` will be set to NaN if the stock is suspended.
If you want to use your own alpha-factor which can't be calculate by OCHLV, like PE, EPS and so on, you could add it to the CSV files with OHCLV together and then dump it to the Qlib format data.
Stock Pool (Market)
--------------------------------
-------------------
``Qlib`` defines `stock pool <https://github.com/microsoft/qlib/blob/main/examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml#L4>`_ as stock list and their date ranges. Predefined stock pools (e.g. csi300) may be imported as follows.
@@ -200,7 +200,7 @@ Stock Pool (Market)
Multiple Stock Modes
--------------------------------
--------------------
``Qlib`` now provides two different stock modes for users: China-Stock Mode & US-Stock Mode. Here are some different settings of these two modes:
@@ -218,23 +218,23 @@ The `trade unit` defines the unit number of stocks can be used in a trade, and t
- Download china-stock in qlib format, please refer to section `Qlib Format Dataset <#qlib-format-dataset>`_.
- Initialize ``Qlib`` in china-stock mode
Supposed that users download their Qlib format data in the directory ``~/.qlib/qlib_data/cn_data``. Users only need to initialize ``Qlib`` as follows.
.. code-block:: python
from qlib.constant import REG_CN
qlib.init(provider_uri='~/.qlib/qlib_data/cn_data', region=REG_CN)
- If users use ``Qlib`` in US-stock mode, US-stock data is required. ``Qlib`` also provides a script to download US-stock data. Users can use ``Qlib`` in US-stock mode according to the following steps:
- Download us-stock in qlib format, please refer to section `Qlib Format Dataset <#qlib-format-dataset>`_.
- Initialize ``Qlib`` in US-stock mode
Supposed that users prepare their Qlib format data in the directory ``~/.qlib/qlib_data/us_data``. Users only need to initialize ``Qlib`` as follows.
.. code-block:: python
from qlib.config import REG_US
qlib.init(provider_uri='~/.qlib/qlib_data/us_data', region=REG_US)
.. note::
@@ -242,14 +242,14 @@ The `trade unit` defines the unit number of stocks can be used in a trade, and t
Data API
========================
========
Data Retrieval
---------------
--------------
Users can use APIs in ``qlib.data`` to retrieve data, please refer to `Data Retrieval <../start/getdata.html>`_.
Feature
------------------
-------
``Qlib`` provides `Feature` and `ExpressionOps` to fetch the features according to users' needs.
@@ -264,7 +264,7 @@ Feature
To know more about ``Feature``, please refer to `Feature API <../reference/api.html#module-qlib.data.base>`_.
Filter
-------------------
------
``Qlib`` provides `NameDFilter` and `ExpressionDFilter` to filter the instruments according to users' needs.
- `NameDFilter`
@@ -272,7 +272,7 @@ Filter
- `ExpressionDFilter`
Expression dynamic instrument filter. Filter the instruments based on a certain expression. An expression rule indicating a certain feature field is required.
- `basic features filter`: rule_expression = '$close/$open>5'
- `cross-sectional features filter` \: rule_expression = '$rank($close)<10'
- `time-sequence features filter`: rule_expression = '$Ref($close, 3)>100'
@@ -299,29 +299,29 @@ Here is a simple example showing how to use filter in a basic ``Qlib`` workflow
To know more about ``Filter``, please refer to `Filter API <../reference/api.html#module-qlib.data.filter>`_.
Reference
-------------
---------
To know more about ``Data API``, please refer to `Data API <../reference/api.html#data>`_.
Data Loader
=================
===========
``Data Loader`` in ``Qlib`` is designed to load raw data from the original data source. It will be loaded and used in the ``Data Handler`` module.
QlibDataLoader
---------------
--------------
The ``QlibDataLoader`` class in ``Qlib`` is such an interface that allows users to load raw data from the ``Qlib`` data source.
StaticDataLoader
---------------
----------------
The ``StaticDataLoader`` class in ``Qlib`` is such an interface that allows users to load raw data from file or as provided.
Interface
------------
---------
Here are some interfaces of the ``QlibDataLoader`` class:
@@ -329,28 +329,28 @@ Here are some interfaces of the ``QlibDataLoader`` class:
:members:
API
-----------
---
To know more about ``Data Loader``, please refer to `Data Loader API <../reference/api.html#module-qlib.data.dataset.loader>`_.
Data Handler
=================
============
The ``Data Handler`` module in ``Qlib`` is designed to handler those common data processing methods which will be used by most of the models.
Users can use ``Data Handler`` in an automatic workflow by ``qrun``, refer to `Workflow: Workflow Management <workflow.html>`_ for more details.
Users can use ``Data Handler`` in an automatic workflow by ``qrun``, refer to `Workflow: Workflow Management <workflow.html>`_ for more details.
DataHandlerLP
--------------
-------------
In addition to use ``Data Handler`` in an automatic workflow with ``qrun``, ``Data Handler`` can be used as an independent module, by which users can easily preprocess data (standardization, remove NaN, etc.) and build datasets.
In addition to use ``Data Handler`` in an automatic workflow with ``qrun``, ``Data Handler`` can be used as an independent module, by which users can easily preprocess data (standardization, remove NaN, etc.) and build datasets.
In order to achieve so, ``Qlib`` provides a base class `qlib.data.dataset.DataHandlerLP <../reference/api.html#qlib.data.dataset.handler.DataHandlerLP>`_. The core idea of this class is that: we will have some learnable ``Processors`` which can learn the parameters of data processing(e.g., parameters for zscore normalization). When new data comes in, these `trained` ``Processors`` can then process the new data and thus processing real-time data in an efficient way becomes possible. More information about ``Processors`` will be listed in the next subsection.
Interface
----------------------
---------
Here are some important interfaces that ``DataHandlerLP`` provides:
@@ -364,7 +364,7 @@ Also, users can pass ``qlib.contrib.data.processor.ConfigSectionProcessor`` that
Processor
----------
---------
The ``Processor`` module in ``Qlib`` is designed to be learnable and it is responsible for handling data processing such as `normalization` and `drop none/nan features/labels`.
@@ -382,14 +382,14 @@ The ``Processor`` module in ``Qlib`` is designed to be learnable and it is respo
- ``CSRankNorm``: `processor` that applies cross sectional rank normalization.
- ``CSZFillna``: `processor` that fills N/A values in a cross sectional way by the mean of the column.
Users can also create their own `processor` by inheriting the base class of ``Processor``. Please refer to the implementation of all the processors for more information (`Processor Link <https://github.com/microsoft/qlib/blob/main/qlib/data/dataset/processor.py>`_).
Users can also create their own `processor` by inheriting the base class of ``Processor``. Please refer to the implementation of all the processors for more information (`Processor Link <https://github.com/microsoft/qlib/blob/main/qlib/data/dataset/processor.py>`_).
To know more about ``Processor``, please refer to `Processor API <../reference/api.html#module-qlib.data.dataset.processor>`_.
Example
--------------
-------
``Data Handler`` can be run with ``qrun`` by modifying the configuration file, and can also be used as a single module.
``Data Handler`` can be run with ``qrun`` by modifying the configuration file, and can also be used as a single module.
Know more about how to run ``Data Handler`` with ``qrun``, please refer to `Workflow: Workflow Management <workflow.html>`_
@@ -427,17 +427,17 @@ Qlib provides implemented data handler `Alpha158`. The following example shows h
.. note:: In the ``Alpha158``, ``Qlib`` uses the label `Ref($close, -2)/Ref($close, -1) - 1` that means the change from T+1 to T+2, rather than `Ref($close, -1)/$close - 1`, of which the reason is that when getting the T day close price of a china stock, the stock can be bought on T+1 day and sold on T+2 day.
API
---------
---
To know more about ``Data Handler``, please refer to `Data Handler API <../reference/api.html#module-qlib.data.dataset.handler>`_.
Dataset
=================
=======
The ``Dataset`` module in ``Qlib`` aims to prepare data for model training and inferencing.
The motivation of this module is that we want to maximize the flexibility of different models to handle data that are suitable for themselves. This module gives the model the flexibility to process their data in an unique way. For instance, models such as ``GBDT`` may work well on data that contains `nan` or `None` value, while neural networks such as ``MLP`` will break down on such data.
The motivation of this module is that we want to maximize the flexibility of different models to handle data that are suitable for themselves. This module gives the model the flexibility to process their data in an unique way. For instance, models such as ``GBDT`` may work well on data that contains `nan` or `None` value, while neural networks such as ``MLP`` will break down on such data.
If user's model need process its data in a different way, user could implement his own ``Dataset`` class. If the model's
data processing is not special, ``DatasetH`` can be used directly.
@@ -448,18 +448,18 @@ The ``DatasetH`` class is the `dataset` with `Data Handler`. Here is the most im
:members:
API
---------
---
To know more about ``Dataset``, please refer to `Dataset API <../reference/api.html#dataset>`_.
Cache
==========
=====
``Cache`` is an optional module that helps accelerate providing data by saving some frequently-used data as cache file. ``Qlib`` provides a `Memcache` class to cache the most-frequently-used data in memory, an inheritable `ExpressionCache` class, and an inheritable `DatasetCache` class.
Global Memory Cache
---------------------
-------------------
`Memcache` is a global memory cache mechanism that composes of three `MemCacheUnit` instances to cache **Calendar**, **Instruments**, and **Features**. The `MemCache` is defined globally in `cache.py` as `H`. Users can use `H['c'], H['i'], H['f']` to get/set `memcache`.
@@ -471,7 +471,7 @@ Global Memory Cache
ExpressionCache
-----------------
---------------
`ExpressionCache` is a cache mechanism that saves expressions such as **Mean($close, 5)**. Users can inherit this base class to define their own cache mechanism that saves expressions according to the following steps.
@@ -486,7 +486,7 @@ The following shows the details about the interfaces:
``Qlib`` has currently provided implemented disk cache `DiskExpressionCache` which inherits from `ExpressionCache` . The expressions data will be stored in the disk.
DatasetCache
-----------------
------------
`DatasetCache` is a cache mechanism that saves datasets. A certain dataset is regulated by a stock pool configuration (or a series of instruments, though not recommended), a list of expressions or static feature fields, the start time, and end time for the collected features and the frequency. Users can inherit this base class to define their own cache mechanism that saves datasets according to the following steps.
@@ -503,7 +503,7 @@ The following shows the details about the interfaces:
Data and Cache File Structure
==================================
=============================
We've specially designed a file structure to manage data and cache, please refer to the `File storage design section in Qlib paper <https://arxiv.org/abs/2009.11189>`_ for detailed information. The file structure of data and cache is listed as follows.
@@ -536,4 +536,3 @@ We've specially designed a file structure to manage data and cache, please refer
- .meta : an assorted meta file recording the stockpool config, field names and visit times
- .index : an assorted index file recording the line index of all calendars
- ...

12
docs/component/highfreq.rst
View File

@@ -1,12 +1,12 @@
.. _highfreq:
============================================
========================================================================
Design of Nested Decision Execution Framework for High-Frequency Trading
============================================
========================================================================
.. currentmodule:: qlib
Introduction
===================
============
Daily trading (e.g. portfolio management) and intraday trading (e.g. orders execution) are two hot topics in Quant investment and usually studied separately.
@@ -15,18 +15,18 @@ In order to support the joint backtest strategies in multiple levels, a correspo
Besides backtesting, the optimization of strategies from different levels is not standalone and can be affected by each other.
For example, the best portfolio management strategy may change with the performance of order executions(e.g. a portfolio with higher turnover may becomes a better choice when we improve the order execution strategies).
To achieve the overall good performance , it is necessary to consider the interaction of strategies in different level.
To achieve the overall good performance , it is necessary to consider the interaction of strategies in different level.
Therefore, building a new framework for trading in multiple levels becomes necessary to solve the various problems mentioned above, for which we designed a nested decision execution framework that consider the interaction of strategies.
.. image:: ../_static/img/framework.svg
The design of the framework is shown in the yellow part in the middle of the figure above. Each level consists of ``Trading Agent`` and ``Execution Env``. ``Trading Agent`` has its own data processing module (``Information Extractor``), forecasting module (``Forecast Model``) and decision generator (``Decision Generator``). The trading algorithm generates the decisions by the ``Decision Generator`` based on the forecast signals output by the ``Forecast Module``, and the decisions generated by the trading algorithm are passed to the ``Execution Env``, which returns the execution results.
The design of the framework is shown in the yellow part in the middle of the figure above. Each level consists of ``Trading Agent`` and ``Execution Env``. ``Trading Agent`` has its own data processing module (``Information Extractor``), forecasting module (``Forecast Model``) and decision generator (``Decision Generator``). The trading algorithm generates the decisions by the ``Decision Generator`` based on the forecast signals output by the ``Forecast Module``, and the decisions generated by the trading algorithm are passed to the ``Execution Env``, which returns the execution results.
The frequency of trading algorithm, decision content and execution environment can be customized by users (e.g. intraday trading, daily-frequency trading, weekly-frequency trading), and the execution environment can be nested with finer-grained trading algorithm and execution environment inside (i.e. sub-workflow in the figure, e.g. daily-frequency orders can be turned into finer-grained decisions by splitting orders within the day). The flexibility of nested decision execution framework makes it easy for users to explore the effects of combining different levels of trading strategies and break down the optimization barriers between different levels of trading algorithm.
Example
===========================
=======
An example of nested decision execution framework for high-frequency can be found `here <https://github.com/microsoft/qlib/blob/main/examples/nested_decision_execution/workflow.py>`_.

24
docs/component/meta.rst
View File

@@ -1,17 +1,17 @@
.. _meta:
=================================
======================================================
Meta Controller: Meta-Task & Meta-Dataset & Meta-Model
=================================
======================================================
.. currentmodule:: qlib
Introduction
=============
============
``Meta Controller`` provides guidance to ``Forecast Model``, which aims to learn regular patterns among a series of forecasting tasks and use learned patterns to guide forthcoming forecasting tasks. Users can implement their own meta-model instance based on ``Meta Controller`` module.
Meta Task
=============
=========
A `Meta Task` instance is the basic element in the meta-learning framework. It saves the data that can be used for the `Meta Model`. Multiple `Meta Task` instances may share the same `Data Handler`, controlled by `Meta Dataset`. Users should use `prepare_task_data()` to obtain the data that can be directly fed into the `Meta Model`.
@@ -19,7 +19,7 @@ A `Meta Task` instance is the basic element in the meta-learning framework. It s
:members:
Meta Dataset
=============
============
`Meta Dataset` controls the meta-information generating process. It is on the duty of providing data for training the `Meta Model`. Users should use `prepare_tasks` to retrieve a list of `Meta Task` instances.
@@ -27,26 +27,26 @@ Meta Dataset
:members:
Meta Model
=============
==========
General Meta Model
------------------
`Meta Model` instance is the part that controls the workflow. The usage of the `Meta Model` includes:
1. Users train their `Meta Model` with the `fit` function.
1. Users train their `Meta Model` with the `fit` function.
2. The `Meta Model` instance guides the workflow by giving useful information via the `inference` function.
.. autoclass:: qlib.model.meta.model.MetaModel
:members:
Meta Task Model
------------------
---------------
This type of meta-model may interact with task definitions directly. Then, the `Meta Task Model` is the class for them to inherit from. They guide the base tasks by modifying the base task definitions. The function `prepare_tasks` can be used to obtain the modified base task definitions.
.. autoclass:: qlib.model.meta.model.MetaTaskModel
:members:
Meta Guide Model
------------------
----------------
This type of meta-model participates in the training process of the base forecasting model. The meta-model may guide the base forecasting models during their training to improve their performances.
.. autoclass:: qlib.model.meta.model.MetaGuideModel
@@ -54,9 +54,9 @@ This type of meta-model participates in the training process of the base forecas
Example
=============
``Qlib`` provides an implementation of ``Meta Model`` module, ``DDG-DA``,
which adapts to the market dynamics.
=======
``Qlib`` provides an implementation of ``Meta Model`` module, ``DDG-DA``,
which adapts to the market dynamics.
``DDG-DA`` includes four steps:

24
docs/component/model.rst
View File

@@ -1,13 +1,13 @@
.. _model:
============================================
===========================================
Forecast Model: Model Training & Prediction
============================================
===========================================
Introduction
===================
============
``Forecast Model`` is designed to make the `prediction score` about stocks. Users can use the ``Forecast Model`` in an automatic workflow by ``qrun``, please refer to `Workflow: Workflow Management <workflow.html>`_.
``Forecast Model`` is designed to make the `prediction score` about stocks. Users can use the ``Forecast Model`` in an automatic workflow by ``qrun``, please refer to `Workflow: Workflow Management <workflow.html>`_.
Because the components in ``Qlib`` are designed in a loosely-coupled way, ``Forecast Model`` can be used as an independent module also.
@@ -22,11 +22,11 @@ The base class provides the following interfaces:
:members:
``Qlib`` also provides a base class `qlib.model.base.ModelFT <../reference/api.html#qlib.model.base.ModelFT>`_, which includes the method for finetuning the model.
For other interfaces such as `finetune`, please refer to `Model API <../reference/api.html#module-qlib.model.base>`_.
Example
==================
=======
``Qlib``'s `Model Zoo` includes models such as ``LightGBM``, ``MLP``, ``LSTM``, etc.. These models are treated as the baselines of ``Forecast Model``. The following steps show how to run`` LightGBM`` as an independent module.
@@ -84,7 +84,7 @@ Example
},
},
}
# model initiaiton
model = init_instance_by_config(task["model"])
dataset = init_instance_by_config(task["dataset"])
@@ -100,22 +100,22 @@ Example
sr = SignalRecord(model, dataset, recorder)
sr.generate()
.. note::
.. note::
`Alpha158` is the data handler provided by ``Qlib``, please refer to `Data Handler <data.html#data-handler>`_.
`SignalRecord` is the `Record Template` in ``Qlib``, please refer to `Workflow <recorder.html#record-template>`_.
Also, the above example has been given in ``examples/train_backtest_analyze.ipynb``.
Technically, the meaning of the model prediction depends on the label setting designed by user.
By default, the meaning of the score is normally the rating of the instruments by the forecasting model. The higher the score, the more profit the instruments.
By default, the meaning of the score is normally the rating of the instruments by the forecasting model. The higher the score, the more profit the instruments.
Custom Model
===================
============
Qlib supports custom models. If users are interested in customizing their own models and integrating the models into ``Qlib``, please refer to `Custom Model Integration <../start/integration.html>`_.
API
===================
===
Please refer to `Model API <../reference/api.html#module-qlib.model.base>`_.

16
docs/component/online.rst
View File

@@ -1,13 +1,13 @@
.. _online:
=================================
==============
Online Serving
=================================
==============
.. currentmodule:: qlib
Introduction
=============
============
.. image:: ../_static/img/online_serving.png
:align: center
@@ -15,7 +15,7 @@ Introduction
In addition to backtesting, one way to test a model is effective is to make predictions in real market conditions or even do real trading based on those predictions.
``Online Serving`` is a set of modules for online models using the latest data,
which including `Online Manager <#Online Manager>`_, `Online Strategy <#Online Strategy>`_, `Online Tool <#Online Tool>`_, `Updater <#Updater>`_.
which including `Online Manager <#Online Manager>`_, `Online Strategy <#Online Strategy>`_, `Online Tool <#Online Tool>`_, `Updater <#Updater>`_.
`Here <https://github.com/microsoft/qlib/tree/main/examples/online_srv>`_ are several examples for reference, which demonstrate different features of ``Online Serving``.
If you have many models or `task` needs to be managed, please consider `Task Management <../advanced/task_management.html>`_.
@@ -28,25 +28,25 @@ Known limitations currently
Online Manager
=============
==============
.. automodule:: qlib.workflow.online.manager
:members:
Online Strategy
=============
===============
.. automodule:: qlib.workflow.online.strategy
:members:
Online Tool
=============
===========
.. automodule:: qlib.workflow.online.utils
:members:
Updater
=============
=======
.. automodule:: qlib.workflow.online.update
:members:

20
docs/component/recorder.rst
View File

@@ -6,8 +6,8 @@ Qlib Recorder: Experiment Management
.. currentmodule:: qlib
Introduction
===================
``Qlib`` contains an experiment management system named ``QlibRecorder``, which is designed to help users handle experiment and analyse results in an efficient way.
============
``Qlib`` contains an experiment management system named ``QlibRecorder``, which is designed to help users handle experiment and analyse results in an efficient way.
There are three components of the system:
@@ -34,13 +34,13 @@ Here is a general view of the structure of the system:
- Recorder 2
- ...
- ...
This experiment management system defines a set of interface and provided a concrete implementation ``MLflowExpManager``, which is based on the machine learning platform: ``MLFlow`` (`link <https://mlflow.org/>`_).
This experiment management system defines a set of interface and provided a concrete implementation ``MLflowExpManager``, which is based on the machine learning platform: ``MLFlow`` (`link <https://mlflow.org/>`_).
If users set the implementation of ``ExpManager`` to be ``MLflowExpManager``, they can use the command `mlflow ui` to visualize and check the experiment results. For more information, please refer to the related documents `here <https://www.mlflow.org/docs/latest/cli.html#mlflow-ui>`_.
Qlib Recorder
===================
=============
``QlibRecorder`` provides a high level API for users to use the experiment management system. The interfaces are wrapped in the variable ``R`` in ``Qlib``, and users can directly use ``R`` to interact with the system. The following command shows how to import ``R`` in Python:
.. code-block:: Python
@@ -55,7 +55,7 @@ Here are the available interfaces of ``QlibRecorder``:
:members:
Experiment Manager
===================
==================
The ``ExpManager`` module in ``Qlib`` is responsible for managing different experiments. Most of the APIs of ``ExpManager`` are similar to ``QlibRecorder``, and the most important API will be the ``get_exp`` method. User can directly refer to the documents above for some detailed information about how to use the ``get_exp`` method.
@@ -65,7 +65,7 @@ The ``ExpManager`` module in ``Qlib`` is responsible for managing different expe
For other interfaces such as `create_exp`, `delete_exp`, please refer to `Experiment Manager API <../reference/api.html#experiment-manager>`_.
Experiment
===================
==========
The ``Experiment`` class is solely responsible for a single experiment, and it will handle any operations that are related to an experiment. Basic methods such as `start`, `end` an experiment are included. Besides, methods related to `recorders` are also available: such methods include `get_recorder` and `list_recorders`.
@@ -77,7 +77,7 @@ For other interfaces such as `search_records`, `delete_recorder`, please refer t
``Qlib`` also provides a default ``Experiment``, which will be created and used under certain situations when users use the APIs such as `log_metrics` or `get_exp`. If the default ``Experiment`` is used, there will be related logged information when running ``Qlib``. Users are able to change the name of the default ``Experiment`` in the config file of ``Qlib`` or during ``Qlib``'s `initialization <../start/initialization.html#parameters>`_, which is set to be '`Experiment`'.
Recorder
===================
========
The ``Recorder`` class is responsible for a single recorder. It will handle some detailed operations such as ``log_metrics``, ``log_params`` of a single run. It is designed to help user to easily track results and things being generated during a run.
@@ -89,7 +89,7 @@ Here are some important APIs that are not included in the ``QlibRecorder``:
For other interfaces such as `save_objects`, `load_object`, please refer to `Recorder API <../reference/api.html#recorder>`_.
Record Template
===================
===============
The ``RecordTemp`` class is a class that enables generate experiment results such as IC and backtest in a certain format. We have provided three different `Record Template` class:
@@ -131,7 +131,7 @@ Here is a simple exampke of what is done in ``PortAnaRecord``, which users can r
"close_cost": 0.0015,
"min_cost": 5,
}
strategy = TopkDropoutStrategy(**STRATEGY_CONFIG)
report_normal, positions_normal = normal_backtest(pred_score, strategy=strategy, **BACKTEST_CONFIG)

92
docs/component/report.rst
View File

@@ -1,11 +1,11 @@
.. _report:
==========================================
=======================================
Analysis: Evaluation & Results Analysis
==========================================
=======================================
Introduction
===================
============
``Analysis`` is designed to show the graphical reports of ``Intraday Trading`` , which helps users to evaluate and analyse investment portfolios visually. The following are some graphics to view:
@@ -24,7 +24,7 @@ All of the accumulated profit metrics(e.g. return, max drawdown) in Qlib are cal
This avoids the metrics or the plots being skewed exponentially over time.
Graphical Reports
===================
=================
Users can run the following code to get all supported reports.
@@ -41,13 +41,13 @@ Users can run the following code to get all supported reports.
Usage & Example
===================
===============
Usage of `analysis_position.report`
-----------------------------------
API
~~~~~~~~~~~~~~~~
~~~
.. automodule:: qlib.contrib.report.analysis_position.report
:members:
@@ -58,7 +58,7 @@ Graphical Result
.. note::
- Axis X: Trading day
- Axis Y:
- Axis Y:
- `cum bench`
Cumulative returns series of benchmark
- `cum return wo cost`
@@ -82,34 +82,34 @@ Graphical Result
- The shaded part above: Maximum drawdown corresponding to `cum return wo cost`
- The shaded part below: Maximum drawdown corresponding to `cum ex return wo cost`
.. image:: ../_static/img/analysis/report.png
.. image:: ../_static/img/analysis/report.png
Usage of `analysis_position.score_ic`
-------------------------------------
API
~~~~~~~~~~~~~~~~
~~~
.. automodule:: qlib.contrib.report.analysis_position.score_ic
:members:
Graphical Result
~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
.. note::
.. note::
- Axis X: Trading day
- Axis Y:
- Axis Y:
- `ic`
The `Pearson correlation coefficient` series between `label` and `prediction score`.
In the above example, the `label` is formulated as `Ref($close, -2)/Ref($close, -1)-1`. Please refer to `Data Feature <data.html#feature>`_ for more details.
- `rank_ic`
The `Spearman's rank correlation coefficient` series between `label` and `prediction score`.
.. image:: ../_static/img/analysis/score_ic.png
.. image:: ../_static/img/analysis/score_ic.png
.. Usage of `analysis_position.cumulative_return`
@@ -124,7 +124,7 @@ Graphical Result
.. Graphical Result
.. ~~~~~~~~~~~~~~~~~
..
.. .. note::
.. .. note::
..
.. - Axis X: Trading day
.. - Axis Y:
@@ -134,27 +134,27 @@ Graphical Result
.. - In the **buy_minus_sell** graph, the **y** value of the **weight** graph at the bottom is `buy_weight + sell_weight`.
.. - In each graph, the **red line** in the histogram on the right represents the average.
..
.. .. image:: ../_static/img/analysis/cumulative_return_buy.png
.. .. image:: ../_static/img/analysis/cumulative_return_buy.png
..
.. .. image:: ../_static/img/analysis/cumulative_return_sell.png
.. .. image:: ../_static/img/analysis/cumulative_return_sell.png
..
.. .. image:: ../_static/img/analysis/cumulative_return_buy_minus_sell.png
.. .. image:: ../_static/img/analysis/cumulative_return_buy_minus_sell.png
..
.. .. image:: ../_static/img/analysis/cumulative_return_hold.png
.. .. image:: ../_static/img/analysis/cumulative_return_hold.png
Usage of `analysis_position.risk_analysis`
----------------------------------------------
------------------------------------------
API
~~~~~~~~~~~~~~~~
~~~
.. automodule:: qlib.contrib.report.analysis_position.risk_analysis
:members:
Graphical Result
~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
.. note::
@@ -210,7 +210,7 @@ Graphical Result
The `Standard Deviation` series of monthly `CAR` (cumulative abnormal return) without cost.
- `excess_return_with_cost_max_drawdown`
The `Standard Deviation` series of monthly `CAR` (cumulative abnormal return) with cost.
.. image:: ../_static/img/analysis/risk_analysis_annualized_return.png
:align: center
@@ -221,58 +221,58 @@ Graphical Result
.. image:: ../_static/img/analysis/risk_analysis_information_ratio.png
:align: center
.. image:: ../_static/img/analysis/risk_analysis_std.png
.. image:: ../_static/img/analysis/risk_analysis_std.png
:align: center
..
.. Usage of `analysis_position.rank_label`
.. ----------------------------------------------
.. ---------------------------------------
..
.. API
.. ~~~~~
.. ~~~
..
.. .. automodule:: qlib.contrib.report.analysis_position.rank_label
.. :members:
..
..
.. Graphical Result
.. ~~~~~~~~~~~~~~~~~
.. ~~~~~~~~~~~~~~~~
..
.. .. note::
.. .. note::
..
.. - hold/sell/buy graphics:
.. - Axis X: Trading day
.. - Axis Y:
.. - Axis Y:
.. Average `ranking ratio`of `label` for stocks that is held/sold/bought on the trading day.
..
.. In the above example, the `label` is formulated as `Ref($close, -1)/$close - 1`. The `ranking ratio` can be formulated as follows.
.. .. math::
..
..
.. ranking\ ratio = \frac{Ascending\ Ranking\ of\ label}{Number\ of\ Stocks\ in\ the\ Portfolio}
..
.. .. image:: ../_static/img/analysis/rank_label_hold.png
.. .. image:: ../_static/img/analysis/rank_label_hold.png
.. :align: center
..
.. .. image:: ../_static/img/analysis/rank_label_buy.png
.. .. image:: ../_static/img/analysis/rank_label_buy.png
.. :align: center
..
.. .. image:: ../_static/img/analysis/rank_label_sell.png
.. .. image:: ../_static/img/analysis/rank_label_sell.png
.. :align: center
..
..
Usage of `analysis_model.analysis_model_performance`
-----------------------------------------------------
----------------------------------------------------
API
~~~~~
~~~
.. automodule:: qlib.contrib.report.analysis_model.analysis_model_performance
:members:
Graphical Results
~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~
.. note::
@@ -291,13 +291,13 @@ Graphical Results
The Difference series between `Cumulative Return` of `Group1` and of `Group5`
- `long-average`
The Difference series between `Cumulative Return` of `Group1` and average `Cumulative Return` for all stocks.
The `ranking ratio` can be formulated as follows.
.. math::
ranking\ ratio = \frac{Ascending\ Ranking\ of\ label}{Number\ of\ Stocks\ in\ the\ Portfolio}
.. image:: ../_static/img/analysis/analysis_model_cumulative_return.png
.. image:: ../_static/img/analysis/analysis_model_cumulative_return.png
:align: center
.. note::
@@ -305,7 +305,7 @@ Graphical Results
The distribution of long-short/long-average returns on each trading day
.. image:: ../_static/img/analysis/analysis_model_long_short.png
.. image:: ../_static/img/analysis/analysis_model_long_short.png
:align: center
.. TODO: ask xiao yang for detial
@@ -315,14 +315,14 @@ Graphical Results
- The `Pearson correlation coefficient` series between `labels` and `prediction scores` of stocks in portfolio.
- The graphics reports can be used to evaluate the `prediction scores`.
.. image:: ../_static/img/analysis/analysis_model_IC.png
.. image:: ../_static/img/analysis/analysis_model_IC.png
:align: center
.. note::
- Monthly IC
Monthly average of the `Information Coefficient`
.. image:: ../_static/img/analysis/analysis_model_monthly_IC.png
.. image:: ../_static/img/analysis/analysis_model_monthly_IC.png
:align: center
.. note::
@@ -331,14 +331,14 @@ Graphical Results
- IC Normal Dist. Q-Q
The `Quantile-Quantile Plot` is used for the normal distribution of `Information Coefficient` on each trading day.
.. image:: ../_static/img/analysis/analysis_model_NDQ.png
.. image:: ../_static/img/analysis/analysis_model_NDQ.png
:align: center
.. note::
- Auto Correlation
- The `Pearson correlation coefficient` series between the latest `prediction scores` and the `prediction scores` `lag` days ago of stocks in portfolio on each trading day.
- The `Pearson correlation coefficient` series between the latest `prediction scores` and the `prediction scores` `lag` days ago of stocks in portfolio on each trading day.
- The graphics reports can be used to estimate the turnover rate.
.. image:: ../_static/img/analysis/analysis_model_auto_correlation.png
.. image:: ../_static/img/analysis/analysis_model_auto_correlation.png
:align: center

28
docs/component/strategy.rst
View File

@@ -6,7 +6,7 @@ Portfolio Strategy: Portfolio Management
.. currentmodule:: qlib
Introduction
===================
============
``Portfolio Strategy`` is designed to adopt different portfolio strategies, which means that users can adopt different algorithms to generate investment portfolios based on the prediction scores of the ``Forecast Model``. Users can use the ``Portfolio Strategy`` in an automatic workflow by ``Workflow`` module, please refer to `Workflow: Workflow Management <workflow.html>`_.
@@ -20,7 +20,7 @@ Base Class & Interface
======================
BaseStrategy
------------------
------------
Qlib provides a base class ``qlib.strategy.base.BaseStrategy``. All strategy classes need to inherit the base class and implement its interface.
@@ -32,7 +32,7 @@ Qlib provides a base class ``qlib.strategy.base.BaseStrategy``. All strategy cla
Users can inherit `BaseStrategy` to customize their strategy class.
WeightStrategyBase
--------------------
------------------
Qlib also provides a class ``qlib.contrib.strategy.WeightStrategyBase`` that is a subclass of `BaseStrategy`.
@@ -60,7 +60,7 @@ Implemented Strategy
Qlib provides a implemented strategy classes named `TopkDropoutStrategy`.
TopkDropoutStrategy
------------------
-------------------
`TopkDropoutStrategy` is a subclass of `BaseStrategy` and implement the interface `generate_order_list` whose process is as follows.
- Adopt the ``Topk-Drop`` algorithm to calculate the target amount of each stock
@@ -74,16 +74,16 @@ TopkDropoutStrategy
In general, the number of stocks currently held is `Topk`, with the exception of being zero at the beginning period of trading.
For each trading day, let $d$ be the number of the instruments currently held and with a rank $\gt K$ when ranked by the prediction scores from high to low.
Then `d` number of stocks currently held with the worst `prediction score` will be sold, and the same number of unheld stocks with the best `prediction score` will be bought.
In general, $d=$`Drop`, especially when the pool of the candidate instruments is large, $K$ is large, and `Drop` is small.
In most cases, ``TopkDrop`` algorithm sells and buys `Drop` stocks every trading day, which yields a turnover rate of 2$\times$`Drop`/$K$.
The following images illustrate a typical scenario.
.. image:: ../_static/img/topk_drop.png
:alt: Topk-Drop
- Generate the order list from the target amount
@@ -98,12 +98,12 @@ and `qlib.contrib.strategy.optimizer.enhanced_indexing.EnhancedIndexingOptimizer
Usage & Example
====================
===============
First, user can create a model to get trading signals(the variable name is ``pred_score`` in following cases).
Prediction Score
-----------------
----------------
The `prediction score` is a pandas DataFrame. Its index is <datetime(pd.Timestamp), instrument(str)> and it must
contains a `score` column.
@@ -134,7 +134,7 @@ Qlib didn't add a step to scale the prediction score to a unified scale due to t
- The model has the flexibility to define the target, loss, and data processing. So we don't think there is a silver bullet to rescale it back directly barely based on the model's outputs. If you want to scale it back to some meaningful values(e.g. stock returns.), an intuitive solution is to create a regression model for the model's recent outputs and your recent target values.
Running backtest
-----------------
----------------
- In most cases, users could backtest their portfolio management strategy with ``backtest_daily``.
@@ -195,7 +195,7 @@ Running backtest
CSI300_BENCH = "SH000300"
# Benchmark is for calculating the excess return of your strategy.
# Its data format will be like **ONE normal instrument**.
# Its data format will be like **ONE normal instrument**.
# For example, you can query its data with the code below
# `D.features(["SH000300"], ["$close"], start_time='2010-01-01', end_time='2017-12-31', freq='day')`
# It is different from the argument `market`, which indicates a universe of stocks (e.g. **A SET** of stocks like csi300)
@@ -262,7 +262,7 @@ Running backtest
Result
------------------
------
The backtest results are in the following form:
@@ -307,5 +307,5 @@ The backtest results are in the following form:
Reference
===================
=========
To know more about the `prediction score` `pred_score` output by ``Forecast Model``, please refer to `Forecast Model: Model Training & Prediction <model.html>`_.

52
docs/component/workflow.rst
View File

@@ -1,12 +1,12 @@
.. _workflow:
=================================
=============================
Workflow: Workflow Management
=================================
=============================
.. currentmodule:: qlib
Introduction
===================
============
The components in `Qlib Framework <../introduction/introduction.html#framework>`_ are designed in a loosely-coupled way. Users could build their own Quant research workflow with these components like `Example <https://github.com/microsoft/qlib/blob/main/examples/workflow_by_code.py>`_.
@@ -28,7 +28,7 @@ With ``qrun``, user can easily start an `execution`, which includes the followin
For each `execution`, ``Qlib`` has a complete system to tracking all the information as well as artifacts generated during training, inference and evaluation phase. For more information about how ``Qlib`` handles this, please refer to the related document: `Recorder: Experiment Management <../component/recorder.html>`_.
Complete Example
===================
================
Before getting into details, here is a complete example of ``qrun``, which defines the workflow in typical Quant research.
Below is a typical config file of ``qrun``.
@@ -54,7 +54,7 @@ Below is a typical config file of ``qrun``.
topk: 50
n_drop: 5
signal:
- <MODEL>
- <MODEL>
- <DATASET>
backtest:
limit_threshold: 0.095
@@ -90,13 +90,13 @@ Below is a typical config file of ``qrun``.
train: [2008-01-01, 2014-12-31]
valid: [2015-01-01, 2016-12-31]
test: [2017-01-01, 2020-08-01]
record:
record:
- class: SignalRecord
module_path: qlib.workflow.record_temp
kwargs: {}
- class: PortAnaRecord
module_path: qlib.workflow.record_temp
kwargs:
kwargs:
config: *port_analysis_config
After saving the config into `configuration.yaml`, users could start the workflow and test their ideas with a single command below.
@@ -111,22 +111,22 @@ If users want to use ``qrun`` under debug mode, please use the following command
python -m pdb qlib/workflow/cli.py examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml
.. note::
.. note::
`qrun` will be placed in your $PATH directory when installing ``Qlib``.
.. note::
.. note::
The symbol `&` in `yaml` file stands for an anchor of a field, which is useful when another fields include this parameter as part of the value. Taking the configuration file above as an example, users can directly change the value of `market` and `benchmark` without traversing the entire configuration file.
Configuration File
===================
==================
Let's get into details of ``qrun`` in this section.
Before using ``qrun``, users need to prepare a configuration file. The following content shows how to prepare each part of the configuration file.
The design logic of the configuration file is very simple. It predefines fixed workflows and provide this yaml interface to users to define how to initialize each component.
The design logic of the configuration file is very simple. It predefines fixed workflows and provide this yaml interface to users to define how to initialize each component.
It follow the design of `init_instance_by_config <https://github.com/microsoft/qlib/blob/2aee9e0145decc3e71def70909639b5e5a6f4b58/qlib/utils/__init__.py#L264>`_ . It defines the initialization of each component of Qlib, which typically include the class and the initialization arguments.
For example, the following yaml and code are equivalent.
@@ -166,7 +166,7 @@ For example, the following yaml and code are equivalent.
Qlib Init Section
--------------------
-----------------
At first, the configuration file needs to contain several basic parameters which will be used for qlib initialization.
@@ -181,21 +181,21 @@ The meaning of each field is as follows:
Type: str. The URI of the Qlib data. For example, it could be the location where the data loaded by ``get_data.py`` are stored.
- `region`
- If `region` == "us", ``Qlib`` will be initialized in US-stock mode.
- If `region` == "us", ``Qlib`` will be initialized in US-stock mode.
- If `region` == "cn", ``Qlib`` will be initialized in China-stock mode.
.. note::
.. note::
The value of `region` should be aligned with the data stored in `provider_uri`.
Task Section
--------------------
------------
The `task` field in the configuration corresponds to a `task`, which contains the parameters of three different subsections: `Model`, `Dataset` and `Record`.
Model Section
~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~
In the `task` field, the `model` section describes the parameters of the model to be used for training and inference. For more information about the base ``Model`` class, please refer to `Qlib Model <../component/model.html>`_.
@@ -224,14 +224,14 @@ The meaning of each field is as follows:
Type: str. The path for the model in qlib.
- `kwargs`
The keywords arguments for the model. Please refer to the specific model implementation for more information: `models <https://github.com/microsoft/qlib/blob/main/qlib/contrib/model>`_.
The keywords arguments for the model. Please refer to the specific model implementation for more information: `models <https://github.com/microsoft/qlib/blob/main/qlib/contrib/model>`_.
.. note::
.. note::
``Qlib`` provides a util named: ``init_instance_by_config`` to initialize any class inside ``Qlib`` with the configuration includes the fields: `class`, `module_path` and `kwargs`.
Dataset Section
~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~
The `dataset` field describes the parameters for the ``Dataset`` module in ``Qlib`` as well those for the module ``DataHandler``. For more information about the ``Dataset`` module, please refer to `Qlib Data <../component/data.html#dataset>`_.
@@ -266,7 +266,7 @@ Here is the configuration for the ``Dataset`` module which will take care of dat
test: [2017-01-01, 2020-08-01]
Record Section
~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~
The `record` field is about the parameters the ``Record`` module in ``Qlib``. ``Record`` is responsible for tracking training process and results such as `information Coefficient (IC)` and `backtest` in a standard format.
@@ -282,7 +282,7 @@ The following script is the configuration of `backtest` and the `strategy` used
topk: 50
n_drop: 5
signal:
- <MODEL>
- <MODEL>
- <DATASET>
backtest:
limit_threshold: 0.095
@@ -299,13 +299,13 @@ Here is the configuration details of different `Record Template` such as ``Signa
.. code-block:: YAML
record:
record:
- class: SignalRecord
module_path: qlib.workflow.record_temp
kwargs: {}
- class: PortAnaRecord
module_path: qlib.workflow.record_temp
kwargs:
kwargs:
config: *port_analysis_config
For more information about the ``Record`` module in ``Qlib``, user can refer to the related document: `Record <../component/recorder.html#record-template>`_.

12
docs/developer/code_standard_and_dev_guide.rst
View File

@@ -1,16 +1,16 @@
.. _code_standard:
=================================
=============
Code Standard
=================================
=============
Docstring
=================================
=========
Please use the `Numpydoc Style <https://stackoverflow.com/a/24385103>`_.
Continuous Integration
=================================
Continuous Integration (CI) tools help you stick to the quality standards by running tests every time you push a new commit and reporting the results to a pull request.
======================
Continuous Integration (CI) tools help you stick to the quality standards by running tests every time you push a new commit and reporting the results to a pull request.
When you submit a PR request, you can check whether your code passes the CI tests in the "check" section at the bottom of the web page.
@@ -23,7 +23,7 @@ When you submit a PR request, you can check whether your code passes the CI test
python -m black . -l 120
2. Qlib will check your code style pylint. The checking command is implemented in [github action workflow](https://github.com/microsoft/qlib/blob/0e8b94a552f1c457cfa6cd2c1bb3b87ebb3fb279/.github/workflows/test.yml#L66).
2. Qlib will check your code style pylint. The checking command is implemented in [github action workflow](https://github.com/microsoft/qlib/blob/0e8b94a552f1c457cfa6cd2c1bb3b87ebb3fb279/.github/workflows/test.yml#L66).
Sometime pylint's restrictions are not that reasonable. You can ignore specific errors like this
.. code-block:: python

10
docs/hidden/client.rst
View File

@@ -1,12 +1,12 @@
.. _client:
Qlib Client-Server Framework
===================
============================
.. currentmodule:: qlib
Introduction
-----------
------------
Client-Server is designed to solve following problems
- Manage the data in a centralized way. Users don't have to manage data of different versions.
@@ -159,13 +159,11 @@ Limitations
2. The rolling operation expression with parameter `0` can not be updated rightly under mechanism of the client-server framework.
API
********************
***
The client is based on `python-socketio<https://python-socketio.readthedocs.io>`_ which is a framework that supports WebSocket client for Python language. The client can only propose requests and receive results, which do not include any calculating procedure.
Class
--------------------
-----
.. automodule:: qlib.data.client

20
docs/hidden/online.rst
View File

@@ -1,11 +1,11 @@
.. _online:
Online
===================
======
.. currentmodule:: qlib
Introduction
-------------------
------------
Welcome to use Online, this module simulates what will be like if we do the real trading use our model and strategy.
@@ -31,11 +31,11 @@ The file structure can be viewed at fileStruct_.
Example
-------------------
-------
Let's take an example,
.. note:: Make sure you have the latest version of `qlib` installed.
.. note:: Make sure you have the latest version of `qlib` installed.
If you want to use the models and data provided by `qlib`, you only need to do as follows.
@@ -93,7 +93,7 @@ If Your account was saved in "./user_data/", you can see the performance of your
Here 'SH000905' represents csi500 and 'SH000300' represents csi300
Manage your account
--------------------
-------------------
Any account processed by `online` should be saved in a folder. you can use commands
defined to manage your accounts.
@@ -161,7 +161,7 @@ be called at each trading date.
>> online update -date 2019-10-16 -path ./user_data/
API
------------------
---
All those operations are based on defined in `qlib.contrib.online.operator`
@@ -170,7 +170,7 @@ All those operations are based on defined in `qlib.contrib.online.operator`
.. _fileStruct:
File structure
------------------
--------------
'user_data' indicates the root of folder.
Name that bold indicates it’s a folder, otherwise it’s a document.
@@ -214,7 +214,7 @@ Configuration file
The configure file used in `online` should contain the model and strategy information.
About the model
~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~
First, your configuration file needs to have a field about the model,
this field and its contents determine the model we used when generating score at predict date.
@@ -243,7 +243,7 @@ contains 2 methods used in `online` module.
About the strategy
~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~
Your need define the strategy used to generate the order list at predict date.
@@ -259,7 +259,7 @@ Followings are two examples for a TopkAmountStrategy
n_drop: 10
Generated files
------------------
---------------
The 'online_generate' command will create the order list at {folder_path}/{user_id}/temp/,
the name of that is orderlist_{YYYY-MM-DD}.json, YYYY-MM-DD is the date that those orders to be executed.

41
docs/hidden/tuner.rst
View File

@@ -1,11 +1,11 @@
.. _tuner:
Tuner
===================
=====
.. currentmodule:: qlib
Introduction
-------------------
------------
Welcome to use Tuner, this document is based on that you can use Estimator proficiently and correctly.
@@ -41,19 +41,19 @@ We write a simple configuration example as following,
tuner_class: QLibTuner
qlib_client:
auto_mount: False
logging_level: INFO
logging_level: INFO
optimization_criteria:
report_type: model
report_factor: model_score
optim_type: max
tuner_pipeline:
-
model:
-
model:
class: SomeModel
space: SomeModelSpace
trainer:
trainer:
class: RollingTrainer
strategy:
strategy:
class: TopkAmountStrategy
space: TopkAmountStrategySpace
max_evals: 2
@@ -166,13 +166,13 @@ Also, there are some optional fields. The meaning of each field is as follows:
The class of tuner, str type, must be an already implemented model, such as `QLibTuner` in `qlib`, or a custom tuner, but it must be a subclass of `qlib.contrib.tuner.Tuner`, the default value is `QLibTuner`.
- `tuner_module_path`
The module path, str type, absolute url is also supported, indicates the path of the implementation of tuner. The default value is `qlib.contrib.tuner.tuner`
The module path, str type, absolute url is also supported, indicates the path of the implementation of tuner. The default value is `qlib.contrib.tuner.tuner`
About the optimization criteria
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You need to designate a factor to optimize, for tuner need a factor to decide which case is better than other cases.
Usually, we use the result of `estimator`, such as backtest results and the score of model.
Usually, we use the result of `estimator`, such as backtest results and the score of model.
This part needs contain these fields:
@@ -203,13 +203,13 @@ The tuner pipeline contains different tuners, and the `tuner` program will proce
.. code-block:: YAML
tuner_pipeline:
-
model:
-
model:
class: SomeModel
space: SomeModelSpace
trainer:
trainer:
class: RollingTrainer
strategy:
strategy:
class: TopkAmountStrategy
space: TopkAmountStrategySpace
max_evals: 2
@@ -249,25 +249,25 @@ You need to use the same dataset to evaluate your different `estimator` experime
test_start_date: 2016-07-01
test_end_date: 2018-04-30
- `rolling_period`
- `rolling_period`
The rolling period, integer type, indicates how many time steps need rolling when rolling the data. The default value is `60`. If you use `RollingTrainer`, this config will be used, or it will be ignored.
- `train_start_date`
Training start time, str type.
- `train_end_date`
- `train_end_date`
Training end time, str type.
- `validate_start_date`
- `validate_start_date`
Validation start time, str type.
- `validate_end_date`
- `validate_end_date`
Validation end time, str type.
- `test_start_date`
- `test_start_date`
Test start time, str type.
- `test_end_date`
- `test_end_date`
Test end time, str type. If `test_end_date` is `-1` or greater than the last date of the data, the last date of the data will be used as `test_end_date`.
About the data and backtest
@@ -315,11 +315,10 @@ About the data and backtest
Experiment Result
-----------------
All the results are stored in experiment file directly, you can check them directly in the corresponding files.
All the results are stored in experiment file directly, you can check them directly in the corresponding files.
What we save are as following:
- Global optimal parameters
- Local optimal parameters of each tuner
- Config file of this `tuner` experiment
- Every `estimator` experiments result in the process

10
docs/index.rst
View File

@@ -1,6 +1,6 @@
============================================================
======================
``Qlib`` Documentation
============================================================
======================
``Qlib`` is an AI-oriented quantitative investment platform, which aims to realize the potential, empower the research, and create the value of AI technologies in quantitative investment.
@@ -24,12 +24,12 @@ Document Structure
.. toctree::
:maxdepth: 3
:caption: FIRST STEPS:
Installation <start/installation.rst>
Initialization <start/initialization.rst>
Data Retrieval <start/getdata.rst>
Custom Model Integration <start/integration.rst>
.. toctree::
:maxdepth: 3
@@ -48,7 +48,7 @@ Document Structure
.. toctree::
:maxdepth: 3
:caption: ADVANCED TOPICS:
Building Formulaic Alphas <advanced/alpha.rst>
Online & Offline mode <advanced/server.rst>
Serialization <advanced/serial.rst>

12
docs/introduction/introduction.rst
View File

@@ -3,7 +3,7 @@
===============================
Introduction
===================
============
.. image:: ../_static/img/logo/white_bg_rec+word.png
:align: center
@@ -13,8 +13,8 @@ Introduction
With ``Qlib``, users can easily try their ideas to create better Quant investment strategies.
Framework
===================
=========
.. image:: ../_static/img/framework.svg
:align: center
@@ -27,7 +27,7 @@ At the module level, Qlib is a platform that consists of above components. The c
Name Description
======================== ==============================================================================
`Infrastructure` layer `Infrastructure` layer provides underlying support for Quant research.
`DataServer` provides high-performance infrastructure for users to manage
`DataServer` provides high-performance infrastructure for users to manage
and retrieve raw data. `Trainer` provides flexible interface to control
the training process of models which enable algorithms controlling the
training process.
@@ -35,13 +35,13 @@ Name Description
`Workflow` layer `Workflow` layer covers the whole workflow of quantitative investment.
`Information Extractor` extracts data for models. `Forecast Model` focuses
on producing all kinds of forecast signals (e.g. *alpha*, risk) for other
modules. With these signals `Decision Generator` will generate the target
modules. With these signals `Decision Generator` will generate the target
trading decisions(i.e. portfolio, orders) to be executed by `Execution Env`
(i.e. the trading market). There may be multiple levels of `Trading Agent`
and `Execution Env` (e.g. an *order executor trading agent and intraday
order execution environment* could behave like an interday trading
environment and nested in *daily portfolio management trading agent and
interday trading environment* )
interday trading environment* )
`Interface` layer `Interface` layer tries to present a user-friendly interface for the underlying
system. `Analyser` module will provide users detailed analysis reports of

24
docs/introduction/quick.rst
View File

@@ -1,10 +1,10 @@
===============================
===========
Quick Start
===============================
===========
Introduction
==============
============
This ``Quick Start`` guide tries to demonstrate
@@ -14,7 +14,7 @@ This ``Quick Start`` guide tries to demonstrate
Installation
==================
============
Users can easily intsall ``Qlib`` according to the following steps:
@@ -34,7 +34,7 @@ Users can easily intsall ``Qlib`` according to the following steps:
To known more about `installation`, please refer to `Qlib Installation <../start/installation.html>`_.
Prepare Data
==============
============
Load and prepare data by running the following code:
@@ -47,14 +47,14 @@ This dataset is created by public data collected by crawler scripts in ``scripts
To known more about `prepare data`, please refer to `Data Preparation <../component/data.html#data-preparation>`_.
Auto Quant Research Workflow
====================================
============================
``Qlib`` provides a tool named ``qrun`` to run the whole workflow automatically (including building dataset, training models, backtest and evaluation). Users can start an auto quant research workflow and have a graphical reports analysis according to the following steps:
``Qlib`` provides a tool named ``qrun`` to run the whole workflow automatically (including building dataset, training models, backtest and evaluation). Users can start an auto quant research workflow and have a graphical reports analysis according to the following steps:
- Quant Research Workflow:
- Quant Research Workflow:
- Run ``qrun`` with a config file of the LightGBM model `workflow_config_lightgbm.yaml` as following.
.. code-block::
.. code-block::
cd examples # Avoid running program under the directory contains `qlib`
qrun benchmarks/LightGBM/workflow_config_lightgbm.yaml
@@ -64,7 +64,7 @@ Auto Quant Research Workflow
The result of ``qrun`` is as follows, which is also the typical result of ``Forecast model(alpha)``. Please refer to `Intraday Trading <../component/backtest.html>`_. for more details about the result.
.. code-block:: python
risk
excess_return_without_cost mean 0.000605
std 0.005481
@@ -77,7 +77,7 @@ Auto Quant Research Workflow
information_ratio 1.187411
max_drawdown -0.075024
To know more about `workflow` and `qrun`, please refer to `Workflow: Workflow Management <../component/workflow.html>`_.
- Graphical Reports Analysis:
@@ -89,6 +89,6 @@ Auto Quant Research Workflow
Custom Model Integration
===============================================
========================
``Qlib`` provides a batch of models (such as ``lightGBM`` and ``MLP`` models) as examples of ``Forecast Model``. In addition to the default model, users can integrate their own custom models into ``Qlib``. If users are interested in the custom model, please refer to `Custom Model Integration <../start/integration.html>`_.

87
docs/reference/api.rst
View File

@@ -1,7 +1,7 @@
.. _api:
================================
=============
API Reference
================================
=============
@@ -9,32 +9,32 @@ Here you can find all ``Qlib`` interfaces.
Data
====================
====
Provider
--------------------
--------
.. automodule:: qlib.data.data
:members:
Filter
--------------------
------
.. automodule:: qlib.data.filter
:members:
Class
--------------------
-----
.. automodule:: qlib.data.base
:members:
Operator
--------------------
--------
.. automodule:: qlib.data.ops
:members:
Cache
----------------
-----
.. autoclass:: qlib.data.cache.MemCacheUnit
:members:
@@ -55,7 +55,7 @@ Cache
Storage
-------------
-------
.. autoclass:: qlib.data.storage.storage.BaseStorage
:members:
@@ -82,52 +82,52 @@ Storage
Dataset
---------------
-------
Dataset Class
~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~
.. automodule:: qlib.data.dataset.__init__
:members:
Data Loader
~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~
.. automodule:: qlib.data.dataset.loader
:members:
Data Handler
~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~
.. automodule:: qlib.data.dataset.handler
:members:
Processor
~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~
.. automodule:: qlib.data.dataset.processor
:members:
Contrib
====================
=======
Model
--------------------
-----
.. automodule:: qlib.model.base
:members:
Strategy
-------------------
--------
.. automodule:: qlib.contrib.strategy.strategy
:members:
Evaluate
-----------------
--------
.. automodule:: qlib.contrib.evaluate
:members:
Report
-----------------
------
.. automodule:: qlib.contrib.report.analysis_position.report
:members:
@@ -159,103 +159,100 @@ Report
Workflow
====================
========
Experiment Manager
--------------------
------------------
.. autoclass:: qlib.workflow.expm.ExpManager
:members:
Experiment
--------------------
----------
.. autoclass:: qlib.workflow.exp.Experiment
:members:
Recorder
--------------------
--------
.. autoclass:: qlib.workflow.recorder.Recorder
:members:
Record Template
--------------------
---------------
.. automodule:: qlib.workflow.record_temp
:members:
Task Management
====================
===============
TaskGen
--------------------
-------
.. automodule:: qlib.workflow.task.gen
:members:
TaskManager
--------------------
-----------
.. automodule:: qlib.workflow.task.manage
:members:
Trainer
--------------------
-------
.. automodule:: qlib.model.trainer
:members:
Collector
--------------------
---------
.. automodule:: qlib.workflow.task.collect
:members:
Group
--------------------
-----
.. automodule:: qlib.model.ens.group
:members:
Ensemble
--------------------
--------
.. automodule:: qlib.model.ens.ensemble
:members:
Utils
--------------------
-----
.. automodule:: qlib.workflow.task.utils
:members:
Online Serving
====================
==============
Online Manager
--------------------
--------------
.. automodule:: qlib.workflow.online.manager
:members:
Online Strategy
--------------------
---------------
.. automodule:: qlib.workflow.online.strategy
:members:
Online Tool
--------------------
-----------
.. automodule:: qlib.workflow.online.utils
:members:
RecordUpdater
--------------------
-------------
.. automodule:: qlib.workflow.online.update
:members:
Utils
====================
=====
Serializable
--------------------
------------
.. automodule:: qlib.utils.serial.Serializable
:members:

22
docs/start/getdata.rst
View File

@@ -1,18 +1,18 @@
.. _getdata:
=============================
==============
Data Retrieval
=============================
==============
.. currentmodule:: qlib
Introduction
====================
============
Users can get stock data with ``Qlib``. The following examples demonstrate the basic user interface.
Examples
====================
========
``QLib`` Initialization:
@@ -30,7 +30,7 @@ If users followed steps in `initialization <initialization.html>`_ and downloade
Load trading calendar with given time range and frequency:
.. code-block:: python
>> from qlib.data import D
>> D.calendar(start_time='2010-01-01', end_time='2017-12-31', freq='day')[:2]
[Timestamp('2010-01-04 00:00:00'), Timestamp('2010-01-05 00:00:00')]
@@ -46,7 +46,7 @@ Parse a given market name into a stock pool config:
Load instruments of certain stock pool in the given time range:
.. code-block:: python
>> from qlib.data import D
>> instruments = D.instruments(market='csi300')
>> D.list_instruments(instruments=instruments, start_time='2010-01-01', end_time='2017-12-31', as_list=True)[:6]
@@ -79,14 +79,14 @@ For more details about filter, please refer `Filter API <../component/data.html>
Load features of certain instruments in a given time range:
.. code-block:: python
>> from qlib.data import D
>> instruments = ['SH600000']
>> fields = ['$close', '$volume', 'Ref($close, 1)', 'Mean($close, 3)', '$high-$low']
>> D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31', freq='day').head()
$close $volume Ref($close, 1) Mean($close, 3) $high-$low
instrument datetime
instrument datetime
SH600000 2010-01-04 86.778313 16162960.0 88.825928 88.061483 2.907631
2010-01-05 87.433578 28117442.0 86.778313 87.679273 3.235252
2010-01-06 85.713585 23632884.0 87.433578 86.641825 1.720009
@@ -108,7 +108,7 @@ Load features of certain stock pool in a given time range:
>> D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31', freq='day').head()
$close $volume Ref($close, 1) Mean($close, 3) $high-$low
instrument datetime
instrument datetime
SH600655 2010-01-04 2699.567383 158193.328125 2619.070312 2626.097738 124.580566
2010-01-08 2612.359619 77501.406250 2584.567627 2623.220133 83.373047
2010-01-11 2712.982422 160852.390625 2612.359619 2636.636556 146.621582
@@ -147,5 +147,5 @@ Here is an exmaple which does the same thing as above examples.
API
====================
===
To know more about how to use the Data, go to API Reference: `Data API <../reference/api.html#data>`_

24
docs/start/initialization.rst
View File

@@ -1,23 +1,23 @@
.. _initialization:
====================
===================
Qlib Initialization
====================
===================
.. currentmodule:: qlib
Initialization
=========================
==============
Please follow the steps below to initialize ``Qlib``.
Download and prepare the Data: execute the following command to download stock data. Please pay `attention` that the data is collected from `Yahoo Finance <https://finance.yahoo.com/lookup>`_ and the data might not be perfect. We recommend users to prepare their own data if they have high-quality datasets. Please refer to `Data <../component/data.html#converting-csv-format-into-qlib-format>`_ for more information about customized dataset.
.. code-block:: bash
python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn
Please refer to `Data Preparation <../component/data.html#data-preparation>`_ for more information about `get_data.py`,
@@ -30,7 +30,7 @@ Initialize Qlib before calling other APIs: run following code in python.
from qlib.constant import REG_CN
provider_uri = "~/.qlib/qlib_data/cn_data" # target_dir
qlib.init(provider_uri=provider_uri, region=REG_CN)
.. note::
Do not import qlib package in the repository directory of ``Qlib``, otherwise, errors may occur.
@@ -56,16 +56,16 @@ The following are several important parameters of `qlib.init` (`Qlib` has a lot
- `redis_port`
Type: int, optional parameter(default: 6379), port of `redis`
.. note::
.. note::
The value of `region` should be aligned with the data stored in `provider_uri`. Currently, ``scripts/get_data.py`` only provides China stock market data. If users want to use the US stock market data, they should prepare their own US-stock data in `provider_uri` and switch to US-stock mode.
.. note::
If Qlib fails to connect redis via `redis_host` and `redis_port`, cache mechanism will not be used! Please refer to `Cache <../component/data.html#cache>`_ for details.
- `exp_manager`
Type: dict, optional parameter, the setting of `experiment manager` to be used in qlib. Users can specify an experiment manager class, as well as the tracking URI for all the experiments. However, please be aware that we only support input of a dictionary in the following style for `exp_manager`. For more information about `exp_manager`, users can refer to `Recorder: Experiment Management <../component/recorder.html>`_.
.. code-block:: Python
# For example, if you want to set your tracking_uri to a <specific folder>, you can initialize qlib below
@@ -78,7 +78,7 @@ The following are several important parameters of `qlib.init` (`Qlib` has a lot
}
})
- `mongo`
Type: dict, optional parameter, the setting of `MongoDB <https://www.mongodb.com/>`_ which will be used in some features such as `Task Management <../advanced/task_management.html>`_, with high performance and clustered processing.
Type: dict, optional parameter, the setting of `MongoDB <https://www.mongodb.com/>`_ which will be used in some features such as `Task Management <../advanced/task_management.html>`_, with high performance and clustered processing.
Users need to follow the steps in `installation <https://www.mongodb.com/try/download/community>`_ to install MongoDB firstly and then access it via a URI.
Users can access mongodb with credential by setting "task_url" to a string like `"mongodb://%s:%s@%s" % (user, pwd, host + ":" + port)`.

11
docs/start/installation.rst
View File

@@ -1,8 +1,8 @@
.. _installation:
====================
============
Installation
====================
============
.. currentmodule:: qlib
@@ -24,7 +24,7 @@ Also, Users can install ``Qlib`` by the source code according to the following s
- Enter the root directory of ``Qlib``, in which the file ``setup.py`` exists.
- Then, please execute the following command to install the environment dependencies and install ``Qlib``:
.. code-block:: bash
$ pip install numpy
@@ -34,7 +34,7 @@ Also, Users can install ``Qlib`` by the source code according to the following s
.. note::
It's recommended to use anaconda/miniconda to setup the environment. ``Qlib`` needs lightgbm and pytorch packages, use pip to install them.
Use the following code to make sure the installation successful:
@@ -44,6 +44,3 @@ Use the following code to make sure the installation successful:
>>> import qlib
>>> qlib.__version__
<LATEST VERSION>
=====================

22
docs/start/integration.rst
View File

@@ -1,9 +1,9 @@
=========================================
========================
Custom Model Integration
=========================================
========================
Introduction
===================
============
``Qlib``'s `Model Zoo` includes models such as ``LightGBM``, ``MLP``, ``LSTM``, etc.. These models are examples of ``Forecast Model``. In addition to the default models ``Qlib`` provide, users can integrate their own custom models into ``Qlib``.
@@ -14,7 +14,7 @@ Users can integrate their own custom models according to the following steps.
- Test the custom model.
Custom Model Class
===========================
==================
The Custom models need to inherit `qlib.model.base.Model <../reference/api.html#module-qlib.model.base>`_ and override the methods in it.
- Override the `__init__` method
@@ -36,7 +36,7 @@ The Custom models need to inherit `qlib.model.base.Model <../reference/api.html#
- The parameters could include some `optional` parameters with default values, such as `num_boost_round = 1000` for `GBDT`.
- Code Example: In the following example, `num_boost_round = 1000` is an optional parameter.
.. code-block:: Python
def fit(self, dataset: DatasetH, num_boost_round = 1000, **kwargs):
# prepare dataset for lgb training and evaluation
@@ -101,14 +101,14 @@ The Custom models need to inherit `qlib.model.base.Model <../reference/api.html#
)
Configuration File
=======================
==================
The configuration file is described in detail in the `Workflow <../component/workflow.html#complete-example>`_ document. In order to integrate the custom model into ``Qlib``, users need to modify the "model" field in the configuration file. The configuration describes which models to use and how we can initialize it.
- Example: The following example describes the `model` field of configuration file about the custom lightgbm model mentioned above, where `module_path` is the module path, `class` is the class name, and `args` is the hyperparameter passed into the __init__ method. All parameters in the field is passed to `self._params` by `\*\*kwargs` in `__init__` except `loss = mse`.
- Example: The following example describes the `model` field of configuration file about the custom lightgbm model mentioned above, where `module_path` is the module path, `class` is the class name, and `args` is the hyperparameter passed into the __init__ method. All parameters in the field is passed to `self._params` by `\*\*kwargs` in `__init__` except `loss = mse`.
.. code-block:: YAML
model:
class: LGBModel
module_path: qlib.contrib.model.gbdt
@@ -126,7 +126,7 @@ The configuration file is described in detail in the `Workflow <../component/wor
Users could find configuration file of the baselines of the ``Model`` in ``examples/benchmarks``. All the configurations of different models are listed under the corresponding model folder.
Model Testing
=====================
=============
Assuming that the configuration file is ``examples/benchmarks/LightGBM/workflow_config_lightgbm.yaml``, users can run the following command to test the custom model:
.. code-block:: bash
@@ -136,10 +136,10 @@ Assuming that the configuration file is ``examples/benchmarks/LightGBM/workflow_
.. note:: ``qrun`` is a built-in command of ``Qlib``.
Also, ``Model`` can also be tested as a single module. An example has been given in ``examples/workflow_by_code.ipynb``.
Also, ``Model`` can also be tested as a single module. An example has been given in ``examples/workflow_by_code.ipynb``.
Reference
=====================
=========
To know more about ``Forecast Model``, please refer to `Forecast Model: Model Training & Prediction <../component/model.html>`_ and `Model API <../reference/api.html#module-qlib.model.base>`_.