liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-09 22:10:56 +08:00
Code Activity

release-0.5.0 (#1)

Browse Source 
* init commit

* change the version number

* rich the docs&fix cache docs

* update index readme

* Modify cache class name

* Modify sharpe to information_ratio

* Modify Group- to Group

* add the description of graphical results & fix the backtest docs

* fix docs in details

* update docs

* Update introduction.rst

* Update README.md

* Update introduction.rst

* Update introduction.rst

* Update introduction.rst

* Update installation.rst

* Update installation.rst

* Update initialization.rst

* Update getdata.rst

* Update integration.rst

* Update initialization.rst

* Update getdata.rst

* Update estimator.rst

Modify some typos.

* Update README.md

Modify the typos.

* Update initialization.rst

* Update data.rst

* Update report.rst

* Update estimator.rst

* Update cumulative_return.py

* Update model.rst

* Update rank_label.py

* Update cumulative_return.py

* Update strategy.rst

* Update getdata.rst

* Update backtest.rst

* Update integration.rst

* Update getdata.rst

* Update introduction.rst

* Update introduction.rst

* Update README.md

* Update report.rst

* Update integration.rst

Fix typos

* Update installation.rst

Fix typos

* Update getdata.rst

* Update initialization.rst

Fix typos.

* add quick start docs&fix detials

* fix estimator docs & fix strategy docs

* fix the cahce in data.rst

* update documents

* Fix Corr && Rsquare

* fix data retrival example to csi300 & fix a data bug

* fix filter bug

* Fix data collector

* Modift model args

* add the log & fix README.md\quick.rst

* add enviroment depend & add intoduction of qlib-server online mode

* fix image center fomat & set log_only of docs is True

* fix README.md format

* update data preparation & readme logo image

* get_data support version

* Modify analysis names

* Modify analysis graph

* update report.rst & data.rst

* commmit estimator for merge

* minimal requirements

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update READEME.md

* Update READEME.md

* update estimator

* Fix doc urls

* fix get_data.py docstring

* update test_get_data.py

* Upate docs

* Upate docs

* Upate docs

Co-authored-by: bxdd <bxddream@gmail.com>
Co-authored-by: zhupr <zhu.pengrong@foxmail.com>
Co-authored-by: Wendi Li <wendili.academic@qq.com>
Co-authored-by: Dingsu Wang <dingsu.wang@gmail.com>
Co-authored-by: bxdd <45119470+bxdd@users.noreply.github.com>
Co-authored-by: cslwqxx <cslwqxx@users.noreply.github.com>
This commit is contained in:
you-n-g
2020-09-23 23:01:39 -05:00
committed by GitHub
parent 99ebd87cba
commit de9e13b171
82 changed files with 1580 additions and 1145 deletions
Download Patch File Download Diff File Expand all files Collapse all files

219
README.md
View File

@@ -1,60 +1,84 @@
<div style="align: center">
<img src="docs/_static/img/logo/white_bg_rec+word.png" />
</div>
Qlib is an AI-oriented quantitative investment platform, which aims to realize the potential, empower the research, and create the value of AI technologies in quantitative investment.
With Qlib, you can easily apply your favorite model to create a better Quant investment strategy.
With Qlib, you can easily try your ideas to create better Quant investment strategies.
For more details, please refer to our paper ["Qlib: An AI-oriented Quantitative Investment Platform"](https://arxiv.org/abs/2009.11189).
- [Framework of Qlib](#framework-of-qlib)
- [Quick start](#quick-start)
- [Installation](#installation)
- [Get Data](#get-data)
- [Auto Quant research workflow with _estimator_](#auto-quant-research-workflow-with-estimator)
- [Customized Quant research workflow by code](#customized-quant-research-workflow-by-code)
- [More About Qlib](#more-about-qlib)
- [Offline mode and online mode](#offline-mode-and-online-mode)
- [Performance of Qlib Data Server](#performance-of-qlib-data-server)
- [Contributing](#contributing)
- [Quick Start](#Quick-Start)
- [Installation](#Installation)
- [Data Preparation](#Data-Preparation)
- [Auto Quant Research Workflow with](#Auto-Quant-Research-Workflow)
- [Building Customized Quant Research Workflow by Code](#Building-Customized-Quant-Research-Workflow-by-Code)
- [More About Qlib](#More-About-Qlib)
- [Offline mode and online mode of data server](#Offline-Mode-and-Online-Mode-of-the-Data-Server)
- [Performance of Qlib Data Server](#Performance-of-Qlib-Data-Server)
- [Contributing](#Contributing)
# Framework of Qlib
![framework](docs/_static/img/framework.png)
<div style="align: center">
<img src="docs/_static/img/framework.png" />
</div>
At the module level, Qlib is a platform that consists of the above components. Each component is loose-coupling and can be used stand-alone.
At the module level, Qlib is a platform that consists of the above components. The components are designed as loose-coupled modules and each component could be used stand-alone.
| Name | Description |
| ------ | ----- |
| _Data layer_ | _DataServer_ focus on providing high performance infrastructure for user to retrieve and get raw data. _DataEnhancement_ will preprocess the data and provide the best dataset to be fed in to the models |
| _Interday Model_ | _Interday model_ focus on producing forecasting signals(aka. _alpha_). Models are trained by _Model Creator_ and managed by _Model Manager_. User could choose one or multiple models for forecasting. Multiple models could be combined with _Ensemble_ module |
| _Interday Strategy_ | _Portfolio Generator_ will take forecasting signals as input and output the orders based on current position to achieve target portfolio |
| _Intraday Trading_ | _Order Executor_ is responsible for executing orders produced by _Interday Strategy_ and returning the executed results. |
| _Analysis_ | User could get detailed analysis report of forecasting signal and portfolio in this part. |
| `Data layer` | `DataServer` focuses on providing high-performance infrastructure for users to manage and retrieve raw data. `DataEnhancement` will preprocess the data and provide the best dataset to be fed into the models. |
| `Interday Model` | `Interday model` focuses on producing prediction scores (aka. _alpha_). Models are trained by `Model Creator` and managed by `Model Manager`. Users could choose one or multiple models for prediction. Multiple models could be combined with `Ensemble` module. |
| `Interday Strategy` | `Portfolio Generator` will take prediction scores as input and output the orders based on the current position to achieve the target portfolio. |
| `Intraday Trading` | `Order Executor` is responsible for executing orders output by `Interday Strategy` and returning the executed results. |
| `Analysis` | Users could get a detailed analysis report of forecasting signals and portfolios in this part. |
* The modules with hand-drawn style is under development and will be released in the future.
* The modules with dashed border is highly user-customizable and extendible.
* The modules with hand-drawn style are under development and will be released in the future.
* The modules with dashed borders are highly user-customizable and extendible.
# Quick start
# Quick Start
This quick start guide tries to demonstrate
1. It's very easy to build a complete Quant research workflow and try your ideas with _Qlib_.
1. Though with *public data* and *simple models*, machine learning technologies **work very well** in practical Quant investment.
## Installation
To install Qlib from source you need _Cython_ in addition to the normal dependencies above:
Users can easily intsall ``Qlib`` according to the following steps:
```bash
pip install numpy
pip install --upgrade cython
```
* Before installing ``Qlib`` from source, you need to install some dependencies:
Clone the repository and then run:
```bash
python setup.py install
```
```bash
pip install numpy
pip install --upgrade cython
```
* Clone the repository and install ``Qlib``:
```bash
git clone https://github.com/microsoft/qlib.git && cd qlib
python setup.py install
```
## Get Data
- Load and prepare the Data: execute the following command to load the stock data:
## Data Preparation
Load and prepare data by running the following code:
```bash
python scripts/get_data.py qlib_data_cn --target_dir ~/.qlib/qlib_data/cn_data
```
This dataset is created by public data collected by [crawler scripts](scripts/data_collector/), which have been released in
the same repository.
Users could create the same dataset with it.
<!--
- Run the initialization code and get stock data:
@@ -82,76 +106,69 @@ python setup.py install
```
-->
## Auto Quant research workflow with _estimator_
Qlib provides a tool named `estimator` to run whole workflow automatically(including building dataset, train models, backtest, analysis)
## Auto Quant Research Workflow
Qlib provides a tool named `Estimator` to run the whole workflow automatically (including building dataset, training models, backtest and evaluation). You can start an auto quant research workflow and have a graphical reports analysis according to the following steps:
1. Run _estimator_ (_config.yaml_ for: [estimator_config.yaml](examples/estimator/estimator_config.yaml)):
1. Quant Research Workflow: Run `Estimator` with [estimator_config.yaml](examples/estimator/estimator_config.yaml) as following.
```bash
cd examples # Avoid running program under the directory contains `qlib`
estimator -c estimator/estimator_config.yaml
```
The result of `Estimator` is as follows, please refer to please refer to [Intraday Trading](https://qlib.readthedocs.io/en/latest/component/backtest.html) for more details about the result.
```bash
cd examples # Avoid running program under the directory contains `qlib`
estimator -c estimator/estimator_config.yaml
risk
excess_return_without_cost mean 0.000605
std 0.005481
annualized_return 0.152373
information_ratio 1.751319
max_drawdown -0.059055
excess_return_with_cost mean 0.000410
std 0.005478
annualized_return 0.103265
information_ratio 1.187411
max_drawdown -0.075024
```
Estimator result:
```bash
Here are detailed documents for [Estimator](https://qlib.readthedocs.io/en/latest/component/estimator.html).
risk
sub_bench mean 0.000662
std 0.004487
annual 0.166720
sharpe 2.340526
mdd -0.080516
sub_cost mean 0.000577
std 0.004482
annual 0.145392
sharpe 2.043494
mdd -0.083584
```
See the full documents for [Use _Estimator_ to Start An Experiment](TODO:URL).
2. Analysis
Run `examples/estimator/analyze_from_estimator.ipynb` in `jupyter notebook`
1. forecasting signal analysis
- Cumulative Return
![Cumulative Return](docs/_static/img/analysis/analysis_model_cumulative_return.png)
![long_short](docs/_static/img/analysis/analysis_model_long_short.png)
- Information Coefficient(IC)
![Information Coefficient](docs/_static/img/analysis/analysis_model_IC.png)
![Monthly IC](docs/_static/img/analysis/analysis_model_monthly_IC.png)
![IC](docs/_static/img/analysis/analysis_model_NDQ.png)
- Auto Correlation
![Auto Correlation](docs/_static/img/analysis/analysis_model_auto_correlation.png)
2. Graphical Reports Analysis: Run `examples/estimator/analyze_from_estimator.ipynb` with `jupyter notebook` to get graphical reports
- Forecasting signal (model prediction) analysis
- Cumulative Return of groups
![Cumulative Return](docs/_static/img/analysis/analysis_model_cumulative_return.png)
- Return distribution
![long_short](docs/_static/img/analysis/analysis_model_long_short.png)
- Information Coefficient (IC)
![Information Coefficient](docs/_static/img/analysis/analysis_model_IC.png)
![Monthly IC](docs/_static/img/analysis/analysis_model_monthly_IC.png)
![IC](docs/_static/img/analysis/analysis_model_NDQ.png)
- Auto Correlation of forecasting signal (model prediction)
![Auto Correlation](docs/_static/img/analysis/analysis_model_auto_correlation.png)
2. portfolio analysis
- Report
![Report](docs/_static/img/analysis/report.png)
<!--
- Score IC
![Score IC](docs/_static/img/score_ic.png)
- Cumulative Return
![Cumulative Return](docs/_static/img/cumulative_return.png)
- Risk Analysis
![Risk Analysis](docs/_static/img/risk_analysis.png)
- Rank Label
![Rank Label](docs/_static/img/rank_label.png)
-->
- Portfolio analysis
- Backtest return
![Report](docs/_static/img/analysis/report.png)
<!--
- Score IC
![Score IC](docs/_static/img/score_ic.png)
- Cumulative Return
![Cumulative Return](docs/_static/img/cumulative_return.png)
- Risk Analysis
![Risk Analysis](docs/_static/img/risk_analysis.png)
- Rank Label
![Rank Label](docs/_static/img/rank_label.png)
-->
## Customized Quant research workflow by code
Automatic workflow may not suite the research workflow of all Quant researchers. To support flexible Quant research workflow, Qlib also provide modularized interface to allow researchers to build their own workflow. [Here](TODO_URL) is a demo for customized Quant research workflow by code
## Building Customized Quant Research Workflow by Code
The automatic workflow may not suite the research workflow of all Quant researchers. To support a flexible Quant research workflow, Qlib also provides a modularized interface to allow researchers to build their own workflow by code. [Here](examples/train_backtest_analyze.ipynb) is a demo for customized Quant research workflow by code
# More About Qlib
The detailed documents are organized in [docs](docs).
The detailed documents are organized in [docs](docs/).
[Sphinx](http://www.sphinx-doc.org) and the readthedocs theme is required to build the documentation in html formats.
```bash
cd docs/
@@ -160,32 +177,32 @@ conda install sphinx sphinx_rtd_theme -y
# pip install sphinx sphinx_rtd_theme
make html
```
You can also view the [latest document](TODO_URL) online directly.
You can also view the [latest document](http://qlib.readthedocs.io/) online directly.
The roadmap is managed as a [github project](https://github.com/microsoft/qlib/projects/1).
Qlib is in active and continuing development. Our plan is in the roadmap, which is managed as a [github project](https://github.com/microsoft/qlib/projects/1).
## Offline mode and online mode
The data server of Qlib can both deployed as offline mode and online mode. The default mode is offline mode.
# Offline Mode and Online Mode of the Data Server
The data server of Qlib can either deployed as offline mode or online mode. The default mode is offline mode.
Under offline mode, the data will be deployed locally.
Under online mode, the data will be deployed as a shared data service. The data and their cache will be shared by clients. The data retrieving performance is expected to be improved due to a higher rate of cache hits. It will use less disk space, too. The documents of the online mode can be found in [Qlib-Server](TODO_link). The online mode can be deployed automatically with [Azure CLI based scripts](TODO_link)
Under online mode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too. The documents of the online mode can be found in [Qlib-Server](https://qlib-server.readthedocs.io/). The online mode can be deployed automatically with [Azure CLI based scripts](https://qlib-server.readthedocs.io/en/latest/build.html#one-click-deployment-in-azure). The source code of online data server can be found in [qlib-server repository](https://github.com/microsoft/qlib-server).
## Performance of Qlib Data Server
The performance of data processing is important to data-driven methods like AI technologies. As an AI-oriented platform, Qlib provides a solution for data storage and data processing. To demonstrate the performance of Qlib, We
compare Qlib with several other solutions.
The performance of data processing is important to data-driven methods like AI technologies. As an AI-oriented platform, Qlib provides a solution for data storage and data processing. To demonstrate the performance of Qlib data server, we
compare it with several other data storage solutions.
We evaluate the performance of several solutions by completing the same task,
which creates a dataset(14 features/factors) from the basic OHLCV daily data of a stock market(800 stocks each day from 2007 to 2020). The task involves data queries and processing.
We evaluate the performance of several storage solutions by finishing the same task,
which creates a dataset (14 features/factors) from the basic OHLCV daily data of a stock market (800 stocks each day from 2007 to 2020). The task involves data queries and processing.
| | HDF5 | MySQL | MongoDB | InfluxDB | Qlib -E -D | Qlib +E -D | Qlib +E +D |
| -- | ------ | ------ | -------- | --------- | ----------- | ------------ | ----------- |
| Total (1CPU) (seconds) | 184.4±3.7 | 365.3±7.5 | 253.6±6.7 | 368.2±3.6 | 147.0±8.8 | 47.6±1.0 | **7.4±0.3** |
| Total (64CPU) (seconds) | | | | | 8.8±0.6 | **4.2±0.2** | |
* `+(-)E` indicates with(out) `ExpressionCache`
* `+(-)D` indicates with(out) `DatasetCache`
* `+(-)E` indicates with (out) `ExpressionCache`
* `+(-)D` indicates with (out) `DatasetCache`
Most general-purpose databases take too much time on loading data. After looking into the underlying implementation, we find that data go through too many layers of interfaces and unnecessary format transformations in general-purpose database solutions.
Such overheads greatly slow down the data loading process.
@@ -199,7 +216,7 @@ Qlib data are stored in a compact format, which is efficient to be combined into
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
the right to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions