liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-06-06 05:51:17 +08:00
Code Activity
Files
59c52eac0aa58271b6c9214f38bf5238c15e2421
qlib/scripts/data_collector
History
Pengrong Zhu 2aee9e0145 Add future calendar collector (#795) 
* fix Windows mount

* add future_calendar_collector

* update docs

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2022-01-16 10:14:27 +08:00
..
cn_index
fix cn_index collector (#780)
2021-12-26 14:12:48 +08:00
contrib
Add a check if change is mutated to YahooNormalize1d
2021-07-15 19:13:25 +08:00
crypto
Add Crypto dataset from coingecko (#733)
2021-12-31 22:24:26 +08:00
fund
[807] Move the REG_CONSTANT/EPS to constant.py. (#811)
2022-01-09 21:39:46 +08:00
us_index
Fix high-freq data (#702)
2021-11-20 15:03:53 +08:00
yahoo
[807] Move the REG_CONSTANT/EPS to constant.py. (#811)
2022-01-09 21:39:46 +08:00
base.py
fix automatic update of daily frequency data
2021-06-23 23:59:59 +08:00
future_calendar_collector.py
Add future calendar collector (#795)
2022-01-16 10:14:27 +08:00
index.py
Fix high-freq data (#702)
2021-11-20 15:03:53 +08:00
README.md
add description of dataset document (#742)
2021-12-12 09:49:10 +08:00
utils.py
Add Crypto dataset from coingecko (#733)
2021-12-31 22:24:26 +08:00

README.md

Data Collector

Introduction

Scripts for data collection

Custom Data Collection

Specific implementation reference: https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo

  1. Create a dataset code directory in the current directory
  2. Add collector.py
    • add collector class: 
      CUR_DIR = Path(__file__).resolve().parent
sys.path.append(str(CUR_DIR.parent.parent))
from data_collector.base import BaseCollector, BaseNormalize, BaseRun
class UserCollector(BaseCollector):
    ...
    • add normalize class: 
      class UserNormalzie(BaseNormalize):
    ...
    • add CLI class: 
      class Run(BaseRun):
    ...
  3. add README.md
  4. add requirements.txt

Description of dataset

Basic data
Features Price/Volume:
   - $close/$open/$low/$high/$volume/$change/$factor
Calendar <freq>.txt:
   - day.txt
   - 1min.txt
Instruments <market>.txt:
   - required: all.txt;
   - csi300.txt/csi500.txt/sp500.txt
  • Features: data, digital
    • if not adjusted, factor=1

Data-dependent component

To make the component running correctly, the dependent data are required

Component required data
Data retrieval Features, Calendar, Instrument
Backtest Features[Price/Volume], Calendar, Instruments