liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-01 10:01:19 +08:00
Code Activity
Files
faa99f30faed0bf40328f5921f98a91ecdd3820c
qlib/scripts/data_collector
History
bxdd faa99f30fa Support Point-in-time Data Operation (#343) 
* add period ops class

* black format

* add pit data read

* fix bug in period ops

* update ops runnable

* update PIT test example

* black format

* update PIT test

* update tets_PIT

* update code format

* add check_feature_exist

* black format

* optimize the PIT Algorithm

* fix bug

* update example

* update test_PIT name

* add pit collector

* black format

* fix bugs

* fix try

* fix bug & add dump_pit.py

* Successfully run and understand PIT

* Add some docs and remove a bug

* mv crypto collector

* black format

* Run succesfully after merging master

* Pass test and fix code

* remove useless PIT code

* fix PYlint

* Rename

Co-authored-by: Young <afe.young@gmail.com>
2022-03-10 14:27:52 +08:00
..
cn_index
fix cn_index collector (#780)
2021-12-26 14:12:48 +08:00
contrib
Add a check if change is mutated to YahooNormalize1d
2021-07-15 19:13:25 +08:00
crypto
Support Point-in-time Data Operation (#343)
2022-03-10 14:27:52 +08:00
fund
[949] - Remove argument internal in BaseRun::download_data. (#953)
2022-03-08 10:26:35 +08:00
pit
Support Point-in-time Data Operation (#343)
2022-03-10 14:27:52 +08:00
us_index
Fix high-freq data (#702)
2021-11-20 15:03:53 +08:00
yahoo
[949] - Remove argument internal in BaseRun::download_data. (#953)
2022-03-08 10:26:35 +08:00
base.py
Support Point-in-time Data Operation (#343)
2022-03-10 14:27:52 +08:00
future_calendar_collector.py
Add future calendar collector (#795)
2022-01-16 10:14:27 +08:00
index.py
Fix high-freq data (#702)
2021-11-20 15:03:53 +08:00
README.md
add description of dataset document (#742)
2021-12-12 09:49:10 +08:00
utils.py
Support Point-in-time Data Operation (#343)
2022-03-10 14:27:52 +08:00

README.md

Data Collector

Introduction

Scripts for data collection

Custom Data Collection

Specific implementation reference: https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo

  1. Create a dataset code directory in the current directory
  2. Add collector.py
    • add collector class: 
      CUR_DIR = Path(__file__).resolve().parent
sys.path.append(str(CUR_DIR.parent.parent))
from data_collector.base import BaseCollector, BaseNormalize, BaseRun
class UserCollector(BaseCollector):
    ...
    • add normalize class: 
      class UserNormalzie(BaseNormalize):
    ...
    • add CLI class: 
      class Run(BaseRun):
    ...
  3. add README.md
  4. add requirements.txt

Description of dataset

Basic data
Features Price/Volume:
   - $close/$open/$low/$high/$volume/$change/$factor
Calendar <freq>.txt:
   - day.txt
   - 1min.txt
Instruments <market>.txt:
   - required: all.txt;
   - csi300.txt/csi500.txt/sp500.txt
  • Features: data, digital
    • if not adjusted, factor=1

Data-dependent component

To make the component running correctly, the dependent data are required

Component required data
Data retrieval Features, Calendar, Instrument
Backtest Features[Price/Volume], Calendar, Instruments