liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-06-06 05:51:17 +08:00
Code Activity
Files
3e79a088ef52c45861135f84baf33a3001366935
qlib/scripts/data_collector
History
Arthur Cui 3e79a088ef Add Crypto dataset from coingecko (#733) 
* add crypto symbols collectors

* add crypto data collector

* add crypto symbols collectors

* add crypto data collector

* solver region and source problem

* fix merge

* fix merge

* clean all cn information

Co-authored-by: DefangCui <170007807@pku.edu.cn>
2021-12-31 22:24:26 +08:00
..
cn_index
fix cn_index collector (#780)
2021-12-26 14:12:48 +08:00
contrib
Add a check if change is mutated to YahooNormalize1d
2021-07-15 19:13:25 +08:00
crypto
Add Crypto dataset from coingecko (#733)
2021-12-31 22:24:26 +08:00
fund
fix_typo (#790)
2021-12-31 22:14:47 +08:00
us_index
Fix high-freq data (#702)
2021-11-20 15:03:53 +08:00
yahoo
Reformat example data names: use {region}_data for 1-day data, and {region}_data_1min for 1-min data (#781)
2021-12-28 23:58:49 +08:00
base.py
fix automatic update of daily frequency data
2021-06-23 23:59:59 +08:00
index.py
Fix high-freq data (#702)
2021-11-20 15:03:53 +08:00
README.md
add description of dataset document (#742)
2021-12-12 09:49:10 +08:00
utils.py
Add Crypto dataset from coingecko (#733)
2021-12-31 22:24:26 +08:00

README.md

Data Collector

Introduction

Scripts for data collection

Custom Data Collection

Specific implementation reference: https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo

  1. Create a dataset code directory in the current directory
  2. Add collector.py
    • add collector class: 
      CUR_DIR = Path(__file__).resolve().parent
sys.path.append(str(CUR_DIR.parent.parent))
from data_collector.base import BaseCollector, BaseNormalize, BaseRun
class UserCollector(BaseCollector):
    ...
    • add normalize class: 
      class UserNormalzie(BaseNormalize):
    ...
    • add CLI class: 
      class Run(BaseRun):
    ...
  3. add README.md
  4. add requirements.txt

Description of dataset

Basic data
Features Price/Volume:
   - $close/$open/$low/$high/$volume/$change/$factor
Calendar <freq>.txt:
   - day.txt
   - 1min.txt
Instruments <market>.txt:
   - required: all.txt;
   - csi300.txt/csi500.txt/sp500.txt
  • Features: data, digital
    • if not adjusted, factor=1

Data-dependent component

To make the component running correctly, the dependent data are required

Component required data
Data retrieval Features, Calendar, Instrument
Backtest Features[Price/Volume], Calendar, Instruments