liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-06-06 14:01:28 +08:00
Code Activity
Files
2e9a00a9f7ceec3ebe97898277bb935190ff04cc
qlib/scripts/data_collector
History
kzhdev 2e9a00a9f7 fix(data_collector): fix us_index collector.py Http Error 403 Forbidden; Remove FutureWarning (#2047) 
* Fix 403 Forbidden error; Remove FutureWarning:

* use fake_useragent

* Fix lint format error

* Add timeout to fix pylint error
2025-11-18 16:06:53 +08:00
..
baostock_5min
fix: replace deprecated pandas fillna(method=) with ffill()/bfill() (#1987)
2025-08-19 16:00:29 +08:00
br_index
Update requirements.txt (#1829)
2024-12-04 12:10:05 +08:00
cn_index
Fix csi300 constituents url (#1883)
2025-01-03 16:57:17 +08:00
contrib
Update README.md (#1179)
2022-07-07 00:06:47 +08:00
crowd_source
Update links to chenditc/investment_data to always point to latest release (#1877)
2025-01-03 13:56:49 +08:00
crypto
feat: data improve, support parquet (#1966)
2025-08-07 15:04:37 +08:00
fund
feat: data improve, support parquet (#1966)
2025-08-07 15:04:37 +08:00
pit
feat: data improve, support parquet (#1966)
2025-08-07 15:04:37 +08:00
us_index
fix(data_collector): fix us_index collector.py Http Error 403 Forbidden; Remove FutureWarning (#2047)
2025-11-18 16:06:53 +08:00
yahoo
fix: typo in integration documentation: 'userd' -> 'used' (#2034)
2025-10-16 11:07:55 +08:00
base.py
fix panic during normalizing the invalid data (#1698)
2024-06-02 06:54:39 +08:00
future_calendar_collector.py
add_baostock_collector (#1641)
2023-11-21 20:31:47 +08:00
index.py
Fixed pandas FutureWarning (#1073)
2022-04-27 18:43:26 +08:00
README.md
add description of dataset document (#742)
2021-12-12 09:49:10 +08:00
utils.py
fix: typo in integration documentation: 'userd' -> 'used' (#2034)
2025-10-16 11:07:55 +08:00

README.md

Data Collector

Introduction

Scripts for data collection

Custom Data Collection

Specific implementation reference: https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo

  1. Create a dataset code directory in the current directory
  2. Add collector.py
    • add collector class: 
      CUR_DIR = Path(__file__).resolve().parent
sys.path.append(str(CUR_DIR.parent.parent))
from data_collector.base import BaseCollector, BaseNormalize, BaseRun
class UserCollector(BaseCollector):
    ...
    • add normalize class: 
      class UserNormalzie(BaseNormalize):
    ...
    • add CLI class: 
      class Run(BaseRun):
    ...
  3. add README.md
  4. add requirements.txt

Description of dataset

Basic data
Features Price/Volume:
   - $close/$open/$low/$high/$volume/$change/$factor
Calendar <freq>.txt:
   - day.txt
   - 1min.txt
Instruments <market>.txt:
   - required: all.txt;
   - csi300.txt/csi500.txt/sp500.txt
  • Features: data, digital
    • if not adjusted, factor=1

Data-dependent component

To make the component running correctly, the dependent data are required

Component required data
Data retrieval Features, Calendar, Instrument
Backtest Features[Price/Volume], Calendar, Instruments