liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-06-06 05:51:17 +08:00
Code Activity
Files
213eb6c2cd12342b6ec98f21300217e1659f3d58
qlib/scripts/data_collector
History
Alaa Kaddour 7095e755fa fix: replace deprecated pandas fillna(method=) with ffill()/bfill() (#1987) 
* fix: replace deprecated pandas fillna(method=) with ffill()/bfill()

  Replace deprecated fillna(method="ffill"/"bfill") calls with modern
  pandas ffill() and bfill() methods to fix FutureWarnings in pandas 2.x.

  Also includes black formatting fixes for compliance.

  This addresses the pandas deprecation warnings portion of issue #1981.
  Other issues (date parsing, type conversion, timezone handling) will be
  addressed in separate commits.

  Fixes:
  - Yahoo collector: 2 instances in calc_change() and adjusted_price()
  - BaoStock collector: 1 instance in calc_change()
  - Core utils: resam.py fillna operations
  - Backtest: profit_attribution.py stock data processing
  - High-freq ops: FFillNan and BFillNan operators
  - Position analysis: parse_position.py weight processing

  Partially addresses GitHub issue #1981

* lint with black

* lint with black

* limit minimum version of pandas

* limit minimum version of pandas

---------

Co-authored-by: Linlang <Lv.Linlang@hotmail.com>
2025-08-19 16:00:29 +08:00
..
baostock_5min
fix: replace deprecated pandas fillna(method=) with ffill()/bfill() (#1987)
2025-08-19 16:00:29 +08:00
br_index
Update requirements.txt (#1829)
2024-12-04 12:10:05 +08:00
cn_index
Fix csi300 constituents url (#1883)
2025-01-03 16:57:17 +08:00
contrib
Update README.md (#1179)
2022-07-07 00:06:47 +08:00
crowd_source
Update links to chenditc/investment_data to always point to latest release (#1877)
2025-01-03 13:56:49 +08:00
crypto
feat: data improve, support parquet (#1966)
2025-08-07 15:04:37 +08:00
fund
feat: data improve, support parquet (#1966)
2025-08-07 15:04:37 +08:00
pit
feat: data improve, support parquet (#1966)
2025-08-07 15:04:37 +08:00
us_index
add_baostock_collector (#1641)
2023-11-21 20:31:47 +08:00
yahoo
fix: replace deprecated pandas fillna(method=) with ffill()/bfill() (#1987)
2025-08-19 16:00:29 +08:00
base.py
fix panic during normalizing the invalid data (#1698)
2024-06-02 06:54:39 +08:00
future_calendar_collector.py
add_baostock_collector (#1641)
2023-11-21 20:31:47 +08:00
index.py
Fixed pandas FutureWarning (#1073)
2022-04-27 18:43:26 +08:00
README.md
add description of dataset document (#742)
2021-12-12 09:49:10 +08:00
utils.py
fixed a problem with multi index caused by the default value of groupkey (#1917)
2025-05-13 16:02:49 +08:00

README.md

Data Collector

Introduction

Scripts for data collection

Custom Data Collection

Specific implementation reference: https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo

  1. Create a dataset code directory in the current directory
  2. Add collector.py
    • add collector class: 
      CUR_DIR = Path(__file__).resolve().parent
sys.path.append(str(CUR_DIR.parent.parent))
from data_collector.base import BaseCollector, BaseNormalize, BaseRun
class UserCollector(BaseCollector):
    ...
    • add normalize class: 
      class UserNormalzie(BaseNormalize):
    ...
    • add CLI class: 
      class Run(BaseRun):
    ...
  3. add README.md
  4. add requirements.txt

Description of dataset

Basic data
Features Price/Volume:
   - $close/$open/$low/$high/$volume/$change/$factor
Calendar <freq>.txt:
   - day.txt
   - 1min.txt
Instruments <market>.txt:
   - required: all.txt;
   - csi300.txt/csi500.txt/sp500.txt
  • Features: data, digital
    • if not adjusted, factor=1

Data-dependent component

To make the component running correctly, the dependent data are required

Component required data
Data retrieval Features, Calendar, Instrument
Backtest Features[Price/Volume], Calendar, Instruments