liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-15 00:36:55 +08:00
Code Activity

Qlib data doc (#1207)

Browse Source 
* Explain data crawler structure

* Add documentation for data and feature

* Update scripts/data_collector/yahoo/README.md

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

* Remove some confusing wording

* Add third party data source

* Fix command typo

* Update commands

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
This commit is contained in:
Di
2022-07-22 09:24:58 +08:00
committed by GitHub
parent 8199822ca0
commit 86f08e47e8
5 changed files with 97 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
scripts/data_collector/yahoo/README.md
View File

@@ -36,7 +36,7 @@ pip install -r requirements.txt
- `target_dir`: save dir, by default *~/.qlib/qlib_data/cn_data*
- `version`: dataset version, value from [`v1`, `v2`], by default `v1`
- `v2` end date is *2021-06*, `v1` end date is *2020-09*
- user can append data to `v2`: [automatic update of daily frequency data](#automatic-update-of-daily-frequency-datafrom-yahoo-finance)
- If users want to incrementally update data, they need to use yahoo collector to [collect data from scratch](#collector-yahoofinance-data-to-qlib).
- **the [benchmarks](https://github.com/microsoft/qlib/tree/main/examples/benchmarks) for qlib use `v1`**, *due to the unstable access to historical data by YahooFinance, there are some differences between `v2` and `v1`*
- `interval`: `1d` or `1min`, by default `1d`
- `region`: `cn` or `us` or `in`, by default `cn`
@@ -62,6 +62,8 @@ pip install -r requirements.txt
> collector *YahooFinance* data and *dump* into `qlib` format.
> If the above ready-made data can't meet users' requirements, users can follow this section to crawl the latest data and convert it to qlib-data.
1. download data to csv: `python scripts/data_collector/yahoo/collector.py download_data`
This will download the raw data such as high, low, open, close, adjclose price from yahoo to a local directory. One file per symbol.
- parameters:
- `source_dir`: save the directory
@@ -99,6 +101,10 @@ pip install -r requirements.txt
```
2. normalize data: `python scripts/data_collector/yahoo/collector.py normalize_data`
This will:
1. Normalize high, low, close, open price using adjclose.
2. Normalize the high, low, close, open price so that the first valid trading date's close price is 1.
- parameters:
- `source_dir`: csv directory
- `normalize_dir`: result directory
@@ -136,6 +142,8 @@ pip install -r requirements.txt
```
3. dump data: `python scripts/dump_bin.py dump_all`
This will convert the normalized csv in `feature` directory as numpy array and store the normalized data one file per column and one symbol per directory.
- parameters:
- `csv_path`: stock data path or directory, **normalize result(normalize_dir)**
- `qlib_dir`: qlib(dump) data director