liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-10 06:20:57 +08:00
Code Activity

Ibovespa index support (#990)

Browse Source 
* feat: download ibovespa index historic composition

ibovespa(ibov) is the largest index in Brazil's stocks exchange.
The br_index folder has support for downloading new companies for the current index composition.
And has support, as well, for downloading companies from historic composition of ibov index.

Partially resolves issue #956

* fix: typo error instead of end_date, it was written end_ate

* feat: adds support for downloading stocks historic prices from Brazil's stocks exchange (B3)

Together with commit c2f933 it resolves issue #956

* fix: code formatted with black.

* wip: Creating code logic for brazils stock market data normalization

* docs: brazils stock market data normalization code documentation

* fix: code formatted the with black

* docs: fixed typo

* docs: more info about python version used to generate requirements.txt file

* docs: added BeautifulSoup requirements

* feat: removed debug prints

* feat: added ibov_index_composition variable as a class attribute of IBOVIndex

* feat: added increment to generate the four month period used by the ibov index

* refactor: Added get_instruments() method inside utils.py for better code usability.

Message in the PR request to understand the context of the change

In the course of reviewing this PR we found two issues.

    1. there are multiple places where the get_instruments() method is used,
	and we feel that scripts.index.py is the best place for the
	get_instruments() method to go.
    2. data_collector.utils has some very generic stuff put inside it.

* refactor: improve brazils stocks download speed

The reason to use retry=2 is due to the fact that
Yahoo Finance unfortunately does not keep track of the majority
of Brazilian stocks.

Therefore, the decorator deco_retry with retry argument
set to 5 will keep trying to get the stock data 5 times,
which makes the code to download Brazilians stocks very slow.

In future, this may change, but for now
I suggest to leave retry argument to 1 or 2 in
order to improve download speed.

In order to achieve this code logic an argument called retry_config
was added into YahooCollectorBR1d and YahooCollectorBR1min

* fix: added __main__ at the bottom of the script

* refactor: changed interface inside each index

Using partial as `fire.Fire(partial(get_instruments, market_index="br_index" ))`
will make the interface easier for the user to execute the script.
Then all the collector.py CLI in each folder can remove a redundant arguments.

* refactor: implemented  class interface retry into YahooCollectorBR

* docs: added BR as a possible region into the documentation

* refactor: make retry attribute part of the interface

This way we don't have to use hasattr to access the retry attribute as previously done
This commit is contained in:
igor17400
2022-04-05 22:01:29 -03:00
committed by GitHub
parent 6edd0bf298
commit 56cfa480dc
10 changed files with 577 additions and 89 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
scripts/data_collector/yahoo/README.md
View File

@@ -66,7 +66,7 @@ pip install -r requirements.txt
- `source_dir`: save the directory
- `interval`: `1d` or `1min`, by default `1d`
> **due to the limitation of the *YahooFinance API*, only the last month's data is available in `1min`**
- `region`: `CN` or `US` or `IN`, by default `CN`
- `region`: `CN` or `US` or `IN` or `BR`, by default `CN`
- `delay`: `time.sleep(delay)`, by default *0.5*
- `start`: start datetime, by default *"2000-01-01"*; *closed interval(including start)*
- `end`: end datetime, by default `pd.Timestamp(datetime.datetime.now() + pd.Timedelta(days=1))`; *open interval(excluding end)*
@@ -80,14 +80,21 @@ pip install -r requirements.txt
python collector.py download_data --source_dir ~/.qlib/stock_data/source/cn_data --start 2020-01-01 --end 2020-12-31 --delay 1 --interval 1d --region CN
# cn 1min data
python collector.py download_data --source_dir ~/.qlib/stock_data/source/cn_data_1min --delay 1 --interval 1min --region CN
# us 1d data
python collector.py download_data --source_dir ~/.qlib/stock_data/source/us_data --start 2020-01-01 --end 2020-12-31 --delay 1 --interval 1d --region US
# us 1min data
python collector.py download_data --source_dir ~/.qlib/stock_data/source/us_data_1min --delay 1 --interval 1min --region US
# in 1d data
python collector.py download_data --source_dir ~/.qlib/stock_data/source/in_data --start 2020-01-01 --end 2020-12-31 --delay 1 --interval 1d --region IN
# in 1min data
python collector.py download_data --source_dir ~/.qlib/stock_data/source/in_data_1min --delay 1 --interval 1min --region IN
# br 1d data
python collector.py download_data --source_dir ~/.qlib/stock_data/source/br_data --start 2003-01-03 --end 2022-03-01 --delay 1 --interval 1d --region BR
# br 1min data
python collector.py download_data --source_dir ~/.qlib/stock_data/source/br_data_1min --delay 1 --interval 1min --region BR
```
2. normalize data: `python scripts/data_collector/yahoo/collector.py normalize_data`
@@ -116,8 +123,15 @@ pip install -r requirements.txt
```bash
# normalize 1d cn
python collector.py normalize_data --source_dir ~/.qlib/stock_data/source/cn_data --normalize_dir ~/.qlib/stock_data/source/cn_1d_nor --region CN --interval 1d
# normalize 1min cn
python collector.py normalize_data --qlib_data_1d_dir ~/.qlib/qlib_data/cn_data --source_dir ~/.qlib/stock_data/source/cn_data_1min --normalize_dir ~/.qlib/stock_data/source/cn_1min_nor --region CN --interval 1min
# normalize 1d br
python scripts/data_collector/yahoo/collector.py normalize_data --source_dir ~/.qlib/stock_data/source/br_data --normalize_dir ~/.qlib/stock_data/source/br_1d_nor --region BR --interval 1d
# normalize 1min br
python collector.py normalize_data --qlib_data_1d_dir ~/.qlib/qlib_data/br_data --source_dir ~/.qlib/stock_data/source/br_data_1min --normalize_dir ~/.qlib/stock_data/source/br_1min_nor --region BR --interval 1min
```
3. dump data: `python scripts/dump_bin.py dump_all`

80
scripts/data_collector/yahoo/collector.py
View File

@@ -2,6 +2,7 @@
# Licensed under the MIT License.
import abc
from re import I
import sys
import copy
import time
@@ -35,6 +36,7 @@ from data_collector.utils import (
get_hs_stock_symbols,
get_us_stock_symbols,
get_in_stock_symbols,
get_br_stock_symbols,
generate_minutes_calendar_from_daily,
)
@@ -42,6 +44,8 @@ INDEX_BENCH_URL = "http://push2his.eastmoney.com/api/qt/stock/kline/get?secid=1.
class YahooCollector(BaseCollector):
retry = 5 # Configuration attribute. How many times will it try to re-request the data if the network fails.
def __init__(
self,
save_dir: [str, Path],
@@ -146,7 +150,7 @@ class YahooCollector(BaseCollector):
def get_data(
self, symbol: str, interval: str, start_datetime: pd.Timestamp, end_datetime: pd.Timestamp
) -> pd.DataFrame:
@deco_retry(retry_sleep=self.delay)
@deco_retry(retry_sleep=self.delay, retry=self.retry)
def _get_simple(start_, end_):
self.sleep()
_remote_interval = "1m" if interval == self.INTERVAL_1min else interval
@@ -311,6 +315,55 @@ class YahooCollectorIN1min(YahooCollectorIN):
pass
class YahooCollectorBR(YahooCollector, ABC):
def retry(cls):
""""
The reason to use retry=2 is due to the fact that
Yahoo Finance unfortunately does not keep track of some
Brazilian stocks.
Therefore, the decorator deco_retry with retry argument
set to 5 will keep trying to get the stock data up to 5 times,
which makes the code to download Brazilians stocks very slow.
In future, this may change, but for now
I suggest to leave retry argument to 1 or 2 in
order to improve download speed.
To achieve this goal an abstract attribute (retry)
was added into YahooCollectorBR base class
"""
raise NotImplementedError
def get_instrument_list(self):
logger.info("get BR stock symbols......")
symbols = get_br_stock_symbols() + [
"^BVSP",
]
logger.info(f"get {len(symbols)} symbols.")
return symbols
def download_index_data(self):
pass
def normalize_symbol(self, symbol):
return code_to_fname(symbol).upper()
@property
def _timezone(self):
return "Brazil/East"
class YahooCollectorBR1d(YahooCollectorBR):
retry = 2
pass
class YahooCollectorBR1min(YahooCollectorBR):
retry = 2
pass
class YahooNormalize(BaseNormalize):
COLUMNS = ["open", "close", "high", "low", "volume"]
DAILY_FORMAT = "%Y-%m-%d"
@@ -833,6 +886,29 @@ class YahooNormalizeCN1min(YahooNormalizeCN, YahooNormalize1minOffline):
return get_calendar_list("ALL")
class YahooNormalizeBR:
def _get_calendar_list(self) -> Iterable[pd.Timestamp]:
return get_calendar_list("BR_ALL")
class YahooNormalizeBR1d(YahooNormalizeBR, YahooNormalize1d):
pass
class YahooNormalizeBR1min(YahooNormalizeBR, YahooNormalize1minOffline):
CALC_PAUSED_NUM = False
def _get_calendar_list(self) -> Iterable[pd.Timestamp]:
# TODO: support 1min
raise ValueError("Does not support 1min")
def _get_1d_calendar_list(self):
return get_calendar_list("BR_ALL")
def symbol_to_yahoo(self, symbol):
return fname_to_code(symbol)
class Run(BaseRun):
def __init__(self, source_dir=None, normalize_dir=None, max_workers=1, interval="1d", region=REGION_CN):
"""
@@ -848,7 +924,7 @@ class Run(BaseRun):
interval: str
freq, value from [1min, 1d], default 1d
region: str
region, value from ["CN", "US"], default "CN"
region, value from ["CN", "US", "BR"], default "CN"
"""
super().__init__(source_dir, normalize_dir, max_workers, interval)
self.region = region

3
scripts/data_collector/yahoo/requirements.txt
View File

@@ -7,3 +7,6 @@ tqdm
lxml
yahooquery
joblib
beautifulsoup4
bs4
soupsieve