mirror of
https://github.com/microsoft/qlib.git
synced 2026-07-07 13:00:58 +08:00
Merge nested main (#597)
* MVP for Indian Stocks in qlib using yahooquery * cleaned with black * cleaned with black * add YahooNormalizeIN and YahooNormalizeIN1d * cleaned the code * added 1min for IN and also updated readme * update comments * fix comments * recorder support upload both raw file and directory * fix comments * Update README.md * Fix docs of QlibRecorder * sort index after loader (#538) make sure the fetch method is based on a index-sorted pd.DataFrame * refactor online serving rolling api * refactor TRA * format by black * fix horizon * fix TRA when use single head * clean up * improve pretrain * update README * fix tra when logdir is None * fix tra when logdir is None * Update strategy.py * Update README.md * Update README.md * Conda Suggestion * code standard docs * Update ensemble.py (#560) * Fix CI Bug (#575) Co-authored-by: yuxwang <anduinnn@foxmail.com> * Update gen.py (#576) * Fix multi-process loop calls (#574) * check lexsort in the 'lazy_sort_index' function (#566) * check lexsort * check lexsort * lexsort comment * lexsort comment * Delete .DS_Store * Update README.md * bug fix & use oracle transport pretrain * mend * Add `backend_freq_config` parameter, support multi-freq uri * Add sample_config to QlibDataLoader, support multi-freq * add multi-freq example * get_cls_kwargs renamed get_callable_kwargs * support multi-freq uri * Add inst_processors to D.features * Fix typo * Fix the index type of the multi-freq example * Fix duplicate mlflow directories in tests * Add DataPathManager to QlibConfig && modify inst_processors to supports list only * Modify the default value in the multi_freq example * Modify client-server mode and dataset-cache to disable inst_processor * Add wheel package to github CI * fix comment * Update FAQ.rst * Update README.md Fix wrong link * Update the docs of TaskManager (#586) * Update manage.py * update yaml * update run_all_model * Modify the Feature to be case sensitive (#589) * update README * remove verbose * fix spell bug * fix typos (#592) * Update Release Note * fix portfolio bug * Add calendar support for resample * add freq kwargs * test.yml: Remove redundant code (#595) * Supporting shared processor (#596) * Supporting shared processor * fix readonly reverse bug * remove pytests dependency * with fit bug * fix parameter error * fix comments * Fix undefined names in Python code (#599) * Update pytorch_tabnet.py $ `flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics` ``` ./qlib/qlib/contrib/model/pytorch_tabnet.py:567:38: F821 undefined name 'inp' self.independ.append(GLU(inp, out_dim, vbs=vbs)) ^ ./qlib/examples/model_rolling/task_manager_rolling.py:75:18: F821 undefined name 'task_train' run_task(task_train, self.task_pool, experiment_name=self.experiment_name) ^ 2 F821 undefined name 'task_train' 2 ``` * Fix undefined names in Python code * from qlib.model.trainer import task_train * update seed * fix some docstring * add comments * Fix SimpleDatasetCache * Update setup.py updated classifiers * Update setup.py change to matplotlib==3.3 * Update python-publish.yml added python 3.9 * updategrade version number * Update model list * fix the type of filter_pipe * fix comment * fix record_temp * update cvxpy version * Update code_standard.rst (#587) * Update code_standard.rst * Update docs/developer/code_standard.rst Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> * Add file lock for MLflowExpManager (#619) * fix torch version * Share version number (#620) * Update initialization.rst (#622) * Update initialization.rst * Update docs/start/initialization.rst Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> * Update docs/start/initialization.rst Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> * fix bugs for running previous exmaple * fix deal amount bug * update change doc (#623) * Add files via upload * Update README.md * Update README.md * Update README.md * Delete change doc.gif * Add files via upload * Update README.md * Delete change doc.gif * Add files via upload * Delete change doc.gif * Add files via upload * Update README.md Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> * update doc * simplify run all model * fix run all model bug * Fix Models (#483) * fix gat dataset * fix tft model * Update tft.py * Fix tft.py Co-authored-by: Pengrong Zhu <zhu.pengrong@foxmail.com> * type and skip empty exp * fix model yaml config * fix tft import bug * skip empty result * fix model and yaml bug * fix wrong generate parameter * Modify multi-freq example (#626) * modify the example of multi-freq * add Copyright * add a comment to average_ops.py * modify the example of multi-freq * add comment to multi_freq_handler.py * add the Ref expression description to multi_freq_handler.py * add expression description to multi_freq_handler.py * update images * fix workflow and update framework Co-authored-by: Gaurav <2796gaurav@gmail.com> Co-authored-by: 2796gaurav <17353992+2796gaurav@users.noreply.github.com> Co-authored-by: bxdd <bxd98@126.com> Co-authored-by: Young <afe.young@gmail.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: Dong Zhou <Zhou.Dong@microsoft.com> Co-authored-by: ZhangTP1996 <ztp18@mails.tsinghua.edu.cn> Co-authored-by: demon143 <59681577+demon143@users.noreply.github.com> Co-authored-by: Wangwuyi123 <51237097+Wangwuyi123@users.noreply.github.com> Co-authored-by: yuxwang <anduinnn@foxmail.com> Co-authored-by: Pengrong Zhu <zhu.pengrong@foxmail.com> Co-authored-by: Mark Zhao <50850474+markzhao98@users.noreply.github.com> Co-authored-by: cslwqxx <cslwqxx@users.noreply.github.com> Co-authored-by: Dong Zhou <evanzd@users.noreply.github.com> Co-authored-by: SaintMalik <37118134+saintmalik@users.noreply.github.com> Co-authored-by: Christian Clauss <cclauss@me.com> Co-authored-by: Anurag Kumar <mailanu98@gmail.com> Co-authored-by: demon143 <785696300@qq.com>
This commit is contained in:
wangwenxi-handsome
committed by
GitHub
GitHub
parent
163e3c6266
commit
3760a18a8d
@@ -17,6 +17,7 @@ import abc
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from typing import Union, Iterable
|
||||
from collections import OrderedDict
|
||||
|
||||
from ..config import C
|
||||
@@ -216,12 +217,14 @@ class CacheUtils:
|
||||
redis_lock.reset_all(r)
|
||||
|
||||
@staticmethod
|
||||
def visit(cache_path):
|
||||
def visit(cache_path: Union[str, Path]):
|
||||
# FIXME: Because read_lock was canceled when reading the cache, multiple processes may have read and write exceptions here
|
||||
try:
|
||||
with open(cache_path + ".meta", "rb") as f:
|
||||
cache_path = Path(cache_path)
|
||||
meta_path = cache_path.with_suffix(".meta")
|
||||
with meta_path.open("rb") as f:
|
||||
d = pickle.load(f)
|
||||
with open(cache_path + ".meta", "wb") as f:
|
||||
with meta_path.open("wb") as f:
|
||||
try:
|
||||
d["meta"]["last_visit"] = str(time.time())
|
||||
d["meta"]["visits"] = d["meta"]["visits"] + 1
|
||||
@@ -249,17 +252,17 @@ class CacheUtils:
|
||||
|
||||
@staticmethod
|
||||
@contextlib.contextmanager
|
||||
def reader_lock(redis_t, lock_name):
|
||||
lock_name = f"{C.provider_uri}:{lock_name}"
|
||||
current_cache_rlock = redis_lock.Lock(redis_t, "%s-rlock" % lock_name)
|
||||
current_cache_wlock = redis_lock.Lock(redis_t, "%s-wlock" % lock_name)
|
||||
def reader_lock(redis_t, lock_name: str):
|
||||
current_cache_rlock = redis_lock.Lock(redis_t, f"{lock_name}-rlock")
|
||||
current_cache_wlock = redis_lock.Lock(redis_t, f"{lock_name}-wlock")
|
||||
lock_reader = f"{lock_name}-reader"
|
||||
# make sure only one reader is entering
|
||||
current_cache_rlock.acquire(timeout=60)
|
||||
try:
|
||||
current_cache_readers = redis_t.get("%s-reader" % lock_name)
|
||||
current_cache_readers = redis_t.get(lock_reader)
|
||||
if current_cache_readers is None or int(current_cache_readers) == 0:
|
||||
CacheUtils.acquire(current_cache_wlock, lock_name)
|
||||
redis_t.incr("%s-reader" % lock_name)
|
||||
redis_t.incr(lock_reader)
|
||||
finally:
|
||||
current_cache_rlock.release()
|
||||
try:
|
||||
@@ -268,9 +271,9 @@ class CacheUtils:
|
||||
# make sure only one reader is leaving
|
||||
current_cache_rlock.acquire(timeout=60)
|
||||
try:
|
||||
redis_t.decr("%s-reader" % lock_name)
|
||||
if int(redis_t.get("%s-reader" % lock_name)) == 0:
|
||||
redis_t.delete("%s-reader" % lock_name)
|
||||
redis_t.decr(lock_reader)
|
||||
if int(redis_t.get(lock_reader)) == 0:
|
||||
redis_t.delete(lock_reader)
|
||||
current_cache_wlock.reset()
|
||||
finally:
|
||||
current_cache_rlock.release()
|
||||
@@ -278,8 +281,7 @@ class CacheUtils:
|
||||
@staticmethod
|
||||
@contextlib.contextmanager
|
||||
def writer_lock(redis_t, lock_name):
|
||||
lock_name = f"{C.provider_uri}:{lock_name}"
|
||||
current_cache_wlock = redis_lock.Lock(redis_t, "%s-wlock" % lock_name, id=CacheUtils.LOCK_ID)
|
||||
current_cache_wlock = redis_lock.Lock(redis_t, f"{lock_name}-wlock", id=CacheUtils.LOCK_ID)
|
||||
CacheUtils.acquire(current_cache_wlock, lock_name)
|
||||
try:
|
||||
yield
|
||||
@@ -297,6 +299,30 @@ class BaseProviderCache:
|
||||
def __getattr__(self, attr):
|
||||
return getattr(self.provider, attr)
|
||||
|
||||
@staticmethod
|
||||
def check_cache_exists(cache_path: Union[str, Path], suffix_list: Iterable = (".index", ".meta")) -> bool:
|
||||
cache_path = Path(cache_path)
|
||||
for p in [cache_path] + [cache_path.with_suffix(_s) for _s in suffix_list]:
|
||||
if not p.exists():
|
||||
return False
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def clear_cache(cache_path: Union[str, Path]):
|
||||
for p in [
|
||||
cache_path,
|
||||
cache_path.with_suffix(".meta"),
|
||||
cache_path.with_suffix(".index"),
|
||||
]:
|
||||
if p.exists():
|
||||
p.unlink()
|
||||
|
||||
@staticmethod
|
||||
def get_cache_dir(dir_name: str, freq: str = None) -> Path:
|
||||
cache_dir = Path(C.dpm.get_data_uri(freq)).joinpath(dir_name)
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_dir
|
||||
|
||||
|
||||
class ExpressionCache(BaseProviderCache):
|
||||
"""Expression cache mechanism base class.
|
||||
@@ -330,15 +356,16 @@ class ExpressionCache(BaseProviderCache):
|
||||
"""
|
||||
raise NotImplementedError("Implement this method if you want to use expression cache")
|
||||
|
||||
def update(self, cache_uri):
|
||||
def update(self, cache_uri: Union[str, Path], freq: str = "day"):
|
||||
"""Update expression cache to latest calendar.
|
||||
|
||||
Overide this method to define how to update expression cache corresponding to users' own cache mechanism.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
cache_uri : str
|
||||
cache_uri : str or Path
|
||||
the complete uri of expression cache file (include dir path).
|
||||
freq : str
|
||||
|
||||
Returns
|
||||
-------
|
||||
@@ -358,7 +385,9 @@ class DatasetCache(BaseProviderCache):
|
||||
|
||||
HDF_KEY = "df"
|
||||
|
||||
def dataset(self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=1):
|
||||
def dataset(
|
||||
self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=1, inst_processors=[]
|
||||
):
|
||||
"""Get feature dataset.
|
||||
|
||||
.. note:: Same interface as `dataset` method in dataset provider
|
||||
@@ -369,13 +398,19 @@ class DatasetCache(BaseProviderCache):
|
||||
"""
|
||||
if disk_cache == 0:
|
||||
# skip cache
|
||||
return self.provider.dataset(instruments, fields, start_time, end_time, freq)
|
||||
return self.provider.dataset(
|
||||
instruments, fields, start_time, end_time, freq, inst_processors=inst_processors
|
||||
)
|
||||
else:
|
||||
# use and replace cache
|
||||
try:
|
||||
return self._dataset(instruments, fields, start_time, end_time, freq, disk_cache)
|
||||
return self._dataset(
|
||||
instruments, fields, start_time, end_time, freq, disk_cache, inst_processors=inst_processors
|
||||
)
|
||||
except NotImplementedError:
|
||||
return self.provider.dataset(instruments, fields, start_time, end_time, freq)
|
||||
return self.provider.dataset(
|
||||
instruments, fields, start_time, end_time, freq, inst_processors=inst_processors
|
||||
)
|
||||
|
||||
def _uri(self, instruments, fields, start_time, end_time, freq, **kwargs):
|
||||
"""Get dataset cache file uri.
|
||||
@@ -384,14 +419,18 @@ class DatasetCache(BaseProviderCache):
|
||||
"""
|
||||
raise NotImplementedError("Implement this function to match your own cache mechanism")
|
||||
|
||||
def _dataset(self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=1):
|
||||
def _dataset(
|
||||
self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=1, inst_processors=[]
|
||||
):
|
||||
"""Get feature dataset using cache.
|
||||
|
||||
Override this method to define how to get feature dataset corresponding to users' own cache mechanism.
|
||||
"""
|
||||
raise NotImplementedError("Implement this method if you want to use dataset feature cache")
|
||||
|
||||
def _dataset_uri(self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=1):
|
||||
def _dataset_uri(
|
||||
self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=1, inst_processors=[]
|
||||
):
|
||||
"""Get a uri of feature dataset using cache.
|
||||
specially:
|
||||
disk_cache=1 means using data set cache and return the uri of cache file.
|
||||
@@ -403,15 +442,16 @@ class DatasetCache(BaseProviderCache):
|
||||
"Implement this method if you want to use dataset feature cache as a cache file for client"
|
||||
)
|
||||
|
||||
def update(self, cache_uri):
|
||||
def update(self, cache_uri: Union[str, Path], freq: str = "day"):
|
||||
"""Update dataset cache to latest calendar.
|
||||
|
||||
Overide this method to define how to update dataset cache corresponding to users' own cache mechanism.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
cache_uri : str
|
||||
cache_uri : str or Path
|
||||
the complete uri of dataset cache file (include dir path).
|
||||
freq : str
|
||||
|
||||
Returns
|
||||
-------
|
||||
@@ -452,25 +492,19 @@ class DiskExpressionCache(ExpressionCache):
|
||||
self.r = get_redis_connection()
|
||||
# remote==True means client is using this module, writing behaviour will not be allowed.
|
||||
self.remote = kwargs.get("remote", False)
|
||||
self.expr_cache_path = os.path.join(C.get_data_path(), C.features_cache_dir_name)
|
||||
os.makedirs(self.expr_cache_path, exist_ok=True)
|
||||
|
||||
def get_cache_dir(self, freq: str = None) -> Path:
|
||||
return super(DiskExpressionCache, self).get_cache_dir(C.features_cache_dir_name, freq)
|
||||
|
||||
def _uri(self, instrument, field, start_time, end_time, freq):
|
||||
field = remove_fields_space(field)
|
||||
instrument = str(instrument).lower()
|
||||
return hash_args(instrument, field, freq)
|
||||
|
||||
@staticmethod
|
||||
def check_cache_exists(cache_path):
|
||||
for p in [cache_path, cache_path + ".meta"]:
|
||||
if not Path(p).exists():
|
||||
return False
|
||||
return True
|
||||
|
||||
def _expression(self, instrument, field, start_time=None, end_time=None, freq="day"):
|
||||
_cache_uri = self._uri(instrument=instrument, field=field, start_time=None, end_time=None, freq=freq)
|
||||
_instrument_dir = os.path.join(self.expr_cache_path, instrument.lower())
|
||||
cache_path = os.path.join(_instrument_dir, _cache_uri)
|
||||
_instrument_dir = self.get_cache_dir(freq).joinpath(instrument.lower())
|
||||
cache_path = _instrument_dir.joinpath(_cache_uri)
|
||||
# get calendar
|
||||
from .data import Cal
|
||||
|
||||
@@ -478,7 +512,7 @@ class DiskExpressionCache(ExpressionCache):
|
||||
|
||||
_, _, start_index, end_index = Cal.locate_index(start_time, end_time, freq, future=False)
|
||||
|
||||
if self.check_cache_exists(cache_path):
|
||||
if self.check_cache_exists(cache_path, suffix_list=[".meta"]):
|
||||
"""
|
||||
In most cases, we do not need reader_lock.
|
||||
Because updating data is a small probability event compare to reading data.
|
||||
@@ -502,8 +536,7 @@ class DiskExpressionCache(ExpressionCache):
|
||||
# normalize field
|
||||
field = remove_fields_space(field)
|
||||
# cache unavailable, generate the cache
|
||||
if not os.path.exists(_instrument_dir):
|
||||
os.makedirs(_instrument_dir, exist_ok=True)
|
||||
_instrument_dir.mkdir(parents=True, exist_ok=True)
|
||||
if not isinstance(eval(parse_field(field)), Feature):
|
||||
# When the expression is not a raw feature
|
||||
# generate expression cache if the feature is not a Feature
|
||||
@@ -511,7 +544,7 @@ class DiskExpressionCache(ExpressionCache):
|
||||
series = self.provider.expression(instrument, field, _calendar[0], _calendar[-1], freq)
|
||||
if not series.empty:
|
||||
# This expresion is empty, we don't generate any cache for it.
|
||||
with CacheUtils.writer_lock(self.r, "expression-%s" % _cache_uri):
|
||||
with CacheUtils.writer_lock(self.r, f"{str(C.dpm.get_data_uri(freq))}:expression-{_cache_uri}"):
|
||||
self.gen_expression_cache(
|
||||
expression_data=series,
|
||||
cache_path=cache_path,
|
||||
@@ -527,14 +560,6 @@ class DiskExpressionCache(ExpressionCache):
|
||||
# If the expression is a raw feature(such as $close, $open)
|
||||
return self.provider.expression(instrument, field, start_time, end_time, freq)
|
||||
|
||||
@staticmethod
|
||||
def clear_cache(cache_path):
|
||||
meta_path = cache_path + ".meta"
|
||||
for p in [cache_path, meta_path]:
|
||||
p = Path(p)
|
||||
if p.exists():
|
||||
p.unlink()
|
||||
|
||||
def gen_expression_cache(self, expression_data, cache_path, instrument, field, freq, last_update):
|
||||
"""use bin file to save like feature-data."""
|
||||
# Make sure the cache runs right when the directory is deleted
|
||||
@@ -544,27 +569,28 @@ class DiskExpressionCache(ExpressionCache):
|
||||
"meta": {"last_visit": time.time(), "visits": 1},
|
||||
}
|
||||
self.logger.debug(f"generating expression cache: {meta}")
|
||||
os.makedirs(self.expr_cache_path, exist_ok=True)
|
||||
self.clear_cache(cache_path)
|
||||
meta_path = cache_path + ".meta"
|
||||
meta_path = cache_path.with_suffix(".meta")
|
||||
|
||||
with open(meta_path, "wb") as f:
|
||||
with meta_path.open("wb") as f:
|
||||
pickle.dump(meta, f)
|
||||
os.chmod(meta_path, stat.S_IRWXU | stat.S_IRGRP | stat.S_IROTH)
|
||||
meta_path.chmod(stat.S_IRWXU | stat.S_IRGRP | stat.S_IROTH)
|
||||
df = expression_data.to_frame()
|
||||
|
||||
r = np.hstack([df.index[0], expression_data]).astype("<f")
|
||||
r.tofile(str(cache_path))
|
||||
|
||||
def update(self, sid, cache_uri):
|
||||
cp_cache_uri = os.path.join(self.expr_cache_path, sid, cache_uri)
|
||||
if not self.check_cache_exists(cp_cache_uri):
|
||||
def update(self, sid, cache_uri, freq: str = "day"):
|
||||
|
||||
cp_cache_uri = self.get_cache_dir(freq).joinpath(sid).joinpath(cache_uri)
|
||||
meta_path = cp_cache_uri.with_suffix(".meta")
|
||||
if not self.check_cache_exists(cp_cache_uri, suffix_list=[".meta"]):
|
||||
self.logger.info(f"The cache {cp_cache_uri} has corrupted. It will be removed")
|
||||
self.clear_cache(cp_cache_uri)
|
||||
return 2
|
||||
|
||||
with CacheUtils.writer_lock(self.r, "expression-%s" % cache_uri):
|
||||
with open(cp_cache_uri + ".meta", "rb") as f:
|
||||
with CacheUtils.writer_lock(self.r, f"{str(C.dpm.get_data_uri())}:expression-{cache_uri}"):
|
||||
with meta_path.open("rb") as f:
|
||||
d = pickle.load(f)
|
||||
instrument = d["info"]["instrument"]
|
||||
field = d["info"]["field"]
|
||||
@@ -611,7 +637,7 @@ class DiskExpressionCache(ExpressionCache):
|
||||
f.write(data)
|
||||
# update meta file
|
||||
d["info"]["last_update"] = str(new_calendar[-1])
|
||||
with open(cp_cache_uri + ".meta", "wb") as f:
|
||||
with meta_path.open("wb") as f:
|
||||
pickle.dump(d, f)
|
||||
return 0
|
||||
|
||||
@@ -623,22 +649,16 @@ class DiskDatasetCache(DatasetCache):
|
||||
super(DiskDatasetCache, self).__init__(provider)
|
||||
self.r = get_redis_connection()
|
||||
self.remote = kwargs.get("remote", False)
|
||||
self.dtst_cache_path = os.path.join(C.get_data_path(), C.dataset_cache_dir_name)
|
||||
os.makedirs(self.dtst_cache_path, exist_ok=True)
|
||||
|
||||
@staticmethod
|
||||
def _uri(instruments, fields, start_time, end_time, freq, disk_cache=1, **kwargs):
|
||||
return hash_args(*DatasetCache.normalize_uri_args(instruments, fields, freq), disk_cache)
|
||||
def _uri(instruments, fields, start_time, end_time, freq, disk_cache=1, inst_processors=[], **kwargs):
|
||||
return hash_args(*DatasetCache.normalize_uri_args(instruments, fields, freq), disk_cache, inst_processors)
|
||||
|
||||
@staticmethod
|
||||
def check_cache_exists(cache_path):
|
||||
for p in [cache_path, cache_path + ".index", cache_path + ".meta"]:
|
||||
if not Path(p).exists():
|
||||
return False
|
||||
return True
|
||||
def get_cache_dir(self, freq: str = None) -> Path:
|
||||
return super(DiskDatasetCache, self).get_cache_dir(C.dataset_cache_dir_name, freq)
|
||||
|
||||
@classmethod
|
||||
def read_data_from_cache(cls, cache_path, start_time, end_time, fields):
|
||||
def read_data_from_cache(cls, cache_path: Union[str, Path], start_time, end_time, fields):
|
||||
"""read_cache_from
|
||||
|
||||
This function can read data from the disk cache dataset
|
||||
@@ -671,17 +691,32 @@ class DiskDatasetCache(DatasetCache):
|
||||
df = pd.DataFrame(columns=fields)
|
||||
return df
|
||||
|
||||
def _dataset(self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=0):
|
||||
def _dataset(
|
||||
self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=0, inst_processors=[]
|
||||
):
|
||||
|
||||
if disk_cache == 0:
|
||||
# In this case, data_set cache is configured but will not be used.
|
||||
return self.provider.dataset(instruments, fields, start_time, end_time, freq)
|
||||
|
||||
return self.provider.dataset(
|
||||
instruments, fields, start_time, end_time, freq, inst_processors=inst_processors
|
||||
)
|
||||
# FIXME: The cache after resample, when read again and intercepted with end_time, results in incomplete data date
|
||||
if inst_processors:
|
||||
raise ValueError(
|
||||
f"{self.__class__.__name__} does not support inst_processor. "
|
||||
f"Please use `D.features(disk_cache=0)` or `qlib.init(dataset_cache=None)`"
|
||||
)
|
||||
_cache_uri = self._uri(
|
||||
instruments=instruments, fields=fields, start_time=None, end_time=None, freq=freq, disk_cache=disk_cache
|
||||
instruments=instruments,
|
||||
fields=fields,
|
||||
start_time=None,
|
||||
end_time=None,
|
||||
freq=freq,
|
||||
disk_cache=disk_cache,
|
||||
inst_processors=inst_processors,
|
||||
)
|
||||
|
||||
cache_path = os.path.join(self.dtst_cache_path, _cache_uri)
|
||||
cache_path = self.get_cache_dir(freq).joinpath(_cache_uri)
|
||||
|
||||
features = pd.DataFrame()
|
||||
gen_flag = False
|
||||
@@ -689,7 +724,7 @@ class DiskDatasetCache(DatasetCache):
|
||||
if self.check_cache_exists(cache_path):
|
||||
if disk_cache == 1:
|
||||
# use cache
|
||||
with CacheUtils.reader_lock(self.r, "dataset-%s" % _cache_uri):
|
||||
with CacheUtils.reader_lock(self.r, f"{str(C.dpm.get_data_uri(freq))}:dataset-{_cache_uri}"):
|
||||
CacheUtils.visit(cache_path)
|
||||
features = self.read_data_from_cache(cache_path, start_time, end_time, fields)
|
||||
elif disk_cache == 2:
|
||||
@@ -699,15 +734,21 @@ class DiskDatasetCache(DatasetCache):
|
||||
|
||||
if gen_flag:
|
||||
# cache unavailable, generate the cache
|
||||
with CacheUtils.writer_lock(self.r, "dataset-%s" % _cache_uri):
|
||||
with CacheUtils.writer_lock(self.r, f"{str(C.dpm.get_data_uri(freq))}:dataset-{_cache_uri}"):
|
||||
features = self.gen_dataset_cache(
|
||||
cache_path=cache_path, instruments=instruments, fields=fields, freq=freq
|
||||
cache_path=cache_path,
|
||||
instruments=instruments,
|
||||
fields=fields,
|
||||
freq=freq,
|
||||
inst_processors=inst_processors,
|
||||
)
|
||||
if not features.empty:
|
||||
features = features.sort_index().loc(axis=0)[:, start_time:end_time]
|
||||
return features
|
||||
|
||||
def _dataset_uri(self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=0):
|
||||
def _dataset_uri(
|
||||
self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=0, inst_processors=[]
|
||||
):
|
||||
if disk_cache == 0:
|
||||
# In this case, server only checks the expression cache.
|
||||
# The client will load the cache data by itself.
|
||||
@@ -715,21 +756,38 @@ class DiskDatasetCache(DatasetCache):
|
||||
|
||||
LocalDatasetProvider.multi_cache_walker(instruments, fields, start_time, end_time, freq)
|
||||
return ""
|
||||
|
||||
# FIXME: The cache after resample, when read again and intercepted with end_time, results in incomplete data date
|
||||
if inst_processors:
|
||||
raise ValueError(
|
||||
f"{self.__class__.__name__} does not support inst_processor. "
|
||||
f"Please use `D.features(disk_cache=0)` or `qlib.init(dataset_cache=None)`"
|
||||
)
|
||||
_cache_uri = self._uri(
|
||||
instruments=instruments, fields=fields, start_time=None, end_time=None, freq=freq, disk_cache=disk_cache
|
||||
instruments=instruments,
|
||||
fields=fields,
|
||||
start_time=None,
|
||||
end_time=None,
|
||||
freq=freq,
|
||||
disk_cache=disk_cache,
|
||||
inst_processors=inst_processors,
|
||||
)
|
||||
cache_path = os.path.join(self.dtst_cache_path, _cache_uri)
|
||||
cache_path = self.get_cache_dir(freq).joinpath(_cache_uri)
|
||||
|
||||
if self.check_cache_exists(cache_path):
|
||||
self.logger.debug(f"The cache dataset has already existed {cache_path}. Return the uri directly")
|
||||
with CacheUtils.reader_lock(self.r, "dataset-%s" % _cache_uri):
|
||||
with CacheUtils.reader_lock(self.r, f"{str(C.dpm.get_data_uri(freq))}:dataset-{_cache_uri}"):
|
||||
CacheUtils.visit(cache_path)
|
||||
return _cache_uri
|
||||
else:
|
||||
# cache unavailable, generate the cache
|
||||
with CacheUtils.writer_lock(self.r, "dataset-%s" % _cache_uri):
|
||||
self.gen_dataset_cache(cache_path=cache_path, instruments=instruments, fields=fields, freq=freq)
|
||||
with CacheUtils.writer_lock(self.r, f"{str(C.dpm.get_data_uri(freq))}:dataset-{_cache_uri}"):
|
||||
self.gen_dataset_cache(
|
||||
cache_path=cache_path,
|
||||
instruments=instruments,
|
||||
fields=fields,
|
||||
freq=freq,
|
||||
inst_processors=inst_processors,
|
||||
)
|
||||
return _cache_uri
|
||||
|
||||
class IndexManager:
|
||||
@@ -740,8 +798,9 @@ class DiskDatasetCache(DatasetCache):
|
||||
|
||||
KEY = "df"
|
||||
|
||||
def __init__(self, cache_path):
|
||||
self.index_path = cache_path + ".index"
|
||||
def __init__(self, cache_path: Union[str, Path]):
|
||||
|
||||
self.index_path = cache_path.with_suffix(".index")
|
||||
self._data = None
|
||||
self.logger = get_module_logger(self.__class__.__name__)
|
||||
|
||||
@@ -757,7 +816,7 @@ class DiskDatasetCache(DatasetCache):
|
||||
self._data.sort_index(inplace=True)
|
||||
self._data.to_hdf(self.index_path, key=self.KEY, mode="w", format="table")
|
||||
# The index should be readable for all users
|
||||
os.chmod(self.index_path, stat.S_IRWXU | stat.S_IRGRP | stat.S_IROTH)
|
||||
self.index_path.chmod(stat.S_IRWXU | stat.S_IRGRP | stat.S_IROTH)
|
||||
|
||||
def sync_from_disk(self):
|
||||
# The file will not be closed directly if we read_hdf from the disk directly
|
||||
@@ -795,15 +854,7 @@ class DiskDatasetCache(DatasetCache):
|
||||
index_data += start_index
|
||||
return index_data
|
||||
|
||||
@staticmethod
|
||||
def clear_cache(cache_path):
|
||||
meta_path = cache_path + ".meta"
|
||||
for p in [cache_path, meta_path, cache_path + ".index", cache_path + ".data"]:
|
||||
p = Path(p)
|
||||
if p.exists():
|
||||
p.unlink()
|
||||
|
||||
def gen_dataset_cache(self, cache_path, instruments, fields, freq):
|
||||
def gen_dataset_cache(self, cache_path: Union[str, Path], instruments, fields, freq, inst_processors=[]):
|
||||
"""gen_dataset_cache
|
||||
|
||||
.. note:: This function does not consider the cache read write lock. Please
|
||||
@@ -838,20 +889,23 @@ class DiskDatasetCache(DatasetCache):
|
||||
:param instruments: The instruments to store the cache.
|
||||
:param fields: The fields to store the cache.
|
||||
:param freq: The freq to store the cache.
|
||||
:param inst_processors: Instrument processors.
|
||||
|
||||
:return type pd.DataFrame; The fields of the returned DataFrame are consistent with the parameters of the function.
|
||||
"""
|
||||
# get calendar
|
||||
from .data import Cal
|
||||
|
||||
cache_path = Path(cache_path)
|
||||
_calendar = Cal.calendar(freq=freq)
|
||||
self.logger.debug(f"Generating dataset cache {cache_path}")
|
||||
# Make sure the cache runs right when the directory is deleted
|
||||
# while running
|
||||
os.makedirs(self.dtst_cache_path, exist_ok=True)
|
||||
self.clear_cache(cache_path)
|
||||
|
||||
features = self.provider.dataset(instruments, fields, _calendar[0], _calendar[-1], freq)
|
||||
features = self.provider.dataset(
|
||||
instruments, fields, _calendar[0], _calendar[-1], freq, inst_processors=inst_processors
|
||||
)
|
||||
|
||||
if features.empty:
|
||||
return features
|
||||
@@ -860,7 +914,7 @@ class DiskDatasetCache(DatasetCache):
|
||||
features = features.swaplevel("instrument", "datetime").sort_index()
|
||||
|
||||
# write cache data
|
||||
with pd.HDFStore(cache_path + ".data") as store:
|
||||
with pd.HDFStore(str(cache_path.with_suffix(".data"))) as store:
|
||||
cache_to_orig_map = dict(zip(remove_fields_space(features.columns), features.columns))
|
||||
orig_to_cache_map = dict(zip(features.columns, remove_fields_space(features.columns)))
|
||||
cache_features = features[list(cache_to_orig_map.values())].rename(columns=orig_to_cache_map)
|
||||
@@ -876,12 +930,13 @@ class DiskDatasetCache(DatasetCache):
|
||||
"fields": cache_columns,
|
||||
"freq": freq,
|
||||
"last_update": str(_calendar[-1]), # The last_update to store the cache
|
||||
"inst_processors": inst_processors, # The last_update to store the cache
|
||||
},
|
||||
"meta": {"last_visit": time.time(), "visits": 1},
|
||||
}
|
||||
with open(cache_path + ".meta", "wb") as f:
|
||||
with cache_path.with_suffix(".meta").open("wb") as f:
|
||||
pickle.dump(meta, f)
|
||||
os.chmod(cache_path + ".meta", stat.S_IRWXU | stat.S_IRGRP | stat.S_IROTH)
|
||||
cache_path.with_suffix(".meta").chmod(stat.S_IRWXU | stat.S_IRGRP | stat.S_IROTH)
|
||||
# write index file
|
||||
im = DiskDatasetCache.IndexManager(cache_path)
|
||||
index_data = im.build_index_from_data(features)
|
||||
@@ -890,26 +945,27 @@ class DiskDatasetCache(DatasetCache):
|
||||
# rename the file after the cache has been generated
|
||||
# this doesn't work well on windows, but our server won't use windows
|
||||
# temporarily
|
||||
os.replace(cache_path + ".data", cache_path)
|
||||
cache_path.with_suffix(".data").rename(cache_path)
|
||||
# the fields of the cached features are converted to the original fields
|
||||
return features.swaplevel("datetime", "instrument")
|
||||
|
||||
def update(self, cache_uri):
|
||||
cp_cache_uri = os.path.join(self.dtst_cache_path, cache_uri)
|
||||
|
||||
def update(self, cache_uri, freq: str = "day"):
|
||||
cp_cache_uri = self.get_cache_dir(freq).joinpath(cache_uri)
|
||||
meta_path = cp_cache_uri.with_suffix(".meta")
|
||||
if not self.check_cache_exists(cp_cache_uri):
|
||||
self.logger.info(f"The cache {cp_cache_uri} has corrupted. It will be removed")
|
||||
self.clear_cache(cp_cache_uri)
|
||||
return 2
|
||||
|
||||
im = DiskDatasetCache.IndexManager(cp_cache_uri)
|
||||
with CacheUtils.writer_lock(self.r, "dataset-%s" % cache_uri):
|
||||
with open(cp_cache_uri + ".meta", "rb") as f:
|
||||
with CacheUtils.writer_lock(self.r, f"{str(C.dpm.get_data_uri())}:dataset-{cache_uri}"):
|
||||
with meta_path.open("rb") as f:
|
||||
d = pickle.load(f)
|
||||
instruments = d["info"]["instruments"]
|
||||
fields = d["info"]["fields"]
|
||||
freq = d["info"]["freq"]
|
||||
last_update_time = d["info"]["last_update"]
|
||||
inst_processors = d["info"]["inst_processors"]
|
||||
index_data = im.get_index()
|
||||
|
||||
self.logger.debug("Updating dataset: {}".format(d))
|
||||
@@ -960,7 +1016,12 @@ class DiskDatasetCache(DatasetCache):
|
||||
)
|
||||
|
||||
data = self.provider.dataset(
|
||||
instruments, fields, whole_calendar[current_index - rm_n_period], new_calendar[-1], freq
|
||||
instruments,
|
||||
fields,
|
||||
whole_calendar[current_index - rm_n_period],
|
||||
new_calendar[-1],
|
||||
freq,
|
||||
inst_processors=inst_processors,
|
||||
)
|
||||
|
||||
if not data.empty:
|
||||
@@ -995,7 +1056,7 @@ class DiskDatasetCache(DatasetCache):
|
||||
|
||||
# update meta file
|
||||
d["info"]["last_update"] = str(new_calendar[-1])
|
||||
with open(cp_cache_uri + ".meta", "wb") as f:
|
||||
with meta_path.open("wb") as f:
|
||||
pickle.dump(d, f)
|
||||
return 0
|
||||
|
||||
@@ -1006,26 +1067,36 @@ class SimpleDatasetCache(DatasetCache):
|
||||
def __init__(self, provider):
|
||||
super(SimpleDatasetCache, self).__init__(provider)
|
||||
try:
|
||||
self.local_cache_path = C["local_cache_path"]
|
||||
except KeyError as e:
|
||||
self.local_cache_path: Path = Path(C["local_cache_path"]).expanduser().resolve()
|
||||
except (KeyError, TypeError) as e:
|
||||
self.logger.error("Assign a local_cache_path in config if you want to use this cache mechanism")
|
||||
raise
|
||||
self.logger.info(
|
||||
f"DatasetCache directory: {self.local_cache_path}, "
|
||||
f"modify the cache directory via the local_cache_path in the config"
|
||||
)
|
||||
|
||||
def _uri(self, instruments, fields, start_time, end_time, freq, disk_cache=1, **kwargs):
|
||||
def _uri(self, instruments, fields, start_time, end_time, freq, disk_cache=1, inst_processors=[], **kwargs):
|
||||
instruments, fields, freq = self.normalize_uri_args(instruments, fields, freq)
|
||||
local_cache_path = str(Path(self.local_cache_path).expanduser().resolve())
|
||||
return hash_args(instruments, fields, start_time, end_time, freq, disk_cache, local_cache_path)
|
||||
return hash_args(
|
||||
instruments, fields, start_time, end_time, freq, disk_cache, str(self.local_cache_path), inst_processors
|
||||
)
|
||||
|
||||
def _dataset(self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=1):
|
||||
def _dataset(
|
||||
self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=1, inst_processors=[]
|
||||
):
|
||||
if disk_cache == 0:
|
||||
# In this case, data_set cache is configured but will not be used.
|
||||
return self.provider.dataset(instruments, fields, start_time, end_time, freq)
|
||||
os.makedirs(os.path.expanduser(self.local_cache_path), exist_ok=True)
|
||||
cache_file = os.path.join(
|
||||
self.local_cache_path, self._uri(instruments, fields, start_time, end_time, freq, disk_cache=disk_cache)
|
||||
self.local_cache_path.mkdir(exist_ok=True, parents=True)
|
||||
cache_file = self.local_cache_path.joinpath(
|
||||
self._uri(
|
||||
instruments, fields, start_time, end_time, freq, disk_cache=disk_cache, inst_processors=inst_processors
|
||||
)
|
||||
)
|
||||
gen_flag = False
|
||||
|
||||
if os.path.exists(cache_file):
|
||||
if cache_file.exists():
|
||||
if disk_cache == 1:
|
||||
# use cache
|
||||
df = pd.read_pickle(cache_file)
|
||||
@@ -1037,7 +1108,9 @@ class SimpleDatasetCache(DatasetCache):
|
||||
gen_flag = True
|
||||
|
||||
if gen_flag:
|
||||
data = self.provider.dataset(instruments, normalize_cache_fields(fields), start_time, end_time, freq)
|
||||
data = self.provider.dataset(
|
||||
instruments, normalize_cache_fields(fields), start_time, end_time, freq, inst_processors=inst_processors
|
||||
)
|
||||
data.to_pickle(cache_file)
|
||||
return self.cache_to_origin_data(data, fields)
|
||||
|
||||
@@ -1045,26 +1118,53 @@ class SimpleDatasetCache(DatasetCache):
|
||||
class DatasetURICache(DatasetCache):
|
||||
"""Prepared cache mechanism for server."""
|
||||
|
||||
def _uri(self, instruments, fields, start_time, end_time, freq, disk_cache=1, **kwargs):
|
||||
return hash_args(*self.normalize_uri_args(instruments, fields, freq), disk_cache)
|
||||
def _uri(self, instruments, fields, start_time, end_time, freq, disk_cache=1, inst_processors=[], **kwargs):
|
||||
return hash_args(*self.normalize_uri_args(instruments, fields, freq), disk_cache, inst_processors)
|
||||
|
||||
def dataset(self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=0):
|
||||
def dataset(
|
||||
self, instruments, fields, start_time=None, end_time=None, freq="day", disk_cache=0, inst_processors=[]
|
||||
):
|
||||
|
||||
if "local" in C.dataset_provider.lower():
|
||||
# use LocalDatasetProvider
|
||||
return self.provider.dataset(instruments, fields, start_time, end_time, freq)
|
||||
return self.provider.dataset(
|
||||
instruments, fields, start_time, end_time, freq, inst_processors=inst_processors
|
||||
)
|
||||
|
||||
if disk_cache == 0:
|
||||
# do not use data_set cache, load data from remote expression cache directly
|
||||
return self.provider.dataset(instruments, fields, start_time, end_time, freq, disk_cache, return_uri=False)
|
||||
|
||||
return self.provider.dataset(
|
||||
instruments,
|
||||
fields,
|
||||
start_time,
|
||||
end_time,
|
||||
freq,
|
||||
disk_cache,
|
||||
return_uri=False,
|
||||
inst_processors=inst_processors,
|
||||
)
|
||||
# FIXME: The cache after resample, when read again and intercepted with end_time, results in incomplete data date
|
||||
if inst_processors:
|
||||
raise ValueError(
|
||||
f"{self.__class__.__name__} does not support inst_processor. "
|
||||
f"Please use `D.features(disk_cache=0)` or `qlib.init(dataset_cache=None)`"
|
||||
)
|
||||
# use ClientDatasetProvider
|
||||
feature_uri = self._uri(instruments, fields, None, None, freq, disk_cache=disk_cache)
|
||||
feature_uri = self._uri(
|
||||
instruments, fields, None, None, freq, disk_cache=disk_cache, inst_processors=inst_processors
|
||||
)
|
||||
value, expire = MemCacheExpire.get_cache(H["f"], feature_uri)
|
||||
mnt_feature_uri = os.path.join(C.get_data_path(), C.dataset_cache_dir_name, feature_uri)
|
||||
if value is None or expire or not os.path.exists(mnt_feature_uri):
|
||||
mnt_feature_uri = C.dpm.get_data_uri(freq).joinpath(C.dataset_cache_dir_name).joinpath(feature_uri)
|
||||
if value is None or expire or not mnt_feature_uri.exists():
|
||||
df, uri = self.provider.dataset(
|
||||
instruments, fields, start_time, end_time, freq, disk_cache, return_uri=True
|
||||
instruments,
|
||||
fields,
|
||||
start_time,
|
||||
end_time,
|
||||
freq,
|
||||
disk_cache,
|
||||
return_uri=True,
|
||||
inst_processors=inst_processors,
|
||||
)
|
||||
# cache uri
|
||||
MemCacheExpire.set_cache(H["f"], uri, uri)
|
||||
@@ -1072,7 +1172,6 @@ class DatasetURICache(DatasetCache):
|
||||
# HZ['f'][uri] = df.copy()
|
||||
get_module_logger("cache").debug(f"get feature from {C.dataset_provider}")
|
||||
else:
|
||||
mnt_feature_uri = os.path.join(C.get_data_path(), C.dataset_cache_dir_name, feature_uri)
|
||||
df = DiskDatasetCache.read_data_from_cache(mnt_feature_uri, start_time, end_time, fields)
|
||||
get_module_logger("cache").debug("get feature from uri cache")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user