liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-07 13:00:58 +08:00
Code Activity

Update docs and fix tabnet

Browse Source
This commit is contained in:
Jactus
2020-11-26 00:55:26 +08:00
parent 5be847909f
commit 87cee85cea
27 changed files with 624 additions and 495 deletions
Download Patch File Download Diff File Expand all files Collapse all files

84
qlib/contrib/evaluate.py
View File

@@ -26,9 +26,9 @@ def risk_analysis(r, N=252):
Parameters
----------
r : pandas.Series
daily return series
daily return series.
N: int
scaler for annualizing information_ratio (day: 250, week: 50, month: 12)
scaler for annualizing information_ratio (day: 250, week: 50, month: 12).
"""
mean = r.mean()
std = r.std(ddof=1)
@@ -61,7 +61,7 @@ def get_strategy(
----------
strategy : Strategy()
strategy used in backtest
strategy used in backtest.
topk : int (Default value: 50)
top-N stocks to buy.
margin : int or float(Default value: 0.5)
@@ -73,14 +73,14 @@ def get_strategy(
sell_limit = pred_in_a_day.count() * margin
buffer margin, in single score_mode, continue holding stock if it is in nlargest(sell_limit)
sell_limit should be no less than topk
buffer margin, in single score_mode, continue holding stock if it is in nlargest(sell_limit).
sell_limit should be no less than topk.
n_drop : int
number of stocks to be replaced in each trading date
number of stocks to be replaced in each trading date.
risk_degree: float
0-1, 0.95 for example, use 95% money to trade
0-1, 0.95 for example, use 95% money to trade.
str_type: 'amount', 'weight' or 'dropout'
strategy type: TopkAmountStrategy ,TopkWeightStrategy or TopkDropoutStrategy
strategy type: TopkAmountStrategy ,TopkWeightStrategy or TopkDropoutStrategy.
Returns
-------
@@ -126,21 +126,21 @@ def get_exchange(
----------
# exchange related arguments
exchange: Exchange()
exchange: Exchange().
subscribe_fields: list
subscribe fields
subscribe fields.
open_cost : float
open transaction cost
open transaction cost.
close_cost : float
close transaction cost
close transaction cost.
min_cost : float
min transaction cost
min transaction cost.
trade_unit : int
100 for China A
100 for China A.
deal_price: str
dealing price type: 'close', 'open', 'vwap'
dealing price type: 'close', 'open', 'vwap'.
limit_threshold : float
limit move 0.1 (10%) for example, long and short with same limit
limit move 0.1 (10%) for example, long and short with same limit.
extract_codes: bool
will we pass the codes extracted from the pred to the exchange.
NOTE: This will be faster with offline qlib.
@@ -193,20 +193,20 @@ def backtest(pred, account=1e9, shift=1, benchmark="SH000905", verbose=True, **k
- **backtest workflow related or commmon arguments**
pred : pandas.DataFrame
predict should has <datetime, instrument> index and one `score` column
predict should has <datetime, instrument> index and one `score` column.
account : float
init account value
init account value.
shift : int
whether to shift prediction by one day
whether to shift prediction by one day.
benchmark : str
benchmark code, default is SH000905 CSI 500
benchmark code, default is SH000905 CSI 500.
verbose : bool
whether to print log
whether to print log.
- **strategy related arguments**
strategy : Strategy()
strategy used in backtest
strategy used in backtest.
topk : int (Default value: 50)
top-N stocks to buy.
margin : int or float(Default value: 0.5)
@@ -218,33 +218,33 @@ def backtest(pred, account=1e9, shift=1, benchmark="SH000905", verbose=True, **k
sell_limit = pred_in_a_day.count() * margin
buffer margin, in single score_mode, continue holding stock if it is in nlargest(sell_limit)
sell_limit should be no less than topk
buffer margin, in single score_mode, continue holding stock if it is in nlargest(sell_limit).
sell_limit should be no less than topk.
n_drop : int
number of stocks to be replaced in each trading date
number of stocks to be replaced in each trading date.
risk_degree: float
0-1, 0.95 for example, use 95% money to trade
0-1, 0.95 for example, use 95% money to trade.
str_type: 'amount', 'weight' or 'dropout'
strategy type: TopkAmountStrategy ,TopkWeightStrategy or TopkDropoutStrategy
strategy type: TopkAmountStrategy ,TopkWeightStrategy or TopkDropoutStrategy.
- **exchange related arguments**
exchange: Exchange()
pass the exchange for speeding up.
subscribe_fields: list
subscribe fields
subscribe fields.
open_cost : float
open transaction cost. The default value is 0.002(0.2%).
close_cost : float
close transaction cost. The default value is 0.002(0.2%).
min_cost : float
min transaction cost
min transaction cost.
trade_unit : int
100 for China A
100 for China A.
deal_price: str
dealing price type: 'close', 'open', 'vwap'
dealing price type: 'close', 'open', 'vwap'.
limit_threshold : float
limit move 0.1 (10%) for example, long and short with same limit
limit move 0.1 (10%) for example, long and short with same limit.
extract_codes: bool
will we pass the codes extracted from the pred to the exchange.
@@ -291,17 +291,17 @@ def long_short_backtest(
"""
A backtest for long-short strategy
:param pred: The trading signal produced on day `T`
:param topk: The short topk securities and long topk securities
:param deal_price: The price to deal the trading
:param pred: The trading signal produced on day `T`.
:param topk: The short topk securities and long topk securities.
:param deal_price: The price to deal the trading.
:param shift: Whether to shift prediction by one day. The trading day will be T+1 if shift==1.
:param open_cost: open transaction cost
:param close_cost: close transaction cost
:param trade_unit: 100 for China A
:param limit_threshold: limit move 0.1 (10%) for example, long and short with same limit
:param min_cost: min transaction cost
:param subscribe_fields: subscribe fields
:param extract_codes: bool
:param open_cost: open transaction cost.
:param close_cost: close transaction cost.
:param trade_unit: 100 for China A.
:param limit_threshold: limit move 0.1 (10%) for example, long and short with same limit.
:param min_cost: min transaction cost.
:param subscribe_fields: subscribe fields.
:param extract_codes: bool.
will we pass the codes extracted from the pred to the exchange.
NOTE: This will be faster with offline qlib.
:return: The result of backtest, it is represented by a dict.

16
qlib/contrib/report/analysis_model/analysis_model_performance.py
View File

@@ -252,7 +252,7 @@ def model_performance_graph(
"""Model performance
:param pred_label: index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[score,
label]**. It is usually same as the label of model training(e.g. "Ref($close, -2)/Ref($close, -1) - 1")
label]**. It is usually same as the label of model training(e.g. "Ref($close, -2)/Ref($close, -1) - 1").
.. code-block:: python
@@ -266,13 +266,13 @@ def model_performance_graph(
:param lag: `pred.groupby(level='instrument')['score'].shift(lag)`. It will be only used in the auto-correlation computing.
:param N: group number, default 5
:param reverse: if `True`, `pred['score'] *= -1`
:param rank: if **True**, calculate rank ic
:param graph_names: graph names; default ['cumulative_return', 'pred_ic', 'pred_autocorr', 'pred_turnover']
:param show_notebook: whether to display graphics in notebook, the default is `True`
:param show_nature_day: whether to display the abscissa of non-trading day
:return: if show_notebook is True, display in notebook; else return `plotly.graph_objs.Figure` list
:param N: group number, default 5.
:param reverse: if `True`, `pred['score'] *= -1`.
:param rank: if **True**, calculate rank ic.
:param graph_names: graph names; default ['cumulative_return', 'pred_ic', 'pred_autocorr', 'pred_turnover'].
:param show_notebook: whether to display graphics in notebook, the default is `True`.
:param show_nature_day: whether to display the abscissa of non-trading day.
:return: if show_notebook is True, display in notebook; else return `plotly.graph_objs.Figure` list.
"""
figure_list = []
for graph_name in graph_names:

6
qlib/contrib/report/analysis_position/cumulative_return.py
View File

@@ -218,10 +218,10 @@ def cumulative_return_graph(
Graph desc:
- Axis X: Trading day
- Axis X: Trading day.
- Axis Y:
- Above axis Y: `(((Ref($close, -1)/$close - 1) * weight).sum() / weight.sum()).cumsum()`
- Below axis Y: Daily weight sum
- Above axis Y: `(((Ref($close, -1)/$close - 1) * weight).sum() / weight.sum()).cumsum()`.
- Below axis Y: Daily weight sum.
- In the **sell** graph, `y < 0` stands for profit; in other cases, `y > 0` stands for profit.
- In the **buy_minus_sell** graph, the **y** value of the **weight** graph at the bottom is `buy_weight + sell_weight`.
- In each graph, the **red line** in the histogram on the right represents the average.

6
qlib/contrib/report/analysis_position/rank_label.py
View File

@@ -97,9 +97,9 @@ def rank_label_graph(
qcr.rank_label_graph(positions, features_df, pred_df_dates.min(), pred_df_dates.max())
:param position: position data; **qlib.contrib.backtest.backtest.backtest** result
:param position: position data; **qlib.contrib.backtest.backtest.backtest** result.
:param label_data: **D.features** result; index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[label]**.
**The label T is the change from T to T+1**, it is recommended to use ``close``, example: `D.features(D.instruments('csi500'), ['Ref($close, -1)/$close-1'])`
**The label T is the change from T to T+1**, it is recommended to use ``close``, example: `D.features(D.instruments('csi500'), ['Ref($close, -1)/$close-1'])`.
.. code-block:: python
@@ -115,7 +115,7 @@ def rank_label_graph(
:param start_date: start date
:param end_date: end_date
:param show_notebook: **True** or **False**. If True, show graph in notebook, else return figures
:param show_notebook: **True** or **False**. If True, show graph in notebook, else return figures.
:return:
"""
position = copy.deepcopy(position)

6
qlib/contrib/report/analysis_position/report.py
View File

@@ -186,7 +186,7 @@ def report_graph(report_df: pd.DataFrame, show_notebook: bool = True) -> [list,
qcr.report_graph(report_normal_df)
:param report_df: **df.index.name** must be **date**, **df.columns** must contain **return**, **turnover**, **cost**, **bench**
:param report_df: **df.index.name** must be **date**, **df.columns** must contain **return**, **turnover**, **cost**, **bench**.
.. code-block:: python
@@ -200,8 +200,8 @@ def report_graph(report_df: pd.DataFrame, show_notebook: bool = True) -> [list,
2017-01-10 -0.000416 0.000440 -0.003350 0.208396
:param show_notebook: whether to display graphics in notebook, the default is **True**
:return: if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure** list
:param show_notebook: whether to display graphics in notebook, the default is **True**.
:return: if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure** list.
"""
report_df = report_df.copy()
fig_list = _report_figure(report_df)

6
qlib/contrib/report/analysis_position/risk_analysis.py
View File

@@ -218,7 +218,7 @@ def risk_analysis_graph(
max_drawdown -0.088263
:param report_normal_df: **df.index.name** must be **date**, df.columns must contain **return**, **turnover**, **cost**, **bench**
:param report_normal_df: **df.index.name** must be **date**, df.columns must contain **return**, **turnover**, **cost**, **bench**.
.. code-block:: python
@@ -232,7 +232,7 @@ def risk_analysis_graph(
2017-01-10 -0.000416 0.000440 -0.003350 0.208396
:param report_long_short_df: **df.index.name** must be **date**, df.columns contain **long**, **short**, **long_short**
:param report_long_short_df: **df.index.name** must be **date**, df.columns contain **long**, **short**, **long_short**.
.. code-block:: python
@@ -246,7 +246,7 @@ def risk_analysis_graph(
2017-01-10 0.000824 -0.001944 -0.001120
:param show_notebook: Whether to display graphics in a notebook, default **True**
:param show_notebook: Whether to display graphics in a notebook, default **True**.
If True, show graph in notebook
If False, return graph figure
:return:

6
qlib/contrib/report/analysis_position/score_ic.py
View File

@@ -36,7 +36,7 @@ def score_ic_graph(pred_label: pd.DataFrame, show_notebook: bool = True) -> [lis
analysis_position.score_ic_graph(pred_label)
:param pred_label: index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[score, label]**
:param pred_label: index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns names is **[score, label]**.
.. code-block:: python
@@ -49,8 +49,8 @@ def score_ic_graph(pred_label: pd.DataFrame, show_notebook: bool = True) -> [lis
2017-12-15 -0.102778 -0.102778
:param show_notebook: whether to display graphics in notebook, the default is **True**
:return: if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure** list
:param show_notebook: whether to display graphics in notebook, the default is **True**.
:return: if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure** list.
"""
_ic_df = _get_score_ic(pred_label)
# FIXME: support HIGH-FREQ

70
qlib/contrib/strategy/strategy.py
View File

@@ -31,16 +31,16 @@ class BaseStrategy:
Parameters
-----------
score_series : pd.Seires
stock_id , score
stock_id , score.
current : Position()
current state of position
DO NOT directly change the state of current
current state of position.
DO NOT directly change the state of current.
trade_exchange : Exchange()
trade exchange
trade exchange.
pred_date : pd.Timestamp
predict date
predict date.
trade_date : pd.Timestamp
trade date
trade date.
"""
pass
@@ -49,11 +49,11 @@ class BaseStrategy:
Parameters
-----------
score_series : pd.Series
stock_id , score
stock_id , score.
pred_date : pd.Timestamp
oredict date
oredict date.
trade_date : pd.Timestamp
trade date
trade date.
"""
pass
@@ -67,7 +67,7 @@ class BaseStrategy:
"""
This method only be used in 'online' module, it will generate the *args to initial the strategy.
:param
mode : model used in 'online' module
mode : model used in 'online' module.
"""
return {}
@@ -82,7 +82,7 @@ class StrategyWrapper:
def __init__(self, inner_strategy):
"""__init__
:param inner_strategy: set the inner strategy
:param inner_strategy: set the inner strategy.
"""
self.inner_strategy = inner_strategy
@@ -99,9 +99,9 @@ class AdjustTimer:
Responsible for timing of position adjusting
This is designed as multiple inheritance mechanism due to:
- the is_adjust may need access to the internel state of a strategy
- the is_adjust may need access to the internel state of a strategy.
- it can be reguard as a enhancement to the existing strategy
- it can be reguard as a enhancement to the existing strategy.
"""
# adjust position in each trade date
@@ -146,12 +146,12 @@ class WeightStrategyBase(BaseStrategy, AdjustTimer):
Parameters
-----------
score : pd.Series
pred score for this trade date, index is stock_id, contain 'score' column
pred score for this trade date, index is stock_id, contain 'score' column.
current : Position()
current position
current position.
trade_exchange : Exchange()
trade_date : pd.Timestamp
trade date
trade date.
"""
raise NotImplementedError()
@@ -160,13 +160,13 @@ class WeightStrategyBase(BaseStrategy, AdjustTimer):
Parameters
-----------
score_series : pd.Seires
stock_id , score
stock_id , score.
current : Position()
current of account
current of account.
trade_exchange : Exchange()
exchange
exchange.
trade_date : pd.Timestamp
date
date.
"""
# judge if to adjust
if not self.is_adjust(trade_date):
@@ -206,26 +206,26 @@ class TopkDropoutStrategy(BaseStrategy, ListAdjustTimer):
Parameters
-----------
topk : int
The number of stocks in the portfolio
the number of stocks in the portfolio.
n_drop : int
number of stocks to be replaced in each trading date
number of stocks to be replaced in each trading date.
method_sell : str
dropout method_sell, random/bottom
dropout method_sell, random/bottom.
method_buy : str
dropout method_buy, random/top
dropout method_buy, random/top.
risk_degree : float
position percentage of total value
position percentage of total value.
thresh : int
minimun holding days since last buy singal of the stock
minimun holding days since last buy singal of the stock.
hold_thresh : int
minimum holding days
before sell stock , will check current.get_stock_count(order.stock_id) >= self.thresh
before sell stock , will check current.get_stock_count(order.stock_id) >= self.thresh.
only_tradable : bool
will the strategy only consider the tradable stock when buying and selling.
if only_tradable:
strategy will make buy sell decision without checking the tradable state of the stock
strategy will make buy sell decision without checking the tradable state of the stock.
else:
strategy will make decision with the tradable state of the stock info and avoid buy and sell them
strategy will make decision with the tradable state of the stock info and avoid buy and sell them.
"""
super(TopkDropoutStrategy, self).__init__()
ListAdjustTimer.__init__(self, kwargs.get("adjust_dates", None))
@@ -245,7 +245,7 @@ class TopkDropoutStrategy(BaseStrategy, ListAdjustTimer):
def get_risk_degree(self, date):
"""get_risk_degree
Return the proportion of your total value you will used in investment.
Dynamically risk_degree will result in Market timing
Dynamically risk_degree will result in Market timing.
"""
# It will use 95% amoutn of your total value by default
return self.risk_degree
@@ -257,15 +257,15 @@ class TopkDropoutStrategy(BaseStrategy, ListAdjustTimer):
Parameters
-----------
score_series : pd.Series
stock_id , score
stock_id , score.
current : Position()
current of account
current of account.
trade_exchange : Exchange()
exchange
exchange.
pred_date : pd.Timestamp
predict date
predict date.
trade_date : pd.Timestamp
trade date
trade date.
"""
if not self.is_adjust(trade_date):
return []

8
qlib/data/base.py
View File

@@ -129,13 +129,13 @@ class Expression(abc.ABC):
Parameters
----------
instrument : str
instrument code
instrument code.
start_index : str
feature start index [in calendar]
feature start index [in calendar].
end_index : str
feature end index [in calendar]
feature end index [in calendar].
freq : str
feature frequency
feature frequency.
Returns
----------

40
qlib/data/cache.py
View File

@@ -76,8 +76,8 @@ class MemCache(object):
Parameters
----------
mem_cache_size_limit: cache max size
limit_type: length or sizeof; length(call fun: len), size(call fun: sys.getsizeof)
mem_cache_size_limit: cache max size.
limit_type: length or sizeof; length(call fun: len), size(call fun: sys.getsizeof).
"""
if limit_type not in ["length", "sizeof"]:
raise ValueError(f"limit_type must be length or sizeof, your limit_type is {limit_type}")
@@ -118,9 +118,9 @@ class MemCacheExpire:
def set_cache(mem_cache, key, value):
"""set cache
:param mem_cache: MemCache attribute('c'/'i'/'f')
:param key: cache key
:param value: cache value
:param mem_cache: MemCache attribute('c'/'i'/'f').
:param key: cache key.
:param value: cache value.
"""
mem_cache[key] = value, time.time()
@@ -128,9 +128,9 @@ class MemCacheExpire:
def get_cache(mem_cache, key):
"""get mem cache
:param mem_cache: MemCache attribute('c'/'i'/'f')
:param key: cache key
:return: cache value; if cache not exist, return None
:param mem_cache: MemCache attribute('c'/'i'/'f').
:param key: cache key.
:return: cache value; if cache not exist, return None.
"""
value = None
expire = False
@@ -275,12 +275,12 @@ class ExpressionCache(BaseProviderCache):
Parameters
----------
cache_uri : str
the complete uri of expression cache file (include dir path)
the complete uri of expression cache file (include dir path).
Returns
-------
int
0(successful update)/ 1(no need to update)/ 2(update failure)
0(successful update)/ 1(no need to update)/ 2(update failure).
"""
raise NotImplementedError("Implement this method if you want to make expression cache up to date")
@@ -348,7 +348,7 @@ class DatasetCache(BaseProviderCache):
Parameters
----------
cache_uri : str
the complete uri of dataset cache file (include dir path)
the complete uri of dataset cache file (include dir path).
Returns
-------
@@ -361,9 +361,9 @@ class DatasetCache(BaseProviderCache):
def cache_to_origin_data(data, fields):
"""cache data to origin data
:param data: pd.DataFrame, cache data
:param fields: feature fields
:return: pd.DataFrame
:param data: pd.DataFrame, cache data.
:param fields: feature fields.
:return: pd.DataFrame.
"""
not_space_fields = remove_fields_space(fields)
data = data.loc[:, not_space_fields]
@@ -583,7 +583,7 @@ class DiskDatasetCache(DatasetCache):
:param cache_path:
:param start_time:
:param end_time:
:param fields: The fields order of the dataset cache is sorted. So rearrange the columns to make it consistent
:param fields: The fields order of the dataset cache is sorted. So rearrange the columns to make it consistent.
:return:
"""
@@ -771,12 +771,12 @@ class DiskDatasetCache(DatasetCache):
- This is a hdf file sorted by datetime
:param cache_path: The path to store the cache
:param instruments: The instruments to store the cache
:param fields: The fields to store the cache
:param freq: The freq to store the cache
:param cache_path: The path to store the cache.
:param instruments: The instruments to store the cache.
:param fields: The fields to store the cache.
:param freq: The freq to store the cache.
:return type pd.DataFrame; The fields of the returned DataFrame are consistent with the parameters of the function
:return type pd.DataFrame; The fields of the returned DataFrame are consistent with the parameters of the function.
"""
# get calendar
from .data import Cal

8
qlib/data/client.py
View File

@@ -51,13 +51,13 @@ class Client(object):
Parameters
----------
request_type : str
type of proposed request, 'calendar'/'instrument'/'feature'
type of proposed request, 'calendar'/'instrument'/'feature'.
request_content : dict
records the information of the request
records the information of the request.
msg_proc_func : func
the function to process the message when receiving response, should have arg `*args`
the function to process the message when receiving response, should have arg `*args`.
msg_queue: Queue
The queue to pass the messsage after callback
The queue to pass the messsage after callback.
"""
head_info = {"version": qlib.__version__}

92
qlib/data/data.py
View File

@@ -41,13 +41,13 @@ class CalendarProvider(abc.ABC):
Parameters
----------
start_time : str
start of the time range
start of the time range.
end_time : str
end of the time range
end of the time range.
freq : str
time frequency, available: year/quarter/month/week/day
time frequency, available: year/quarter/month/week/day.
future : bool
whether including future trading day
whether including future trading day.
Returns
----------
@@ -62,24 +62,24 @@ class CalendarProvider(abc.ABC):
Parameters
----------
start_time : str
start of the time range
start of the time range.
end_time : str
end of the time range
end of the time range.
freq : str
time frequency, available: year/quarter/month/week/day
time frequency, available: year/quarter/month/week/day.
future : bool
whether including future trading day
whether including future trading day.
Returns
-------
pd.Timestamp
the real start time
the real start time.
pd.Timestamp
the real end time
the real end time.
int
the index of start time
the index of start time.
int
the index of end time
the index of end time.
"""
start_time = pd.Timestamp(start_time)
end_time = pd.Timestamp(end_time)
@@ -103,16 +103,16 @@ class CalendarProvider(abc.ABC):
Parameters
----------
freq : str
frequency of read calendar file
frequency of read calendar file.
future : bool
whether including future trading day
whether including future trading day.
Returns
-------
list
list of timestamps
list of timestamps.
dict
dict composed by timestamp as key and index as value for fast search
dict composed by timestamp as key and index as value for fast search.
"""
flag = f"{freq}_future_{future}"
if flag in H["c"]:
@@ -141,14 +141,14 @@ class InstrumentProvider(abc.ABC):
Parameters
----------
market : str
market/industry/index shortname, e.g. all/sse/szse/sse50/csi300/csi500
market/industry/index shortname, e.g. all/sse/szse/sse50/csi300/csi500.
filter_pipe : list
the list of dynamic filters
the list of dynamic filters.
Returns
----------
dict
dict of stockpool config
dict of stockpool config.
{`market`=>base market name, `filter_pipe`=>list of filters}
example :
@@ -182,13 +182,13 @@ class InstrumentProvider(abc.ABC):
Parameters
----------
instruments : dict
stockpool config
stockpool config.
start_time : str
start of the time range
start of the time range.
end_time : str
end of the time range
end of the time range.
as_list : bool
return instruments as list or dict
return instruments as list or dict.
Returns
-------
@@ -243,15 +243,15 @@ class FeatureProvider(abc.ABC):
Parameters
----------
instrument : str
a certain instrument
a certain instrument.
field : str
a certain field of feature
a certain field of feature.
start_time : str
start of the time range
start of the time range.
end_time : str
end of the time range
end of the time range.
freq : str
time frequency, available: year/quarter/month/week/day
time frequency, available: year/quarter/month/week/day.
Returns
-------
@@ -294,15 +294,15 @@ class ExpressionProvider(abc.ABC):
Parameters
----------
instrument : str
a certain instrument
a certain instrument.
field : str
a certain field of feature
a certain field of feature.
start_time : str
start of the time range
start of the time range.
end_time : str
end of the time range
end of the time range.
freq : str
time frequency, available: year/quarter/month/week/day
time frequency, available: year/quarter/month/week/day.
Returns
-------
@@ -325,20 +325,20 @@ class DatasetProvider(abc.ABC):
Parameters
----------
instruments : list or dict
list/dict of instruments or dict of stockpool config
list/dict of instruments or dict of stockpool config.
fields : list
list of feature instances
list of feature instances.
start_time : str
start of the time range
start of the time range.
end_time : str
end of the time range
end of the time range.
freq : str
time frequency
time frequency.
Returns
----------
pd.DataFrame
a pandas dataframe with <instrument, datetime> index
a pandas dataframe with <instrument, datetime> index.
"""
raise NotImplementedError("Subclass of DatasetProvider must implement `Dataset` method")
@@ -357,17 +357,17 @@ class DatasetProvider(abc.ABC):
Parameters
----------
instruments : list or dict
list/dict of instruments or dict of stockpool config
list/dict of instruments or dict of stockpool config.
fields : list
list of feature instances
list of feature instances.
start_time : str
start of the time range
start of the time range.
end_time : str
end of the time range
end of the time range.
freq : str
time frequency
time frequency.
disk_cache : int
whether to skip(0)/use(1)/replace(2) disk_cache
whether to skip(0)/use(1)/replace(2) disk_cache.
"""
return DiskDatasetCache._uri(instruments, fields, start_time, end_time, freq, disk_cache)
@@ -526,7 +526,7 @@ class LocalCalendarProvider(CalendarProvider):
Parameters
----------
freq : str
frequency of read calendar file
frequency of read calendar file.
Returns
----------

30
qlib/data/dataset/__init__.py
View File

@@ -17,7 +17,7 @@ class Dataset(Serializable):
init is designed to finish following steps:
- setup data
- The data related attributes' names should start with '_' so that it will not be saved on disk when serializing
- The data related attributes' names should start with '_' so that it will not be saved on disk when serializing.
- initialize the state of the dataset(info to prepare the data)
- The name of essential state for preparing data should not start with '_' so that it could be serialized on disk when serializing.
@@ -29,17 +29,17 @@ class Dataset(Serializable):
def setup_data(self, *args, **kwargs):
"""
setup the data
Setup the data.
We split the setup_data function for following situation:
- User have a Dataset object with learned status on disk
- User have a Dataset object with learned status on disk.
- User load the Dataset object from the disk(Note the init function is skiped)
- User load the Dataset object from the disk(Note the init function is skiped).
- User call `setup_data` to load new data
- User call `setup_data` to load new data.
- User prepare data for model based on previous status
- User prepare data for model based on previous status.
"""
pass
@@ -66,9 +66,10 @@ class DatasetH(Dataset):
User should try to put the data preprocessing functions into handler.
Only following data processing functions should be placed in Dataset:
- The processing is related to specific model.
- The processing is related to data split
- The processing is related to data split.
"""
def __init__(self, handler: Union[dict, DataHandler], segments: list):
@@ -76,15 +77,15 @@ class DatasetH(Dataset):
Parameters
----------
handler : Union[dict, DataHandler]
handler will be passed into setup_data
handler will be passed into setup_data.
segments : list
handler will be passed into setup_data
handler will be passed into setup_data.
"""
super().__init__(handler, segments)
def setup_data(self, handler: Union[dict, DataHandler], segments: list):
"""
setup the underlying data
Setup the underlying data.
Parameters
----------
@@ -121,7 +122,7 @@ class DatasetH(Dataset):
**kwargs,
) -> Union[List[pd.DataFrame], pd.DataFrame]:
"""
prepare the data for learning and inference
Prepare the data for learning and inference.
Parameters
----------
@@ -132,11 +133,12 @@ class DatasetH(Dataset):
- 'train'
- ['train', 'valid']
col_set : str
The col_set will be passed to self._handler when fetching data
data_key: str
The col_set will be passed to self._handler when fetching data.
data_key : str
The data to fetch: DK_*
Default is DK_I, which indicate fetching data for **inference**
Default is DK_I, which indicate fetching data for **inference**.
Returns
-------

38
qlib/data/dataset/handler.py
View File

@@ -29,7 +29,7 @@ class DataHandler(Serializable):
"""
The steps to using a handler
1. initialized data handler (call by `init`).
2. use the data
2. use the data.
The data handler try to maintain a handler with 2 level.
@@ -65,17 +65,17 @@ class DataHandler(Serializable):
Parameters
----------
instruments :
The stock list to retrive
The stock list to retrive.
start_time :
start_time of the original data
start_time of the original data.
end_time :
end_time of the original data
end_time of the original data.
data_loader : Tuple[dict, str, DataLoader]
data loader to load the data
data loader to load the data.
init_data :
intialize the original data in the constructor
intialize the original data in the constructor.
fetch_orig : bool
Return the original data instead of copy if possible
Return the original data instead of copy if possible.
"""
# Set logger
self.logger = get_module_logger("DataHandler")
@@ -219,9 +219,9 @@ class DataHandler(Serializable):
get a iterator of sliced data with given periods
Args:
periods (int): number of periods
min_periods (int): minimum periods for sliced dataframe
kwargs (dict): will be passed to `self.fetch`
periods (int): number of periods.
min_periods (int): minimum periods for sliced dataframe.
kwargs (dict): will be passed to `self.fetch`.
"""
trading_dates = self._data.index.unique(level="datetime")
if min_periods is None:
@@ -377,7 +377,7 @@ class DataHandlerLP(DataHandler):
Parameters
----------
init_type : str
The type `IT_*` listed above
The type `IT_*` listed above.
enable_cache : bool
default value is false:
@@ -419,13 +419,13 @@ class DataHandlerLP(DataHandler):
Parameters
----------
selector : Union[pd.Timestamp, slice, str]
describe how to select data by index
describe how to select data by index.
level : Union[str, int]
which index level to select the data
which index level to select the data.
col_set : str
select a set of meaningful columns.(e.g. features, columns)
data_key: str
The data to fetch: DK_*
select a set of meaningful columns.(e.g. features, columns).
data_key : str
the data to fetch: DK_*.
Returns
-------
@@ -443,9 +443,9 @@ class DataHandlerLP(DataHandler):
Parameters
----------
col_set : str
select a set of meaningful columns.(e.g. features, columns)
data_key: str
The data to fetch: DK_*
select a set of meaningful columns.(e.g. features, columns).
data_key : str
the data to fetch: DK_*.
Returns
-------

8
qlib/data/dataset/loader.py
View File

@@ -100,16 +100,16 @@ class DLWParser(DataLoader):
Parameters
----------
instruments :
the instruments
the instruments.
exprs : list
The expressions to describe the content of the data
the expressions to describe the content of the data.
names : list
The name of the data
the name of the data.
Returns
-------
pd.DataFrame:
the queried dataframe
the queried dataframe.
"""
pass

6
qlib/data/dataset/processor.py
View File

@@ -21,7 +21,7 @@ def get_group_columns(df: pd.DataFrame, group: str):
Parameters
----------
df : pd.DataFrame
with multi of columns
with multi of columns.
group : str
the name of the feature group, i.e. the first level value of the group index.
"""
@@ -56,7 +56,7 @@ class Processor(Serializable):
Parameters
----------
df : pd.DataFrame
The raw_df of handler or result from previous processor
The raw_df of handler or result from previous processor.
"""
pass
@@ -68,7 +68,7 @@ class Processor(Serializable):
Returns
-------
bool:
if it is usable for infenrece
if it is usable for infenrece.
"""
return True

60
qlib/data/filter.py
View File

@@ -32,7 +32,7 @@ class BaseDFilter(abc.ABC):
Parameters
----------
config : dict
dict of config parameters
dict of config parameters.
"""
raise NotImplementedError("Subclass of BaseDFilter must reimplement `from_config` method")
@@ -43,7 +43,7 @@ class BaseDFilter(abc.ABC):
Returns
----------
dict
return the dict of config parameters
return the dict of config parameters.
"""
raise NotImplementedError("Subclass of BaseDFilter must reimplement `to_config` method")
@@ -69,9 +69,9 @@ class SeriesDFilter(BaseDFilter):
Parameters
----------
fstart_time: str
the time for the filter rule to start filter the instruments
the time for the filter rule to start filter the instruments.
fend_time: str
the time for the filter rule to stop filter the instruments
the time for the filter rule to stop filter the instruments.
"""
super(SeriesDFilter, self).__init__()
self.filter_start_time = pd.Timestamp(fstart_time) if fstart_time else None
@@ -83,12 +83,12 @@ class SeriesDFilter(BaseDFilter):
Parameters
----------
instruments: dict
the dict of instruments in the form {instrument_name => list of timestamp tuple}
the dict of instruments in the form {instrument_name => list of timestamp tuple}.
Returns
----------
pd.Timestamp, pd.Timestamp
the lower time bound and upper time bound of all the instruments
the lower time bound and upper time bound of all the instruments.
"""
trange = Cal.calendar(freq=self.filter_freq)
ubound, lbound = trange[0], trange[-1]
@@ -105,14 +105,14 @@ class SeriesDFilter(BaseDFilter):
Parameters
----------
time_range : D.calendar
the time range of the instruments
the time range of the instruments.
target_timestamp : list
the list of tuple (timestamp, timestamp)
the list of tuple (timestamp, timestamp).
Returns
----------
pd.Series
the series of bool value for an instrument
the series of bool value for an instrument.
"""
# Construct a whole dict of {date => bool}
timestamp_series = {timestamp: False for timestamp in time_range}
@@ -124,19 +124,19 @@ class SeriesDFilter(BaseDFilter):
return timestamp_series
def _filterSeries(self, timestamp_series, filter_series):
"""Filter the timestamp series with filter series by using element-wise AND operation of the two series
"""Filter the timestamp series with filter series by using element-wise AND operation of the two series.
Parameters
----------
timestamp_series : pd.Series
the series of bool value indicating existing time
the series of bool value indicating existing time.
filter_series : pd.Series
the series of bool value indicating filter feature
the series of bool value indicating filter feature.
Returns
----------
pd.Series
the series of bool value indicating whether the date satisfies the filter condition and exists in target timestamp
the series of bool value indicating whether the date satisfies the filter condition and exists in target timestamp.
"""
fstart, fend = list(filter_series.keys())[0], list(filter_series.keys())[-1]
filter_series = filter_series.astype("bool") # Make sure the filter_series is boolean
@@ -144,17 +144,17 @@ class SeriesDFilter(BaseDFilter):
return timestamp_series
def _toTimestamp(self, timestamp_series):
"""Convert the timestamp series to a list of tuple (timestamp, timestamp) indicating a continuous range of TRUE
"""Convert the timestamp series to a list of tuple (timestamp, timestamp) indicating a continuous range of TRUE.
Parameters
----------
timestamp_series: pd.Series
the series of bool value after being filtered
the series of bool value after being filtered.
Returns
----------
list
the list of tuple (timestamp, timestamp)
the list of tuple (timestamp, timestamp).
"""
# sort the timestamp_series according to the timestamps
timestamp_series.sort_index()
@@ -194,18 +194,18 @@ class SeriesDFilter(BaseDFilter):
Parameters
----------
instruments : dict
the dict of instruments to be filtered
the dict of instruments to be filtered.
fstart : pd.Timestamp
start time of filter
start time of filter.
fend : pd.Timestamp
end time of filter
end time of filter.
.. note:: fstart/fend indicates the intersection of instruments start/end time and filter start/end time
.. note:: fstart/fend indicates the intersection of instruments start/end time and filter start/end time.
Returns
----------
pd.Dataframe
a series of {pd.Timestamp => bool}
a series of {pd.Timestamp => bool}.
"""
raise NotImplementedError("Subclass of SeriesDFilter must reimplement `getFilterSeries` method")
@@ -215,16 +215,16 @@ class SeriesDFilter(BaseDFilter):
Parameters
----------
instruments: dict
input instruments to be filtered
input instruments to be filtered.
start_time: str
start of the time range
start of the time range.
end_time: str
end of the time range
end of the time range.
Returns
----------
dict
filtered instruments, same structure as input instruments
filtered instruments, same structure as input instruments.
"""
lbound, ubound = self._getTimeBound(instruments)
start_time = pd.Timestamp(start_time or lbound)
@@ -272,7 +272,7 @@ class NameDFilter(SeriesDFilter):
params:
------
name_rule_re: str
regular expression for the name rule
regular expression for the name rule.
"""
super(NameDFilter, self).__init__(fstart_time, fend_time)
self.name_rule_re = name_rule_re
@@ -325,13 +325,13 @@ class ExpressionDFilter(SeriesDFilter):
params:
------
fstart_time: str
filter the feature starting from this time
filter the feature starting from this time.
fend_time: str
filter the feature ending by this time
filter the feature ending by this time.
rule_expression: str
an input expression for the rule
an input expression for the rule.
keep: bool
whether to keep the instruments of which features don't exist in the filter time span
whether to keep the instruments of which features don't exist in the filter time span.
"""
super(ExpressionDFilter, self).__init__(fstart_time, fend_time)
self.rule_expression = rule_expression

6
qlib/model/base.py
View File

@@ -33,7 +33,7 @@ class Model(BaseModel):
Parameters
----------
dataset : Dataset
dataset will generate the processed data from model training
dataset will generate the processed data from model training.
"""
raise NotImplementedError()
@@ -44,7 +44,7 @@ class Model(BaseModel):
Parameters
----------
dataset : Dataset
dataset will generate the processed dataset from model training
dataset will generate the processed dataset from model training.
"""
raise NotImplementedError()
@@ -59,6 +59,6 @@ class ModelFT(Model):
Parameters
----------
dataset : Dataset
dataset will generate the processed dataset from model training
dataset will generate the processed dataset from model training.
"""
raise NotImplementedError()

44
qlib/model/riskmodel.py
View File

@@ -23,9 +23,9 @@ class RiskModel(BaseModel):
def __init__(self, nan_option: str = "ignore", assume_centered: bool = False, scale_return: bool = True):
"""
Args:
nan_option (str): nan handling option (`ignore`/`mask`/`fill`)
assume_centered (bool): whether the data is assumed to be centered
scale_return (bool): whether scale returns as percentage
nan_option (str): nan handling option (`ignore`/`mask`/`fill`).
assume_centered (bool): whether the data is assumed to be centered.
scale_return (bool): whether scale returns as percentage.
"""
# nan
assert nan_option in [
@@ -45,11 +45,11 @@ class RiskModel(BaseModel):
Args:
X (pd.Series, pd.DataFrame or np.ndarray): data from which to estimate the covariance,
with variables as columns and observations as rows.
return_corr (bool): whether return the correlation matrix
is_price (bool): whether `X` contains price (if not assume stock returns)
return_corr (bool): whether return the correlation matrix.
is_price (bool): whether `X` contains price (if not assume stock returns).
Returns:
pd.DataFrame or np.ndarray: estimated covariance (or correlation)
pd.DataFrame or np.ndarray: estimated covariance (or correlation).
"""
# transform input into 2D array
if not isinstance(X, (pd.Series, pd.DataFrame)):
@@ -101,10 +101,10 @@ class RiskModel(BaseModel):
By default, this method implements the empirical covariance estimation.
Args:
X (np.ndarray): data matrix containing multiple variables (columns) and observations (rows)
X (np.ndarray): data matrix containing multiple variables (columns) and observations (rows).
Returns:
np.ndarray: covariance matrix
np.ndarray: covariance matrix.
"""
xTx = np.asarray(X.T.dot(X))
N = len(X)
@@ -117,7 +117,7 @@ class RiskModel(BaseModel):
"""handle nan and centerize data
Note:
if `nan_option='mask'` then the returned array will be `np.ma.MaskedArray`
if `nan_option='mask'` then the returned array will be `np.ma.MaskedArray`.
"""
# handle nan
if self.nan_option == self.FILL_NAN:
@@ -139,15 +139,15 @@ class ShrinkCovEstimator(RiskModel):
where `alpha` is the shrink parameter and `F` is the shrinking target.
The following shrinking parameters (`alpha`) are supported:
- `lw` [1][2][3]: use Ledoit-Wolf shrinking parameter
- `oas` [4]: use Oracle Approximating Shrinkage shrinking parameter
- float: directly specify the shrink parameter, should be between [0, 1]
- `lw` [1][2][3]: use Ledoit-Wolf shrinking parameter.
- `oas` [4]: use Oracle Approximating Shrinkage shrinking parameter.
- float: directly specify the shrink parameter, should be between [0, 1].
The following shrinking targets (`F`) are supported:
- `const_var` [1][4][5]: assume stocks have the same constant variance and zero correlation
- `const_corr` [2][6]: assume stocks have different variance but equal correlation
- `single_factor` [3][7]: assume single factor model as the shrinking target
- np.ndarray: provide the shrinking targets directly
- `const_var` [1][4][5]: assume stocks have the same constant variance and zero correlation.
- `const_corr` [2][6]: assume stocks have different variance but equal correlation.
- `single_factor` [3][7]: assume single factor model as the shrinking target.
- np.ndarray: provide the shrinking targets directly.
Note:
- The optimal shrinking parameter depends on the selection of the shrinking target.
@@ -402,13 +402,13 @@ class POETCovEstimator(RiskModel):
def __init__(self, num_factors: int = 0, thresh: float = 1.0, thresh_method: str = "soft", **kwargs):
"""
Args:
num_factors (int): number of factors (if set to zero, no factor model will be used)
thresh (float): the positive constant for thresholding
num_factors (int): number of factors (if set to zero, no factor model will be used).
thresh (float): the positive constant for thresholding.
thresh_method (str): thresholding method, which can be
- 'soft': soft thresholding
- 'hard': hard thresholding
- 'scad': scad thresholding
kwargs: see `RiskModel` for more information
- 'soft': soft thresholding.
- 'hard': hard thresholding.
- 'scad': scad thresholding.
kwargs: see `RiskModel` for more information.
"""
super().__init__(**kwargs)