liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-11 14:56:55 +08:00
Code Activity

Merge nested main (#597)

Browse Source 
* MVP for Indian Stocks in qlib using yahooquery

* cleaned with black

* cleaned with black

* add YahooNormalizeIN and YahooNormalizeIN1d

* cleaned the code

* added 1min for IN and also updated readme

* update comments

* fix comments

* recorder support upload both raw file and directory

* fix comments

* Update README.md

* Fix docs of QlibRecorder

* sort index after loader (#538)

make sure the fetch method is based on a index-sorted pd.DataFrame

* refactor online serving rolling api

* refactor TRA

* format by black

* fix horizon

* fix TRA when use single head

* clean up

* improve pretrain

* update README

* fix tra when logdir is None

* fix tra when logdir is None

* Update strategy.py

* Update README.md

* Update README.md

* Conda Suggestion

* code standard docs

* Update ensemble.py (#560)

* Fix CI  Bug (#575)


Co-authored-by: yuxwang <anduinnn@foxmail.com>

* Update gen.py (#576)

* Fix multi-process loop calls (#574)

* check lexsort in the 'lazy_sort_index' function (#566)

* check lexsort

* check lexsort

* lexsort comment

* lexsort comment

* Delete .DS_Store

* Update README.md

* bug fix & use oracle transport pretrain

* mend

* Add `backend_freq_config` parameter, support multi-freq uri

* Add sample_config to QlibDataLoader, support multi-freq

* add multi-freq example

* get_cls_kwargs renamed get_callable_kwargs

* support multi-freq uri

* Add inst_processors to D.features

* Fix typo

* Fix the index type of the multi-freq example

* Fix duplicate mlflow directories in tests

* Add DataPathManager to QlibConfig && modify inst_processors to supports list only

* Modify the default value in the multi_freq example

* Modify client-server mode and dataset-cache to disable inst_processor

* Add wheel package to github CI

* fix comment

* Update FAQ.rst

* Update README.md

Fix wrong link

* Update the docs of TaskManager (#586)

* Update manage.py

* update yaml

* update run_all_model

* Modify the Feature to be case sensitive (#589)

* update README

* remove verbose

* fix spell bug

* fix typos (#592)

* Update Release Note

* fix portfolio bug

* Add calendar support for resample

* add freq kwargs

* test.yml: Remove redundant code (#595)

* Supporting shared processor (#596)

* Supporting shared processor

* fix readonly reverse bug

* remove pytests dependency

* with fit bug

* fix parameter error

* fix comments

* Fix undefined names in Python code (#599)

* Update pytorch_tabnet.py

$ `flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics`
```
./qlib/qlib/contrib/model/pytorch_tabnet.py:567:38: F821 undefined name 'inp'
            self.independ.append(GLU(inp, out_dim, vbs=vbs))
                                     ^
./qlib/examples/model_rolling/task_manager_rolling.py:75:18: F821 undefined name 'task_train'
        run_task(task_train, self.task_pool, experiment_name=self.experiment_name)
                 ^
2     F821 undefined name 'task_train'
2
```

* Fix undefined names in Python code

* from qlib.model.trainer import task_train

* update seed

* fix some docstring

* add comments

* Fix SimpleDatasetCache

* Update setup.py

updated classifiers

* Update setup.py

change to matplotlib==3.3

* Update python-publish.yml

added python 3.9

* updategrade version number

* Update model list

* fix the type of filter_pipe

* fix comment

* fix record_temp

* update cvxpy version

* Update code_standard.rst (#587)

* Update code_standard.rst

* Update docs/developer/code_standard.rst

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

* Add file lock for MLflowExpManager (#619)

* fix torch version

* Share version number (#620)

* Update initialization.rst (#622)

* Update initialization.rst

* Update docs/start/initialization.rst

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

* Update docs/start/initialization.rst

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

* fix bugs for running previous exmaple

* fix deal amount bug

* update change doc (#623)

* Add files via upload

* Update README.md

* Update README.md

* Update README.md

* Delete change doc.gif

* Add files via upload

* Update README.md

* Delete change doc.gif

* Add files via upload

* Delete change doc.gif

* Add files via upload

* Update README.md

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

* update doc

* simplify run all model

* fix run all model bug

* Fix Models (#483)

* fix gat dataset

* fix tft model

* Update tft.py

* Fix tft.py

Co-authored-by: Pengrong Zhu <zhu.pengrong@foxmail.com>

* type and skip empty exp

* fix model yaml config

* fix tft import bug

* skip empty result

* fix model and yaml bug

* fix wrong generate parameter

* Modify multi-freq example (#626)

* modify the example of multi-freq

* add Copyright

* add a comment to average_ops.py

* modify the example of multi-freq

* add comment to multi_freq_handler.py

* add the Ref expression description to multi_freq_handler.py

* add expression description to multi_freq_handler.py

* update images

* fix workflow and update framework

Co-authored-by: Gaurav <2796gaurav@gmail.com>
Co-authored-by: 2796gaurav <17353992+2796gaurav@users.noreply.github.com>
Co-authored-by: bxdd <bxd98@126.com>
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: Dong Zhou <Zhou.Dong@microsoft.com>
Co-authored-by: ZhangTP1996 <ztp18@mails.tsinghua.edu.cn>
Co-authored-by: demon143 <59681577+demon143@users.noreply.github.com>
Co-authored-by: Wangwuyi123 <51237097+Wangwuyi123@users.noreply.github.com>
Co-authored-by: yuxwang <anduinnn@foxmail.com>
Co-authored-by: Pengrong Zhu <zhu.pengrong@foxmail.com>
Co-authored-by: Mark Zhao <50850474+markzhao98@users.noreply.github.com>
Co-authored-by: cslwqxx <cslwqxx@users.noreply.github.com>
Co-authored-by: Dong Zhou <evanzd@users.noreply.github.com>
Co-authored-by: SaintMalik <37118134+saintmalik@users.noreply.github.com>
Co-authored-by: Christian Clauss <cclauss@me.com>
Co-authored-by: Anurag Kumar <mailanu98@gmail.com>
Co-authored-by: demon143 <785696300@qq.com>
This commit is contained in:
wangwenxi-handsome
2021-10-01 02:15:30 +08:00
committed by GitHub
parent 163e3c6266
commit 3760a18a8d
145 changed files with 3982 additions and 1221 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
qlib/backtest/__init__.py
View File

@@ -9,13 +9,13 @@ from .account import Account
if TYPE_CHECKING:
from ..strategy.base import BaseStrategy
from .executor import BaseExecutor
from .order import BaseTradeDecision
from .order import Order
from .decision import BaseTradeDecision
from .position import Position
from .exchange import Exchange
from .backtest import backtest_loop
from .backtest import collect_data_loop
from .utils import CommonInfrastructure, LevelInfrastructure, TradeCalendarManager
from .utils import CommonInfrastructure
from .decision import Order
from ..utils import init_instance_by_config
from ..log import get_module_logger
from ..config import C
@@ -231,10 +231,9 @@ def backtest(
Returns
-------
report: Report
it records the trading report information
It is organized in a dict format
indicator: Indicator
portfolio_metrics_dict: Dict[PortfolioMetrics]
it records the trading portfolio_metrics information
indicator_dict: Dict[Indicator]
it computes the trading indicator
It is organized in a dict format
@@ -249,9 +248,8 @@ def backtest(
exchange_kwargs,
pos_type=pos_type,
)
report, indicator = backtest_loop(start_time, end_time, trade_strategy, trade_executor)
return report, indicator
portfolio_metrics, indicator = backtest_loop(start_time, end_time, trade_strategy, trade_executor)
return portfolio_metrics, indicator
def collect_data(

217
qlib/backtest/account.py
View File

@@ -4,22 +4,19 @@ from __future__ import annotations
import copy
from typing import Dict, List, Tuple, TYPE_CHECKING
from qlib.utils import init_instance_by_config
import warnings
import pandas as pd
from .position import BasePosition, InfPosition, Position
from .report import Report, Indicator
from .order import BaseTradeDecision, Order
if TYPE_CHECKING:
from .exchange import Exchange
from .report import PortfolioMetrics, Indicator
from .decision import BaseTradeDecision, Order
from .exchange import Exchange
"""
rtn & earning in the Account
rtn:
from order's view
1.change if any order is executed, sell order or buy order
2.change at the end of today, (today_clse - stock_price) * amount
2.change at the end of today, (today_close - stock_price) * amount
earning
from value of current position
earning will be updated at the end of trade date
@@ -32,7 +29,7 @@ rtn & earning in the Account
class AccumulatedInfo:
"""accumulated trading info, including accumulated return\cost\turnover"""
"""accumulated trading info, including accumulated return/cost/turnover"""
def __init__(self):
self.reset()
@@ -94,9 +91,12 @@ class Account:
self._pos_type = pos_type
self._port_metr_enabled = port_metr_enabled
self.benchmark_config = None # avoid no attribute error
self.init_vars(init_cash, position_dict, freq, benchmark_config)
def init_vars(self, init_cash, position_dict, freq: str, benchmark_config: dict):
self.init_cash = init_cash
self.current: BasePosition = init_instance_by_config(
self.current_position: BasePosition = init_instance_by_config(
{
"class": self._pos_type,
"kwargs": {
@@ -106,37 +106,33 @@ class Account:
"module_path": "qlib.backtest.position",
}
)
self.report = None
self.positions = {}
# in of reset ignore None values
self.benchmark_config = benchmark_config
self.freq = freq
self.reset(freq=freq, benchmark_config=benchmark_config, init_report=True)
self.portfolio_metrics = None
self.hist_positions = {}
self.reset(freq=freq, benchmark_config=benchmark_config)
def is_port_metr_enabled(self):
"""
Is portfolio-based metrics enabled.
"""
return self._port_metr_enabled and not self.current.skip_update()
return self._port_metr_enabled and not self.current_position.skip_update()
def reset_report(self, freq, benchmark_config):
# portfolio related metrics
if self.is_port_metr_enabled():
self.accum_info = AccumulatedInfo()
self.report = Report(freq, benchmark_config)
self.positions = {}
self.portfolio_metrics = PortfolioMetrics(freq, benchmark_config)
self.hist_positions = {}
# fill stock value
# The frequency of account may not align with the trading frequency.
# This may result in obscure bugs when data quality is low.
if isinstance(self.benchmark_config, dict) and self.benchmark_config.get("start_time") is not None:
self.current.fill_stock_value(self.benchmark_config["start_time"], self.freq)
self.current_position.fill_stock_value(self.benchmark_config["start_time"], self.freq)
# trading related metrics(e.g. high-frequency trading)
self.indicator = Indicator()
def reset(self, freq=None, benchmark_config=None, init_report=False, port_metr_enabled: bool = None):
def reset(self, freq=None, benchmark_config=None, port_metr_enabled: bool = None):
"""reset freq and report of account
Parameters
@@ -145,27 +141,23 @@ class Account:
frequency of account & report, by default None
benchmark_config : {}, optional
benchmark config of report, by default None
init_report : bool, optional
whether to initialize the report, by default False
"""
if freq is not None:
self.freq = freq
if benchmark_config is not None:
self.benchmark_config = benchmark_config
if port_metr_enabled is not None:
self._port_metr_enabled = port_metr_enabled
if freq is not None or benchmark_config is not None or init_report:
self.reset_report(self.freq, self.benchmark_config)
self.reset_report(self.freq, self.benchmark_config)
def get_positions(self):
return self.positions
def get_hist_positions(self):
return self.hist_positions
def get_cash(self):
return self.current.get_cash()
return self.current_position.get_cash()
def _update_accum_info_from_order(self, order, trade_val, cost, trade_price):
def _update_state_from_order(self, order, trade_val, cost, trade_price):
if self.is_port_metr_enabled():
# update turnover
self.accum_info.add_turnover(trade_val)
@@ -176,17 +168,17 @@ class Account:
trade_amount = trade_val / trade_price
if order.direction == Order.SELL: # 0 for sell
# when sell stock, get profit from price change
profit = trade_val - self.current.get_stock_price(order.stock_id) * trade_amount
profit = trade_val - self.current_position.get_stock_price(order.stock_id) * trade_amount
self.accum_info.add_return_value(profit) # note here do not consider cost
elif order.direction == Order.BUY: # 1 for buy
# when buy stock, we get return for the rtn computing method
# profit in buy order is to make rtn is consistent with earning at the end of bar
profit = self.current.get_stock_price(order.stock_id) * trade_amount - trade_val
profit = self.current_position.get_stock_price(order.stock_id) * trade_amount - trade_val
self.accum_info.add_return_value(profit) # note here do not consider cost
def update_order(self, order, trade_val, cost, trade_price):
if self.current.skip_update():
if self.current_position.skip_update():
# TODO: supporting polymorphism for account
# updating order for infinite position is meaningless
return
@@ -196,65 +188,61 @@ class Account:
# The cost will be substracted from the cash at last. So the trading logic can ignore the cost calculation
if order.direction == Order.SELL:
# sell stock
self._update_accum_info_from_order(order, trade_val, cost, trade_price)
self._update_state_from_order(order, trade_val, cost, trade_price)
# update current position
# for may sell all of stock_id
self.current.update_order(order, trade_val, cost, trade_price)
self.current_position.update_order(order, trade_val, cost, trade_price)
else:
# buy stock
# deal order, then update state
self.current.update_order(order, trade_val, cost, trade_price)
self._update_accum_info_from_order(order, trade_val, cost, trade_price)
self.current_position.update_order(order, trade_val, cost, trade_price)
self._update_state_from_order(order, trade_val, cost, trade_price)
def update_bar_count(self):
"""at the end of the trading bar, update holding bar, count of stock"""
# update holding day count
# NOTE: updating bar_count does not only serve portfolio metrics, it also serve the strategy
if not self.current.skip_update():
self.current.add_count_all(bar=self.freq)
def update_current(self, trade_start_time, trade_end_time, trade_exchange):
"""update current to make rtn consistent with earning at the end of bar"""
def update_current_position(self, trade_start_time, trade_end_time, trade_exchange):
"""update current to make rtn consistent with earning at the end of bar, and update holding bar count of stock"""
# update price for stock in the position and the profit from changed_price
# NOTE: updating position does not only serve portfolio metrics, it also serve the strategy
if not self.current.skip_update():
stock_list = self.current.get_stock_list()
if not self.current_position.skip_update():
stock_list = self.current_position.get_stock_list()
for code in stock_list:
# if suspend, no new price to be updated, profit is 0
if trade_exchange.check_stock_suspended(code, trade_start_time, trade_end_time):
continue
bar_close = trade_exchange.get_close(code, trade_start_time, trade_end_time)
self.current.update_stock_price(stock_id=code, price=bar_close)
self.current_position.update_stock_price(stock_id=code, price=bar_close)
# update holding day count
# NOTE: updating bar_count does not only serve portfolio metrics, it also serve the strategy
self.current_position.add_count_all(bar=self.freq)
def update_report(self, trade_start_time, trade_end_time):
"""update position history, report"""
def update_portfolio_metrics(self, trade_start_time, trade_end_time):
"""update portfolio_metrics"""
# calculate earning
# account_value - last_account_value
# for the first trade date, account_value - init_cash
# self.report.is_empty() to judge is_first_trade_date
# self.portfolio_metrics.is_empty() to judge is_first_trade_date
# get last_account_value, last_total_cost, last_total_turnover
if self.report.is_empty():
if self.portfolio_metrics.is_empty():
last_account_value = self.init_cash
last_total_cost = 0
last_total_turnover = 0
else:
last_account_value = self.report.get_latest_account_value()
last_total_cost = self.report.get_latest_total_cost()
last_total_turnover = self.report.get_latest_total_turnover()
last_account_value = self.portfolio_metrics.get_latest_account_value()
last_total_cost = self.portfolio_metrics.get_latest_total_cost()
last_total_turnover = self.portfolio_metrics.get_latest_total_turnover()
# get now_account_value, now_stock_value, now_earning, now_cost, now_turnover
now_account_value = self.current.calculate_value()
now_stock_value = self.current.calculate_stock_value()
now_account_value = self.current_position.calculate_value()
now_stock_value = self.current_position.calculate_stock_value()
now_earning = now_account_value - last_account_value
now_cost = self.accum_info.get_cost - last_total_cost
now_turnover = self.accum_info.get_turnover - last_total_turnover
# update report for today
# update portfolio_metrics for today
# judge whether the the trading is begin.
# and don't add init account state into report, due to we don't have excess return in those days.
self.report.update_report_record(
# and don't add init account state into portfolio_metrics, due to we don't have excess return in those days.
self.portfolio_metrics.update_portfolio_metrics_record(
trade_start_time=trade_start_time,
trade_end_time=trade_end_time,
account_value=now_account_value,
cash=self.current.position["cash"],
cash=self.current_position.position["cash"],
return_rate=(now_earning + now_cost) / last_account_value,
# here use earning to calculate return, position's view, earning consider cost, true return
# in order to make same definition with original backtest in evaluate.py
@@ -264,12 +252,51 @@ class Account:
cost_rate=now_cost / last_account_value,
stock_value=now_stock_value,
)
def update_hist_positions(self, trade_start_time):
"""update history position"""
now_account_value = self.current_position.calculate_value()
# set now_account_value to position
self.current.position["now_account_value"] = now_account_value
self.current.update_weight_all()
# update positions
self.current_position.position["now_account_value"] = now_account_value
self.current_position.update_weight_all()
# update hist_positions
# note use deepcopy
self.positions[trade_start_time] = copy.deepcopy(self.current)
self.hist_positions[trade_start_time] = copy.deepcopy(self.current_position)
def update_indicator(
self,
trade_start_time: pd.Timestamp,
trade_exchange: Exchange,
atomic: bool,
outer_trade_decision: BaseTradeDecision,
trade_info: list = None,
inner_order_indicators: List[Dict[str, pd.Series]] = None,
decision_list: List[Tuple[BaseTradeDecision, pd.Timestamp, pd.Timestamp]] = None,
indicator_config: dict = {},
):
"""update trade indicators and order indicators in each bar end"""
# TODO: will skip empty decisions make it faster? `outer_trade_decision.empty():`
# indicator is trading (e.g. high-frequency order execution) related analysis
self.indicator.reset()
# aggregate the information for each order
if atomic:
self.indicator.update_order_indicators(trade_info)
else:
self.indicator.agg_order_indicators(
inner_order_indicators,
decision_list=decision_list,
outer_trade_decision=outer_trade_decision,
trade_exchange=trade_exchange,
indicator_config=indicator_config,
)
# aggregate all the order metrics a single step
self.indicator.cal_trade_indicators(trade_start_time, self.freq, indicator_config)
# record the metrics
self.indicator.record(trade_start_time)
def update_bar_end(
self,
@@ -316,44 +343,34 @@ class Account:
elif atomic is False and inner_order_indicators is None:
raise ValueError("inner_order_indicators is necessary in un-atomic executor")
# TODO: `update_bar_count` and `update_current` should placed in Position and be merged.
self.update_bar_count()
self.update_current(trade_start_time, trade_end_time, trade_exchange)
# update current position and hold bar count in each bar end
self.update_current_position(trade_start_time, trade_end_time, trade_exchange)
if self.is_port_metr_enabled():
# report is portfolio related analysis
self.update_report(trade_start_time, trade_end_time)
# portfolio_metrics is portfolio related analysis
self.update_portfolio_metrics(trade_start_time, trade_end_time)
self.update_hist_positions(trade_start_time)
# TODO: will skip empty decisions make it faster? `outer_trade_decision.empty():`
# update indicator in each bar end
self.update_indicator(
trade_start_time=trade_start_time,
trade_exchange=trade_exchange,
atomic=atomic,
outer_trade_decision=outer_trade_decision,
trade_info=trade_info,
inner_order_indicators=inner_order_indicators,
decision_list=decision_list,
indicator_config=indicator_config,
)
# indicator is trading (e.g. high-frequency order execution) related analysis
self.indicator.reset()
# aggregate the information for each order
if atomic:
self.indicator.update_order_indicators(trade_info)
else:
self.indicator.agg_order_indicators(
inner_order_indicators,
decision_list=decision_list,
outer_trade_decision=outer_trade_decision,
trade_exchange=trade_exchange,
indicator_config=indicator_config,
)
# aggregate all the order metrics a single step
self.indicator.cal_trade_indicators(trade_start_time, self.freq, indicator_config)
# record the metrics
self.indicator.record(trade_start_time)
def get_report(self):
"""get the history report and postions instance"""
def get_portfolio_metrics(self):
"""get the history portfolio_metrics and postions instance"""
if self.is_port_metr_enabled():
_report = self.report.generate_report_dataframe()
_positions = self.get_positions()
return _report, _positions
_portfolio_metrics = self.portfolio_metrics.generate_portfolio_metrics_dataframe()
_positions = self.get_hist_positions()
return _portfolio_metrics, _positions
else:
raise ValueError("generate_report should be True if you want to generate report")
raise ValueError("generate_portfolio_metrics should be True if you want to generate portfolio_metrics")
def get_trade_indicator(self) -> Indicator:
"""get the trade indicator instance, which has pa/pos/ffr info."""

15
qlib/backtest/backtest.py
View File

@@ -2,7 +2,7 @@
# Licensed under the MIT License.
from __future__ import annotations
from qlib.backtest.order import BaseTradeDecision
from qlib.backtest.decision import BaseTradeDecision
from typing import TYPE_CHECKING
if TYPE_CHECKING:
@@ -19,15 +19,15 @@ def backtest_loop(start_time, end_time, trade_strategy: BaseStrategy, trade_exec
Returns
-------
report: Report
it records the trading report information
portfolio_metrics: PortfolioMetrics
it records the trading portfolio_metrics information
indicator: Indicator
it computes the trading indicator
"""
return_value = {}
for _decision in collect_data_loop(start_time, end_time, trade_strategy, trade_executor, return_value):
pass
return return_value.get("report"), return_value.get("indicator")
return return_value.get("portfolio_metrics"), return_value.get("indicator")
def collect_data_loop(
@@ -68,9 +68,8 @@ def collect_data_loop(
if return_value is not None:
all_executors = trade_executor.get_all_executors()
all_reports = {
"{}{}".format(*Freq.parse(_executor.time_per_step)): _executor.trade_account.get_report()
all_portfolio_metrics = {
"{}{}".format(*Freq.parse(_executor.time_per_step)): _executor.trade_account.get_portfolio_metrics()
for _executor in all_executors
if _executor.trade_account.is_port_metr_enabled()
}
@@ -79,4 +78,4 @@ def collect_data_loop(
key = "{}{}".format(*Freq.parse(_executor.time_per_step))
all_indicators[key] = _executor.trade_account.get_trade_indicator().generate_trade_indicators_dataframe()
all_indicators[key + "_obj"] = _executor.trade_account.get_trade_indicator()
return_value.update({"report": all_reports, "indicator": all_indicators})
return_value.update({"portfolio_metrics": all_portfolio_metrics, "indicator": all_indicators})

2
qlib/backtest/order.py → qlib/backtest/decision.py
View File

@@ -1,6 +1,6 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
# TODO: rename it with decision.py
from __future__ import annotations
from enum import IntEnum
from qlib.data.data import Cal

98
qlib/backtest/exchange.py
View File

@@ -15,10 +15,9 @@ import pandas as pd
from ..data.data import D
from ..config import C, REG_CN
from ..utils.resam import resam_ts_data, ts_data_last
from ..log import get_module_logger
from .order import Order, OrderDir, OrderHelper
from .high_performance_ds import BaseQuote, PandasQuote, CN1minNumpyQuote
from .decision import Order, OrderDir, OrderHelper
from .high_performance_ds import BaseQuote, PandasQuote, NumpyQuote
class Exchange:
@@ -36,29 +35,24 @@ class Exchange:
close_cost=0.0025,
min_cost=5,
extra_quote=None,
quote_cls=CN1minNumpyQuote,
quote_cls=NumpyQuote,
**kwargs,
):
"""__init__
:param freq: frequency of data
:param start_time: closed start time for backtest
:param end_time: closed end time for backtest
:param codes: list stock_id list or a string of instruments(i.e. all, csi500, sse50)
:param deal_price: Union[str, Tuple[str, str], List[str]]
The `deal_price` supports following two types of input
- <deal_price> : str
- (<buy_price>, <sell_price>): Tuple[str] or List[str]
<deal_price>, <buy_price> or <sell_price> := <price>
<price> := str
- for example '$close', '$open', '$vwap' ("close" is OK. `Exchange` will help to prepend
"$" to the expression)
:param subscribe_fields: list, subscribe fields. This expressions will be added to the query and `self.quote`.
It is useful when users want more fields to be queried
:param limit_threshold: Union[Tuple[str, str], float, None]
1) `None`: no limitation
2) float, 0.1 for example, default None
@@ -66,7 +60,6 @@ class Exchange:
<the expression for sell stock limitation>)
`False` value indicates the stock is tradable
`True` value indicates the stock is limited and not tradable
:param volume_threshold: Union[
Dict[
"all": ("cum" or "current", limit_str),
@@ -85,26 +78,22 @@ class Exchange:
- "current" means that this is a real-time value and will not accumulate over time,
so it can be directly used as a capacity limit.
e.g. ("cum", "0.2 * DayCumsum($volume, '9:45', '14:45')"), ("current", "$bidV1")
2) "all" means the volume limits are both buying and selling.
"buy" means the volume limits of buying. "sell" means the volume limits of selling.
Different volume limits will be aggregated with min(). If volume_threshold is only
("cum" or "current", limit_str) instead of a dict, the volume limits are for
both by deault. In other words, it is same as {"all": ("cum" or "current", limit_str)}.
3) e.g. "volume_threshold": {
"all": ("cum", "0.2 * DayCumsum($volume, '9:45', '14:45')"),
"buy": ("current", "$askV1"),
"sell": ("current", "$bidV1"),
}
:param open_cost: cost rate for open, default 0.0015
:param close_cost: cost rate for close, default 0.0025
:param trade_unit: trade unit, 100 for China A market.
None for disable trade unit.
**NOTE**: `trade_unit` is included in the `kwargs`. It is necessary because we must
distinguish `not set` and `disable trade_unit`
:param min_cost: min cost, default 5
:param extra_quote: pandas, dataframe consists of
columns: like ['$vwap', '$close', '$volume', '$factor', 'limit_sell', 'limit_buy'].
@@ -185,7 +174,7 @@ class Exchange:
# init quote by quote_df
self.quote_cls = quote_cls
self.quote: BaseQuote = self.quote_cls(self.quote_df)
self.quote: BaseQuote = self.quote_cls(self.quote_df, freq)
def get_quote_from_qlib(self):
# get stock data from qlib
@@ -273,12 +262,10 @@ class Exchange:
preproccess the volume limit.
get the fields need to get from qlib.
get the volume limit list of buying and selling which is composed of all limits.
Parameters
----------
volume_threshold :
please refer to the doc of exchange.
Returns
-------
fields: set
@@ -287,7 +274,6 @@ class Exchange:
all volume limits of buying.
sell_vol_limit: List[Tuple[str]]
all volume limits of selling.
Raises
------
ValueError
@@ -324,7 +310,6 @@ class Exchange:
- if direction is None, check if tradable for buying and selling.
- if direction == Order.BUY, check the if tradable for buying
- if direction == Order.SELL, check the sell limit for selling.
"""
if direction is None:
buy_limit = self.quote.get_data(stock_id, start_time, end_time, field="limit_buy", method="all")
@@ -372,9 +357,7 @@ class Exchange:
):
"""
Deal order when the actual transaction
the results section in `Order` will be changed.
:param order: Deal the order.
:param trade_account: Trade account to be updated after dealing the order.
:param position: position to be updated after dealing the order.
@@ -393,12 +376,12 @@ class Exchange:
# NOTE: order will be changed in this function
trade_price, trade_val, trade_cost = self._calc_trade_info_by_order(
order, trade_account.current if trade_account else position, dealt_order_amount
order, trade_account.current_position if trade_account else position, dealt_order_amount
)
if order.deal_amount > 1e-5:
# If the order can only be deal 0 amount. Nothing to be updated
if trade_val > 1e-5:
# If the order can only be deal 0 value. Nothing to be updated
# Otherwise, it will result in
# 1) some stock with 0 amount in the position
# 1) some stock with 0 value in the position
# 2) `trade_unit` of trade_cost will be lost in user account
if trade_account:
trade_account.update_order(order=order, trade_val=trade_val, cost=trade_cost, trade_price=trade_price)
@@ -407,16 +390,17 @@ class Exchange:
return trade_val, trade_cost, trade_price
def get_quote_info(self, stock_id, start_time, end_time, method=ts_data_last):
def get_quote_info(self, stock_id, start_time, end_time, method="ts_data_last"):
return self.quote.get_data(stock_id, start_time, end_time, method=method)
def get_close(self, stock_id, start_time, end_time, method=ts_data_last):
def get_close(self, stock_id, start_time, end_time, method="ts_data_last"):
return self.quote.get_data(stock_id, start_time, end_time, field="$close", method=method)
def get_volume(self, stock_id, start_time, end_time, method="sum"):
return self.quote.get_data(stock_id, start_time, end_time, field="$volume", method=method)
def get_volume(self, stock_id, start_time, end_time):
"""get the total deal volume of stock with `stock_id` between the time interval [start_time, end_time)"""
return self.quote.get_data(stock_id, start_time, end_time, field="$volume", method="sum")
def get_deal_price(self, stock_id, start_time, end_time, direction: OrderDir, method=ts_data_last):
def get_deal_price(self, stock_id, start_time, end_time, direction: OrderDir, method="ts_data_last"):
if direction == OrderDir.SELL:
pstr = self.sell_price
elif direction == OrderDir.BUY:
@@ -441,7 +425,7 @@ class Exchange:
assert start_time is not None and end_time is not None, "the time range must be given"
if stock_id not in self.quote.get_all_stock():
return None
return self.quote.get_data(stock_id, start_time, end_time, field="$factor", method=ts_data_last)
return self.quote.get_data(stock_id, start_time, end_time, field="$factor", method="ts_data_last")
def generate_amount_position_from_weight_position(
self, weight_position, cash, start_time, end_time, direction=OrderDir.BUY
@@ -449,7 +433,6 @@ class Exchange:
"""
The generate the target position according to the weight and the cash.
NOTE: All the cash will assigned to the tadable stock.
Parameter:
weight_position : dict {stock_id : weight}; allocate cash by weight_position
among then, weight must be in this range: 0 < weight < 1
@@ -493,7 +476,6 @@ class Exchange:
def get_real_deal_amount(self, current_amount, target_amount, factor):
"""
Calculate the real adjust deal amount when considering the trading unit
:param current_amount:
:param target_amount:
:param factor:
@@ -516,7 +498,6 @@ class Exchange:
def generate_order_for_target_amount_position(self, target_position, current_position, start_time, end_time):
"""
Note: some future information is used in this function
Parameter:
target_position : dict { stock_id : amount }
current_postion : dict { stock_id : amount}
@@ -590,8 +571,10 @@ class Exchange:
value = 0
for stock_id in amount_dict:
if (
self.check_stock_suspended(stock_id=stock_id, start_time=start_time, end_time=end_time) is False
only_tradable is True
and self.check_stock_suspended(stock_id=stock_id, start_time=start_time, end_time=end_time) is False
and self.check_stock_limit(stock_id=stock_id, start_time=start_time, end_time=end_time) is False
or only_tradable is False
):
value += (
self.get_deal_price(
@@ -613,10 +596,8 @@ class Exchange:
def get_amount_of_trade_unit(self, factor: float = None, stock_id: str = None, start_time=None, end_time=None):
"""
get the trade unit of amount based on **factor**
the factor can be given directly or calculated in given time range and stock id.
`factor` has higher priority than `stock_id`, `start_time` and `end_time`
Parameters
----------
factor : float
@@ -641,7 +622,6 @@ class Exchange:
):
"""Parameter
Please refer to the docs of get_amount_of_trade_unit
deal_amount : float, adjusted amount
factor : float, adjusted factor
return : float, real amount
@@ -656,11 +636,9 @@ class Exchange:
def _clip_amount_by_volume(self, order: Order, dealt_order_amount: dict) -> int:
"""parse the capacity limit string and return the actual amount of orders that can be executed.
NOTE:
this function will change the order.deal_amount **inplace**
- This will make the order info more accurate
Parameters
----------
order : Order
@@ -694,7 +672,7 @@ class Exchange:
order.start_time,
order.end_time,
field=limit[1],
method=ts_data_last,
method="ts_data_last",
)
vol_limit_num.append(limit_value - dealt_order_amount[order.stock_id])
else:
@@ -709,12 +687,10 @@ class Exchange:
def _get_buy_amount_by_cash_limit(self, trade_price, cash):
"""return the real order amount after cash limit for buying.
Parameters
----------
trade_price : float
position : cash
Return
----------
float
@@ -735,9 +711,7 @@ class Exchange:
def _calc_trade_info_by_order(self, order, position: Position, dealt_order_amount):
"""
Calculation of trade info
**NOTE**: Order will be changed in this function
:param order:
:param position: Position
:param dealt_order_amount: the dealt order amount dict with the format of {stock_id: float}
@@ -745,18 +719,27 @@ class Exchange:
"""
trade_price = self.get_deal_price(order.stock_id, order.start_time, order.end_time, direction=order.direction)
order.factor = self.get_factor(order.stock_id, order.start_time, order.end_time)
order.deal_amount = order.amount # set to full amount and clip it step by step
# Clipping amount first
# - It simulates that the order is rejected directly by the exchange due to large order
# Another choice is placing it after rounding the order
# - It simulates that the large order is submitted, but partial is dealt regardless of rounding by trading unit.
self._clip_amount_by_volume(order, dealt_order_amount)
if order.direction == Order.SELL:
cost_ratio = self.close_cost
# sell
# if we don't know current position, we choose to sell all
# Otherwise, we clip the amount based on current position
if position is not None:
current_amount = (
position.get_stock_amount(order.stock_id) if position.check_stock(order.stock_id) else 0
)
if np.isclose(order.amount, current_amount):
# when selling last stock. The amount don't need rounding
order.deal_amount = order.amount
else:
order.deal_amount = self.round_amount_by_trade_unit(min(current_amount, order.amount), order.factor)
if not np.isclose(order.deal_amount, current_amount):
# when not selling last stock. rounding is necessary
order.deal_amount = self.round_amount_by_trade_unit(
min(current_amount, order.deal_amount), order.factor
)
# in case of negative value of cash
if position.get_cash() + order.deal_amount * trade_price < max(
@@ -765,33 +748,30 @@ class Exchange:
):
order.deal_amount = 0
self.logger.debug(f"Order clipped due to cash limitation: {order}")
else:
# TODO: We don't know current position.
# We choose to sell all
order.deal_amount = order.amount
elif order.direction == Order.BUY:
cost_ratio = self.open_cost
# buy
if position is not None:
cash = position.get_cash()
trade_val = order.amount * trade_price
trade_val = order.deal_amount * trade_price
if cash < trade_val + max(trade_val * cost_ratio, self.min_cost):
# The money is not enough
max_buy_amount = self._get_buy_amount_by_cash_limit(trade_price, cash)
order.deal_amount = self.round_amount_by_trade_unit(max_buy_amount, order.factor)
order.deal_amount = self.round_amount_by_trade_unit(
min(max_buy_amount, order.deal_amount), order.factor
)
self.logger.debug(f"Order clipped due to cash limitation: {order}")
else:
# The money is enough
order.deal_amount = self.round_amount_by_trade_unit(order.amount, order.factor)
order.deal_amount = self.round_amount_by_trade_unit(order.deal_amount, order.factor)
else:
# Unknown amount of money. Just round the amount
order.deal_amount = self.round_amount_by_trade_unit(order.amount, order.factor)
order.deal_amount = self.round_amount_by_trade_unit(order.deal_amount, order.factor)
else:
raise NotImplementedError("order type {} error".format(order.type))
self._clip_amount_by_volume(order, dealt_order_amount)
trade_val = order.deal_amount * trade_price
trade_cost = max(trade_val * cost_ratio, self.min_cost)
if trade_val <= 1e-5:

28
qlib/backtest/executor.py
View File

@@ -11,7 +11,7 @@ from collections import defaultdict
from qlib.backtest.report import Indicator
from .order import EmptyTradeDecision, Order, BaseTradeDecision
from .decision import EmptyTradeDecision, Order, BaseTradeDecision
from .exchange import Exchange
from .utils import TradeCalendarManager, CommonInfrastructure, LevelInfrastructure, get_start_end_idx
@@ -29,7 +29,7 @@ class BaseExecutor:
start_time: Union[str, pd.Timestamp] = None,
end_time: Union[str, pd.Timestamp] = None,
indicator_config: dict = {},
generate_report: bool = False,
generate_portfolio_metrics: bool = False,
verbose: bool = False,
track_data: bool = False,
trade_exchange: Exchange = None,
@@ -77,8 +77,8 @@ class BaseExecutor:
'weight_method': 'value_weighted',
}
}
generate_report : bool, optional
whether to generate report, by default False
generate_portfolio_metrics : bool, optional
whether to generate portfolio_metrics, by default False
verbose : bool, optional
whether to print trading info, by default False
track_data : bool, optional
@@ -87,8 +87,8 @@ class BaseExecutor:
- Else, `trade_decision` will not be generated
trade_exchange : Exchange
exchange that provides market info, used to generate report
- If generate_report is None, trade_exchange will be ignored
exchange that provides market info, used to generate portfolio_metrics
- If generate_portfolio_metrics is None, trade_exchange will be ignored
- Else If `trade_exchange` is None, self.trade_exchange will be set with common_infra
common_infra : CommonInfrastructure, optional:
@@ -103,7 +103,7 @@ class BaseExecutor:
"""
self.time_per_step = time_per_step
self.indicator_config = indicator_config
self.generate_report = generate_report
self.generate_portfolio_metrics = generate_portfolio_metrics
self.verbose = verbose
self.track_data = track_data
self._trade_exchange = trade_exchange
@@ -132,7 +132,7 @@ class BaseExecutor:
# NOTE: there is a trick in the code.
# copy is used instead of deepcopy. So positions are shared
self.trade_account: Account = copy.copy(common_infra.get("trade_account"))
self.trade_account.reset(freq=self.time_per_step, init_report=True, port_metr_enabled=self.generate_report)
self.trade_account.reset(freq=self.time_per_step, port_metr_enabled=self.generate_portfolio_metrics)
@property
def trade_exchange(self) -> Exchange:
@@ -246,7 +246,7 @@ class BaseExecutor:
raise ValueError("atomic executor doesn't support specify `range_limit`")
if self._settle_type != BasePosition.ST_NO:
self.trade_account.current.settle_start(self._settle_type)
self.trade_account.current_position.settle_start(self._settle_type)
obj = self._collect_data(trade_decision=trade_decision, level=level)
@@ -271,7 +271,7 @@ class BaseExecutor:
self.trade_calendar.step()
if self._settle_type != BasePosition.ST_NO:
self.trade_account.current.settle_commit()
self.trade_account.current_position.settle_commit()
if return_value is not None:
return_value.update({"execute_result": res})
@@ -296,7 +296,7 @@ class NestedExecutor(BaseExecutor):
start_time: Union[str, pd.Timestamp] = None,
end_time: Union[str, pd.Timestamp] = None,
indicator_config: dict = {},
generate_report: bool = False,
generate_portfolio_metrics: bool = False,
verbose: bool = False,
track_data: bool = False,
skip_empty_decision: bool = True,
@@ -335,7 +335,7 @@ class NestedExecutor(BaseExecutor):
start_time=start_time,
end_time=end_time,
indicator_config=indicator_config,
generate_report=generate_report,
generate_portfolio_metrics=generate_portfolio_metrics,
verbose=verbose,
track_data=track_data,
common_infra=common_infra,
@@ -444,7 +444,7 @@ class SimulatorExecutor(BaseExecutor):
start_time: Union[str, pd.Timestamp] = None,
end_time: Union[str, pd.Timestamp] = None,
indicator_config: dict = {},
generate_report: bool = False,
generate_portfolio_metrics: bool = False,
verbose: bool = False,
track_data: bool = False,
common_infra: CommonInfrastructure = None,
@@ -462,7 +462,7 @@ class SimulatorExecutor(BaseExecutor):
start_time=start_time,
end_time=end_time,
indicator_config=indicator_config,
generate_report=generate_report,
generate_portfolio_metrics=generate_portfolio_metrics,
verbose=verbose,
track_data=track_data,
common_infra=common_infra,

39
qlib/backtest/high_performance_ds.py
View File

@@ -1,7 +1,6 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
from functools import lru_cache
import logging
from typing import List, Text, Union, Callable, Iterable, Dict
@@ -14,12 +13,12 @@ import numpy as np
from ..utils.index_data import IndexData, SingleData
from ..utils.resam import resam_ts_data, ts_data_last
from ..log import get_module_logger
from ..utils.time import is_single_value
from ..utils.time import is_single_value, Freq
import qlib.utils.index_data as idd
class BaseQuote:
def __init__(self, quote_df: pd.DataFrame):
def __init__(self, quote_df: pd.DataFrame, freq):
self.logger = get_module_logger("online operator", level=logging.INFO)
def get_all_stock(self) -> Iterable:
@@ -39,7 +38,7 @@ class BaseQuote:
start_time: Union[pd.Timestamp, str],
end_time: Union[pd.Timestamp, str],
field: Union[str],
method: Union[str, Callable, None] = None,
method: Union[str, None] = None,
) -> Union[None, int, float, bool, IndexData]:
"""get the specific field of stock data during start time and end_time,
and apply method to the data.
@@ -83,9 +82,9 @@ class BaseQuote:
closed end time for backtest
field : str
the columns of data to fetch
method : Union[str, Callable, None]
method : Union[str, None]
the method apply to data.
e.g [None, "last", "all", "sum", "mean", qlib/utils/resam.py/ts_data_last]
e.g [None, "last", "all", "sum", "mean", "ts_data_last"]
Return
----------
@@ -99,8 +98,8 @@ class BaseQuote:
class PandasQuote(BaseQuote):
def __init__(self, quote_df: pd.DataFrame):
super().__init__(quote_df=quote_df)
def __init__(self, quote_df: pd.DataFrame, freq):
super().__init__(quote_df=quote_df, freq=freq)
quote_dict = {}
for stock_id, stock_val in quote_df.groupby(level="instrument"):
quote_dict[stock_id] = stock_val.droplevel(level="instrument")
@@ -110,6 +109,8 @@ class PandasQuote(BaseQuote):
return self.data.keys()
def get_data(self, stock_id, start_time, end_time, field, method=None):
if method == "ts_data_last":
method = ts_data_last
stock_data = resam_ts_data(self.data[stock_id][field], start_time, end_time, method=method)
if stock_data is None:
return None
@@ -121,9 +122,9 @@ class PandasQuote(BaseQuote):
raise ValueError(f"stock data from resam_ts_data must be a number, pd.Series or pd.DataFrame")
class CN1minNumpyQuote(BaseQuote):
def __init__(self, quote_df: pd.DataFrame):
"""CN1minNumpyQuote
class NumpyQuote(BaseQuote):
def __init__(self, quote_df: pd.DataFrame, freq, region="cn"):
"""NumpyQuote
Parameters
----------
@@ -131,13 +132,19 @@ class CN1minNumpyQuote(BaseQuote):
the init dataframe from qlib.
self.data : Dict(stock_id, IndexData.DataFrame)
"""
super().__init__(quote_df=quote_df)
super().__init__(quote_df=quote_df, freq=freq)
quote_dict = {}
for stock_id, stock_val in quote_df.groupby(level="instrument"):
quote_dict[stock_id] = idd.MultiData(stock_val.droplevel(level="instrument"))
quote_dict[stock_id].sort_index() # To support more flexible slicing, we must sort data first
self.data = quote_dict
self.freq = pd.Timedelta(minutes=1)
n, unit = Freq.parse(freq)
if unit in Freq.SUPPORT_CAL_LIST:
self.freq = Freq.get_timedelta(1, unit)
else:
raise ValueError(f"{freq} is not supported in NumpyQuote")
self.region = region
def get_all_stock(self):
return self.data.keys()
@@ -150,7 +157,7 @@ class CN1minNumpyQuote(BaseQuote):
# single data
# If it don't consider the classification of single data, it will consume a lot of time.
if is_single_value(start_time, end_time, self.freq):
if is_single_value(start_time, end_time, self.freq, self.region):
# this is a very special case.
# skip aggregating function to speed-up the query calculation
try:
@@ -178,9 +185,7 @@ class CN1minNumpyQuote(BaseQuote):
return data[-1]
elif method == "all":
return data.all()
elif method == "any":
return data.any()
elif method == ts_data_last:
elif method == "ts_data_last":
valid_data = data.loc[~data.isna().data.astype(bool)]
if len(valid_data) == 0:
return None

9
qlib/backtest/position.py
View File

@@ -10,7 +10,7 @@ import pandas as pd
from datetime import timedelta
import numpy as np
from .order import Order
from .decision import Order
from ..data.data import D
@@ -151,7 +151,8 @@ class BasePosition:
def get_stock_weight_dict(self, only_stock: bool = False) -> Dict:
"""
generate stock weight dict {stock_id : value weight of stock in the position}
it is meaningful in the beginning or the end of each trade date
it is meaningful in the beginning or the end of each trade step
- During execution of each trading step, the weight may be not consistant with the portfolio value
Parameters
----------
@@ -408,7 +409,7 @@ class Position(BasePosition):
return self.position[code]["price"]
def get_stock_amount(self, code):
return self.position[code]["amount"]
return self.position[code]["amount"] if code in self.position else 0
def get_stock_count(self, code, bar):
"""the days the account has been hold, it may be used in some special strategies"""
@@ -531,7 +532,7 @@ class InfPosition(BasePosition):
raise NotImplementedError(f"InfPosition doesn't support get_stock_weight_dict")
def add_count_all(self, bar):
raise NotImplementedError(f"InfPosition doesn't support get_stock_weight_dict")
raise NotImplementedError(f"InfPosition doesn't support add_count_all")
def update_weight_all(self):
raise NotImplementedError(f"InfPosition doesn't support update_weight_all")

10
qlib/backtest/profit_attribution.py
View File

@@ -18,6 +18,7 @@ def get_benchmark_weight(
start_date=None,
end_date=None,
path=None,
freq="day",
):
"""get_benchmark_weight
@@ -27,6 +28,7 @@ def get_benchmark_weight(
:param start_date:
:param end_date:
:param path:
:param freq:
:return: The weight distribution of the the benchmark described by a pandas dataframe
Every row corresponds to a trading day.
@@ -35,7 +37,7 @@ def get_benchmark_weight(
"""
if not path:
path = Path(C.get_data_path()).expanduser() / "raw" / "AIndexMembers" / "weights.csv"
path = Path(C.dpm.get_data_uri(freq)).expanduser() / "raw" / "AIndexMembers" / "weights.csv"
# TODO: the storage of weights should be implemented in a more elegent way
# TODO: The benchmark is not consistant with the filename in instruments.
bench_weight_df = pd.read_csv(path, usecols=["code", "date", "index", "weight"])
@@ -224,6 +226,7 @@ def brinson_pa(
group_method="category",
group_n=None,
deal_price="vwap",
freq="day",
):
"""brinson profit attribution
@@ -245,7 +248,7 @@ def brinson_pa(
start_date, end_date = min(dates), max(dates)
bench_stock_weight = get_benchmark_weight(bench, start_date, end_date)
bench_stock_weight = get_benchmark_weight(bench, start_date, end_date, freq)
# The attributes for allocation will not
if not group_field.startswith("$"):
@@ -261,13 +264,14 @@ def brinson_pa(
start_time=shift_start_date,
end_time=end_date,
as_list=True,
freq=freq,
)
stock_df = D.features(
instruments,
[group_field, deal_price],
start_time=shift_start_date,
end_time=end_date,
freq="day",
freq=freq,
)
stock_df.columns = [group_field, "deal_price"]

73
qlib/backtest/report.py
View File

@@ -10,21 +10,24 @@ import numpy as np
import pandas as pd
from qlib.backtest.exchange import Exchange
from qlib.backtest.order import BaseTradeDecision, Order, OrderDir
from .decision import IdxTradeRange
from qlib.backtest.decision import BaseTradeDecision, Order, OrderDir
from qlib.backtest.utils import TradeCalendarManager
from .high_performance_ds import BaseOrderIndicator, PandasOrderIndicator, NumpyOrderIndicator, SingleMetric
from ..data import D
from ..tests.config import CSI300_BENCH
from ..utils.resam import get_higher_eq_freq_feature, resam_ts_data
from .order import IdxTradeRange
import qlib.utils.index_data as idd
class Report:
class PortfolioMetrics:
"""
Motivation:
Report is for supporting portfolio related metrics.
PortfolioMetrics is for supporting portfolio related metrics.
Implementation:
daily report of the account
daily portfolio metrics of the account
contain those followings: return, cost, turnover, account, cash, bench, value
For each step(bar/day/minute), each column represents
- return: the return of the portfolio generated by strategy **without transaction fee**.
@@ -33,7 +36,7 @@ class Report:
- cash: the amount of cash in user's account.
- bench: the return of the benchmark
- value: the total value of securities/stocks/instruments (cash is excluded).
update report
"""
@@ -79,7 +82,7 @@ class Report:
self.values = OrderedDict() # value for each trade time
self.cashes = OrderedDict()
self.benches = OrderedDict()
self.latest_report_time = None # pd.TimeStamp
self.latest_pm_time = None # pd.TimeStamp
def init_bench(self, freq=None, benchmark_config=None):
if freq is not None:
@@ -123,18 +126,18 @@ class Report:
return len(self.accounts) == 0
def get_latest_date(self):
return self.latest_report_time
return self.latest_pm_time
def get_latest_account_value(self):
return self.accounts[self.latest_report_time]
return self.accounts[self.latest_pm_time]
def get_latest_total_cost(self):
return self.total_costs[self.latest_report_time]
return self.total_costs[self.latest_pm_time]
def get_latest_total_turnover(self):
return self.total_turnovers[self.latest_report_time]
return self.total_turnovers[self.latest_pm_time]
def update_report_record(
def update_portfolio_metrics_record(
self,
trade_start_time=None,
trade_end_time=None,
@@ -169,7 +172,7 @@ class Report:
elif bench_value is None:
bench_value = self._sample_benchmark(self.bench, trade_start_time, trade_end_time)
# update report data
# update pm data
self.accounts[trade_start_time] = account_value
self.returns[trade_start_time] = return_rate
self.total_turnovers[trade_start_time] = total_turnover
@@ -179,30 +182,30 @@ class Report:
self.values[trade_start_time] = stock_value
self.cashes[trade_start_time] = cash
self.benches[trade_start_time] = bench_value
# update latest_report_date
self.latest_report_time = trade_start_time
# finish report update in each step
# update pm
self.latest_pm_time = trade_start_time
# finish pm update in each step
def generate_report_dataframe(self):
report = pd.DataFrame()
report["account"] = pd.Series(self.accounts)
report["return"] = pd.Series(self.returns)
report["total_turnover"] = pd.Series(self.total_turnovers)
report["turnover"] = pd.Series(self.turnovers)
report["total_cost"] = pd.Series(self.total_costs)
report["cost"] = pd.Series(self.costs)
report["value"] = pd.Series(self.values)
report["cash"] = pd.Series(self.cashes)
report["bench"] = pd.Series(self.benches)
report.index.name = "datetime"
return report
def generate_portfolio_metrics_dataframe(self):
pm = pd.DataFrame()
pm["account"] = pd.Series(self.accounts)
pm["return"] = pd.Series(self.returns)
pm["total_turnover"] = pd.Series(self.total_turnovers)
pm["turnover"] = pd.Series(self.turnovers)
pm["total_cost"] = pd.Series(self.total_costs)
pm["cost"] = pd.Series(self.costs)
pm["value"] = pd.Series(self.values)
pm["cash"] = pd.Series(self.cashes)
pm["bench"] = pd.Series(self.benches)
pm.index.name = "datetime"
return pm
def save_report(self, path):
r = self.generate_report_dataframe()
def save_portfolio_metrics(self, path):
r = self.generate_portfolio_metrics_dataframe()
r.to_csv(path)
def load_report(self, path):
"""load report from a file
def load_portfolio_metrics(self, path):
"""load pm from a file
should have format like
columns = ['account', 'return', 'total_turnover', 'turnover', 'cost', 'total_cost', 'value', 'cash', 'bench']
:param
@@ -215,7 +218,7 @@ class Report:
index = r.index
self.init_vars()
for trade_start_time in index:
self.update_report_record(
self.update_portfolio_metrics_record(
trade_start_time=trade_start_time,
account_value=r.loc[trade_start_time]["account"],
cash=r.loc[trade_start_time]["cash"],
@@ -376,8 +379,6 @@ class Indicator:
price = pa_config.get("price", "deal_price").lower()
if decision.trade_range is not None:
if isinstance(decision.trade_range, IdxTradeRange):
raise TypeError(f"IdxTradeRange is not supported")
trade_start_time, trade_end_time = decision.trade_range.clip_time_range(
start_time=trade_start_time, end_time=trade_end_time
)

11
qlib/backtest/utils.py
View File

@@ -1,18 +1,17 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
from __future__ import annotations
import bisect
from qlib.utils.time import epsilon_change
from typing import Union, TYPE_CHECKING, Tuple, Union, List, Set
from typing import TYPE_CHECKING, Tuple, Union
if TYPE_CHECKING:
from qlib.backtest.order import BaseTradeDecision
from qlib.strategy.base import BaseStrategy
from qlib.backtest.decision import BaseTradeDecision
import pandas as pd
import warnings
from ..utils.resam import get_resam_calendar
from ..data.data import Cal
@@ -56,9 +55,9 @@ class TradeCalendarManager:
self.start_time = pd.Timestamp(start_time) if start_time else None
self.end_time = pd.Timestamp(end_time) if end_time else None
_calendar, freq, freq_sam = get_resam_calendar(freq=freq)
_calendar = Cal.calendar(freq=freq)
self._calendar = _calendar
_, _, _start_index, _end_index = Cal.locate_index(start_time, end_time, freq=freq, freq_sam=freq_sam)
_, _, _start_index, _end_index = Cal.locate_index(start_time, end_time, freq=freq)
self.start_index = _start_index
self.end_index = _end_index
self.trade_len = _end_index - _start_index + 1