mirror of
https://github.com/microsoft/qlib.git
synced 2026-07-14 08:16:54 +08:00
Merge nested main (#597)
* MVP for Indian Stocks in qlib using yahooquery * cleaned with black * cleaned with black * add YahooNormalizeIN and YahooNormalizeIN1d * cleaned the code * added 1min for IN and also updated readme * update comments * fix comments * recorder support upload both raw file and directory * fix comments * Update README.md * Fix docs of QlibRecorder * sort index after loader (#538) make sure the fetch method is based on a index-sorted pd.DataFrame * refactor online serving rolling api * refactor TRA * format by black * fix horizon * fix TRA when use single head * clean up * improve pretrain * update README * fix tra when logdir is None * fix tra when logdir is None * Update strategy.py * Update README.md * Update README.md * Conda Suggestion * code standard docs * Update ensemble.py (#560) * Fix CI Bug (#575) Co-authored-by: yuxwang <anduinnn@foxmail.com> * Update gen.py (#576) * Fix multi-process loop calls (#574) * check lexsort in the 'lazy_sort_index' function (#566) * check lexsort * check lexsort * lexsort comment * lexsort comment * Delete .DS_Store * Update README.md * bug fix & use oracle transport pretrain * mend * Add `backend_freq_config` parameter, support multi-freq uri * Add sample_config to QlibDataLoader, support multi-freq * add multi-freq example * get_cls_kwargs renamed get_callable_kwargs * support multi-freq uri * Add inst_processors to D.features * Fix typo * Fix the index type of the multi-freq example * Fix duplicate mlflow directories in tests * Add DataPathManager to QlibConfig && modify inst_processors to supports list only * Modify the default value in the multi_freq example * Modify client-server mode and dataset-cache to disable inst_processor * Add wheel package to github CI * fix comment * Update FAQ.rst * Update README.md Fix wrong link * Update the docs of TaskManager (#586) * Update manage.py * update yaml * update run_all_model * Modify the Feature to be case sensitive (#589) * update README * remove verbose * fix spell bug * fix typos (#592) * Update Release Note * fix portfolio bug * Add calendar support for resample * add freq kwargs * test.yml: Remove redundant code (#595) * Supporting shared processor (#596) * Supporting shared processor * fix readonly reverse bug * remove pytests dependency * with fit bug * fix parameter error * fix comments * Fix undefined names in Python code (#599) * Update pytorch_tabnet.py $ `flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics` ``` ./qlib/qlib/contrib/model/pytorch_tabnet.py:567:38: F821 undefined name 'inp' self.independ.append(GLU(inp, out_dim, vbs=vbs)) ^ ./qlib/examples/model_rolling/task_manager_rolling.py:75:18: F821 undefined name 'task_train' run_task(task_train, self.task_pool, experiment_name=self.experiment_name) ^ 2 F821 undefined name 'task_train' 2 ``` * Fix undefined names in Python code * from qlib.model.trainer import task_train * update seed * fix some docstring * add comments * Fix SimpleDatasetCache * Update setup.py updated classifiers * Update setup.py change to matplotlib==3.3 * Update python-publish.yml added python 3.9 * updategrade version number * Update model list * fix the type of filter_pipe * fix comment * fix record_temp * update cvxpy version * Update code_standard.rst (#587) * Update code_standard.rst * Update docs/developer/code_standard.rst Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> * Add file lock for MLflowExpManager (#619) * fix torch version * Share version number (#620) * Update initialization.rst (#622) * Update initialization.rst * Update docs/start/initialization.rst Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> * Update docs/start/initialization.rst Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> * fix bugs for running previous exmaple * fix deal amount bug * update change doc (#623) * Add files via upload * Update README.md * Update README.md * Update README.md * Delete change doc.gif * Add files via upload * Update README.md * Delete change doc.gif * Add files via upload * Delete change doc.gif * Add files via upload * Update README.md Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> * update doc * simplify run all model * fix run all model bug * Fix Models (#483) * fix gat dataset * fix tft model * Update tft.py * Fix tft.py Co-authored-by: Pengrong Zhu <zhu.pengrong@foxmail.com> * type and skip empty exp * fix model yaml config * fix tft import bug * skip empty result * fix model and yaml bug * fix wrong generate parameter * Modify multi-freq example (#626) * modify the example of multi-freq * add Copyright * add a comment to average_ops.py * modify the example of multi-freq * add comment to multi_freq_handler.py * add the Ref expression description to multi_freq_handler.py * add expression description to multi_freq_handler.py * update images * fix workflow and update framework Co-authored-by: Gaurav <2796gaurav@gmail.com> Co-authored-by: 2796gaurav <17353992+2796gaurav@users.noreply.github.com> Co-authored-by: bxdd <bxd98@126.com> Co-authored-by: Young <afe.young@gmail.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: Dong Zhou <Zhou.Dong@microsoft.com> Co-authored-by: ZhangTP1996 <ztp18@mails.tsinghua.edu.cn> Co-authored-by: demon143 <59681577+demon143@users.noreply.github.com> Co-authored-by: Wangwuyi123 <51237097+Wangwuyi123@users.noreply.github.com> Co-authored-by: yuxwang <anduinnn@foxmail.com> Co-authored-by: Pengrong Zhu <zhu.pengrong@foxmail.com> Co-authored-by: Mark Zhao <50850474+markzhao98@users.noreply.github.com> Co-authored-by: cslwqxx <cslwqxx@users.noreply.github.com> Co-authored-by: Dong Zhou <evanzd@users.noreply.github.com> Co-authored-by: SaintMalik <37118134+saintmalik@users.noreply.github.com> Co-authored-by: Christian Clauss <cclauss@me.com> Co-authored-by: Anurag Kumar <mailanu98@gmail.com> Co-authored-by: demon143 <785696300@qq.com>
This commit is contained in:
wangwenxi-handsome
committed by
GitHub
GitHub
parent
163e3c6266
commit
3760a18a8d
@@ -4,22 +4,19 @@ from __future__ import annotations
|
||||
import copy
|
||||
from typing import Dict, List, Tuple, TYPE_CHECKING
|
||||
from qlib.utils import init_instance_by_config
|
||||
import warnings
|
||||
import pandas as pd
|
||||
|
||||
from .position import BasePosition, InfPosition, Position
|
||||
from .report import Report, Indicator
|
||||
from .order import BaseTradeDecision, Order
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .exchange import Exchange
|
||||
from .report import PortfolioMetrics, Indicator
|
||||
from .decision import BaseTradeDecision, Order
|
||||
from .exchange import Exchange
|
||||
|
||||
"""
|
||||
rtn & earning in the Account
|
||||
rtn:
|
||||
from order's view
|
||||
1.change if any order is executed, sell order or buy order
|
||||
2.change at the end of today, (today_clse - stock_price) * amount
|
||||
2.change at the end of today, (today_close - stock_price) * amount
|
||||
earning
|
||||
from value of current position
|
||||
earning will be updated at the end of trade date
|
||||
@@ -32,7 +29,7 @@ rtn & earning in the Account
|
||||
|
||||
|
||||
class AccumulatedInfo:
|
||||
"""accumulated trading info, including accumulated return\cost\turnover"""
|
||||
"""accumulated trading info, including accumulated return/cost/turnover"""
|
||||
|
||||
def __init__(self):
|
||||
self.reset()
|
||||
@@ -94,9 +91,12 @@ class Account:
|
||||
|
||||
self._pos_type = pos_type
|
||||
self._port_metr_enabled = port_metr_enabled
|
||||
self.benchmark_config = None # avoid no attribute error
|
||||
self.init_vars(init_cash, position_dict, freq, benchmark_config)
|
||||
|
||||
def init_vars(self, init_cash, position_dict, freq: str, benchmark_config: dict):
|
||||
self.init_cash = init_cash
|
||||
self.current: BasePosition = init_instance_by_config(
|
||||
self.current_position: BasePosition = init_instance_by_config(
|
||||
{
|
||||
"class": self._pos_type,
|
||||
"kwargs": {
|
||||
@@ -106,37 +106,33 @@ class Account:
|
||||
"module_path": "qlib.backtest.position",
|
||||
}
|
||||
)
|
||||
self.report = None
|
||||
self.positions = {}
|
||||
|
||||
# in of reset ignore None values
|
||||
self.benchmark_config = benchmark_config
|
||||
self.freq = freq
|
||||
|
||||
self.reset(freq=freq, benchmark_config=benchmark_config, init_report=True)
|
||||
self.portfolio_metrics = None
|
||||
self.hist_positions = {}
|
||||
self.reset(freq=freq, benchmark_config=benchmark_config)
|
||||
|
||||
def is_port_metr_enabled(self):
|
||||
"""
|
||||
Is portfolio-based metrics enabled.
|
||||
"""
|
||||
return self._port_metr_enabled and not self.current.skip_update()
|
||||
return self._port_metr_enabled and not self.current_position.skip_update()
|
||||
|
||||
def reset_report(self, freq, benchmark_config):
|
||||
# portfolio related metrics
|
||||
if self.is_port_metr_enabled():
|
||||
self.accum_info = AccumulatedInfo()
|
||||
self.report = Report(freq, benchmark_config)
|
||||
self.positions = {}
|
||||
self.portfolio_metrics = PortfolioMetrics(freq, benchmark_config)
|
||||
self.hist_positions = {}
|
||||
|
||||
# fill stock value
|
||||
# The frequency of account may not align with the trading frequency.
|
||||
# This may result in obscure bugs when data quality is low.
|
||||
if isinstance(self.benchmark_config, dict) and self.benchmark_config.get("start_time") is not None:
|
||||
self.current.fill_stock_value(self.benchmark_config["start_time"], self.freq)
|
||||
self.current_position.fill_stock_value(self.benchmark_config["start_time"], self.freq)
|
||||
|
||||
# trading related metrics(e.g. high-frequency trading)
|
||||
self.indicator = Indicator()
|
||||
|
||||
def reset(self, freq=None, benchmark_config=None, init_report=False, port_metr_enabled: bool = None):
|
||||
def reset(self, freq=None, benchmark_config=None, port_metr_enabled: bool = None):
|
||||
"""reset freq and report of account
|
||||
|
||||
Parameters
|
||||
@@ -145,27 +141,23 @@ class Account:
|
||||
frequency of account & report, by default None
|
||||
benchmark_config : {}, optional
|
||||
benchmark config of report, by default None
|
||||
init_report : bool, optional
|
||||
whether to initialize the report, by default False
|
||||
"""
|
||||
if freq is not None:
|
||||
self.freq = freq
|
||||
if benchmark_config is not None:
|
||||
self.benchmark_config = benchmark_config
|
||||
|
||||
if port_metr_enabled is not None:
|
||||
self._port_metr_enabled = port_metr_enabled
|
||||
|
||||
if freq is not None or benchmark_config is not None or init_report:
|
||||
self.reset_report(self.freq, self.benchmark_config)
|
||||
self.reset_report(self.freq, self.benchmark_config)
|
||||
|
||||
def get_positions(self):
|
||||
return self.positions
|
||||
def get_hist_positions(self):
|
||||
return self.hist_positions
|
||||
|
||||
def get_cash(self):
|
||||
return self.current.get_cash()
|
||||
return self.current_position.get_cash()
|
||||
|
||||
def _update_accum_info_from_order(self, order, trade_val, cost, trade_price):
|
||||
def _update_state_from_order(self, order, trade_val, cost, trade_price):
|
||||
if self.is_port_metr_enabled():
|
||||
# update turnover
|
||||
self.accum_info.add_turnover(trade_val)
|
||||
@@ -176,17 +168,17 @@ class Account:
|
||||
trade_amount = trade_val / trade_price
|
||||
if order.direction == Order.SELL: # 0 for sell
|
||||
# when sell stock, get profit from price change
|
||||
profit = trade_val - self.current.get_stock_price(order.stock_id) * trade_amount
|
||||
profit = trade_val - self.current_position.get_stock_price(order.stock_id) * trade_amount
|
||||
self.accum_info.add_return_value(profit) # note here do not consider cost
|
||||
|
||||
elif order.direction == Order.BUY: # 1 for buy
|
||||
# when buy stock, we get return for the rtn computing method
|
||||
# profit in buy order is to make rtn is consistent with earning at the end of bar
|
||||
profit = self.current.get_stock_price(order.stock_id) * trade_amount - trade_val
|
||||
profit = self.current_position.get_stock_price(order.stock_id) * trade_amount - trade_val
|
||||
self.accum_info.add_return_value(profit) # note here do not consider cost
|
||||
|
||||
def update_order(self, order, trade_val, cost, trade_price):
|
||||
if self.current.skip_update():
|
||||
if self.current_position.skip_update():
|
||||
# TODO: supporting polymorphism for account
|
||||
# updating order for infinite position is meaningless
|
||||
return
|
||||
@@ -196,65 +188,61 @@ class Account:
|
||||
# The cost will be substracted from the cash at last. So the trading logic can ignore the cost calculation
|
||||
if order.direction == Order.SELL:
|
||||
# sell stock
|
||||
self._update_accum_info_from_order(order, trade_val, cost, trade_price)
|
||||
self._update_state_from_order(order, trade_val, cost, trade_price)
|
||||
# update current position
|
||||
# for may sell all of stock_id
|
||||
self.current.update_order(order, trade_val, cost, trade_price)
|
||||
self.current_position.update_order(order, trade_val, cost, trade_price)
|
||||
else:
|
||||
# buy stock
|
||||
# deal order, then update state
|
||||
self.current.update_order(order, trade_val, cost, trade_price)
|
||||
self._update_accum_info_from_order(order, trade_val, cost, trade_price)
|
||||
self.current_position.update_order(order, trade_val, cost, trade_price)
|
||||
self._update_state_from_order(order, trade_val, cost, trade_price)
|
||||
|
||||
def update_bar_count(self):
|
||||
"""at the end of the trading bar, update holding bar, count of stock"""
|
||||
# update holding day count
|
||||
# NOTE: updating bar_count does not only serve portfolio metrics, it also serve the strategy
|
||||
if not self.current.skip_update():
|
||||
self.current.add_count_all(bar=self.freq)
|
||||
|
||||
def update_current(self, trade_start_time, trade_end_time, trade_exchange):
|
||||
"""update current to make rtn consistent with earning at the end of bar"""
|
||||
def update_current_position(self, trade_start_time, trade_end_time, trade_exchange):
|
||||
"""update current to make rtn consistent with earning at the end of bar, and update holding bar count of stock"""
|
||||
# update price for stock in the position and the profit from changed_price
|
||||
# NOTE: updating position does not only serve portfolio metrics, it also serve the strategy
|
||||
if not self.current.skip_update():
|
||||
stock_list = self.current.get_stock_list()
|
||||
if not self.current_position.skip_update():
|
||||
stock_list = self.current_position.get_stock_list()
|
||||
for code in stock_list:
|
||||
# if suspend, no new price to be updated, profit is 0
|
||||
if trade_exchange.check_stock_suspended(code, trade_start_time, trade_end_time):
|
||||
continue
|
||||
bar_close = trade_exchange.get_close(code, trade_start_time, trade_end_time)
|
||||
self.current.update_stock_price(stock_id=code, price=bar_close)
|
||||
self.current_position.update_stock_price(stock_id=code, price=bar_close)
|
||||
# update holding day count
|
||||
# NOTE: updating bar_count does not only serve portfolio metrics, it also serve the strategy
|
||||
self.current_position.add_count_all(bar=self.freq)
|
||||
|
||||
def update_report(self, trade_start_time, trade_end_time):
|
||||
"""update position history, report"""
|
||||
def update_portfolio_metrics(self, trade_start_time, trade_end_time):
|
||||
"""update portfolio_metrics"""
|
||||
# calculate earning
|
||||
# account_value - last_account_value
|
||||
# for the first trade date, account_value - init_cash
|
||||
# self.report.is_empty() to judge is_first_trade_date
|
||||
# self.portfolio_metrics.is_empty() to judge is_first_trade_date
|
||||
# get last_account_value, last_total_cost, last_total_turnover
|
||||
if self.report.is_empty():
|
||||
if self.portfolio_metrics.is_empty():
|
||||
last_account_value = self.init_cash
|
||||
last_total_cost = 0
|
||||
last_total_turnover = 0
|
||||
else:
|
||||
last_account_value = self.report.get_latest_account_value()
|
||||
last_total_cost = self.report.get_latest_total_cost()
|
||||
last_total_turnover = self.report.get_latest_total_turnover()
|
||||
last_account_value = self.portfolio_metrics.get_latest_account_value()
|
||||
last_total_cost = self.portfolio_metrics.get_latest_total_cost()
|
||||
last_total_turnover = self.portfolio_metrics.get_latest_total_turnover()
|
||||
# get now_account_value, now_stock_value, now_earning, now_cost, now_turnover
|
||||
now_account_value = self.current.calculate_value()
|
||||
now_stock_value = self.current.calculate_stock_value()
|
||||
now_account_value = self.current_position.calculate_value()
|
||||
now_stock_value = self.current_position.calculate_stock_value()
|
||||
now_earning = now_account_value - last_account_value
|
||||
now_cost = self.accum_info.get_cost - last_total_cost
|
||||
now_turnover = self.accum_info.get_turnover - last_total_turnover
|
||||
# update report for today
|
||||
# update portfolio_metrics for today
|
||||
# judge whether the the trading is begin.
|
||||
# and don't add init account state into report, due to we don't have excess return in those days.
|
||||
self.report.update_report_record(
|
||||
# and don't add init account state into portfolio_metrics, due to we don't have excess return in those days.
|
||||
self.portfolio_metrics.update_portfolio_metrics_record(
|
||||
trade_start_time=trade_start_time,
|
||||
trade_end_time=trade_end_time,
|
||||
account_value=now_account_value,
|
||||
cash=self.current.position["cash"],
|
||||
cash=self.current_position.position["cash"],
|
||||
return_rate=(now_earning + now_cost) / last_account_value,
|
||||
# here use earning to calculate return, position's view, earning consider cost, true return
|
||||
# in order to make same definition with original backtest in evaluate.py
|
||||
@@ -264,12 +252,51 @@ class Account:
|
||||
cost_rate=now_cost / last_account_value,
|
||||
stock_value=now_stock_value,
|
||||
)
|
||||
|
||||
def update_hist_positions(self, trade_start_time):
|
||||
"""update history position"""
|
||||
now_account_value = self.current_position.calculate_value()
|
||||
# set now_account_value to position
|
||||
self.current.position["now_account_value"] = now_account_value
|
||||
self.current.update_weight_all()
|
||||
# update positions
|
||||
self.current_position.position["now_account_value"] = now_account_value
|
||||
self.current_position.update_weight_all()
|
||||
# update hist_positions
|
||||
# note use deepcopy
|
||||
self.positions[trade_start_time] = copy.deepcopy(self.current)
|
||||
self.hist_positions[trade_start_time] = copy.deepcopy(self.current_position)
|
||||
|
||||
def update_indicator(
|
||||
self,
|
||||
trade_start_time: pd.Timestamp,
|
||||
trade_exchange: Exchange,
|
||||
atomic: bool,
|
||||
outer_trade_decision: BaseTradeDecision,
|
||||
trade_info: list = None,
|
||||
inner_order_indicators: List[Dict[str, pd.Series]] = None,
|
||||
decision_list: List[Tuple[BaseTradeDecision, pd.Timestamp, pd.Timestamp]] = None,
|
||||
indicator_config: dict = {},
|
||||
):
|
||||
"""update trade indicators and order indicators in each bar end"""
|
||||
# TODO: will skip empty decisions make it faster? `outer_trade_decision.empty():`
|
||||
|
||||
# indicator is trading (e.g. high-frequency order execution) related analysis
|
||||
self.indicator.reset()
|
||||
|
||||
# aggregate the information for each order
|
||||
if atomic:
|
||||
self.indicator.update_order_indicators(trade_info)
|
||||
else:
|
||||
self.indicator.agg_order_indicators(
|
||||
inner_order_indicators,
|
||||
decision_list=decision_list,
|
||||
outer_trade_decision=outer_trade_decision,
|
||||
trade_exchange=trade_exchange,
|
||||
indicator_config=indicator_config,
|
||||
)
|
||||
|
||||
# aggregate all the order metrics a single step
|
||||
self.indicator.cal_trade_indicators(trade_start_time, self.freq, indicator_config)
|
||||
|
||||
# record the metrics
|
||||
self.indicator.record(trade_start_time)
|
||||
|
||||
def update_bar_end(
|
||||
self,
|
||||
@@ -316,44 +343,34 @@ class Account:
|
||||
elif atomic is False and inner_order_indicators is None:
|
||||
raise ValueError("inner_order_indicators is necessary in un-atomic executor")
|
||||
|
||||
# TODO: `update_bar_count` and `update_current` should placed in Position and be merged.
|
||||
self.update_bar_count()
|
||||
self.update_current(trade_start_time, trade_end_time, trade_exchange)
|
||||
# update current position and hold bar count in each bar end
|
||||
self.update_current_position(trade_start_time, trade_end_time, trade_exchange)
|
||||
|
||||
if self.is_port_metr_enabled():
|
||||
# report is portfolio related analysis
|
||||
self.update_report(trade_start_time, trade_end_time)
|
||||
# portfolio_metrics is portfolio related analysis
|
||||
self.update_portfolio_metrics(trade_start_time, trade_end_time)
|
||||
self.update_hist_positions(trade_start_time)
|
||||
|
||||
# TODO: will skip empty decisions make it faster? `outer_trade_decision.empty():`
|
||||
# update indicator in each bar end
|
||||
self.update_indicator(
|
||||
trade_start_time=trade_start_time,
|
||||
trade_exchange=trade_exchange,
|
||||
atomic=atomic,
|
||||
outer_trade_decision=outer_trade_decision,
|
||||
trade_info=trade_info,
|
||||
inner_order_indicators=inner_order_indicators,
|
||||
decision_list=decision_list,
|
||||
indicator_config=indicator_config,
|
||||
)
|
||||
|
||||
# indicator is trading (e.g. high-frequency order execution) related analysis
|
||||
self.indicator.reset()
|
||||
|
||||
# aggregate the information for each order
|
||||
if atomic:
|
||||
self.indicator.update_order_indicators(trade_info)
|
||||
else:
|
||||
self.indicator.agg_order_indicators(
|
||||
inner_order_indicators,
|
||||
decision_list=decision_list,
|
||||
outer_trade_decision=outer_trade_decision,
|
||||
trade_exchange=trade_exchange,
|
||||
indicator_config=indicator_config,
|
||||
)
|
||||
|
||||
# aggregate all the order metrics a single step
|
||||
self.indicator.cal_trade_indicators(trade_start_time, self.freq, indicator_config)
|
||||
|
||||
# record the metrics
|
||||
self.indicator.record(trade_start_time)
|
||||
|
||||
def get_report(self):
|
||||
"""get the history report and postions instance"""
|
||||
def get_portfolio_metrics(self):
|
||||
"""get the history portfolio_metrics and postions instance"""
|
||||
if self.is_port_metr_enabled():
|
||||
_report = self.report.generate_report_dataframe()
|
||||
_positions = self.get_positions()
|
||||
return _report, _positions
|
||||
_portfolio_metrics = self.portfolio_metrics.generate_portfolio_metrics_dataframe()
|
||||
_positions = self.get_hist_positions()
|
||||
return _portfolio_metrics, _positions
|
||||
else:
|
||||
raise ValueError("generate_report should be True if you want to generate report")
|
||||
raise ValueError("generate_portfolio_metrics should be True if you want to generate portfolio_metrics")
|
||||
|
||||
def get_trade_indicator(self) -> Indicator:
|
||||
"""get the trade indicator instance, which has pa/pos/ffr info."""
|
||||
|
||||
Reference in New Issue
Block a user