* Refine several todos
* CI issues
* Remove Dropna limitation of `quote_df` in Exchange (#1334)
* Remove Dropna limitation of `quote_df` of Exchange
* Impreove docstring
* Fix type error when expression is specified (#1335)
* Refine fill_missing_data()
* Remove several TODO comments
* Add back env for interpreters
* Change Literal import
* Resolve PR comments
* Move to SAOEState
* Add Trainer.get_policy_state_dict()
* Mypy issue
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
* Don't disable existing logger when initializing qlib.
* Add comma in the end of the config line.
* Add comment to the added config.
Co-authored-by: Jinge Wang <jingewang@microsoft.com>
* add missing parameters to doc string in order_generate
* fix some typos in doc strings
* reformat base on code style standard
* Update qlib/backtest/__init__.py
* Update examples/run_all_model.py
* Update examples/run_all_model.py
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
* Add REG_US and REG_TW into test case: test_utils.py.
* Fix black.
* Trigger checks.
* Add REG_US and REG_TW into test case: test_utils.py.
* Fix black.
* Trigger checks.
* RL backtest with simulator
* Minor modification in init_qlib
* Cherry pick PR 1302
* Resolve PR comments
* Fix missing data processing
* Minor bugfix
* Add TODOs and docs
* Add a comment
* feat(data): ✨ add a general highfreq data handler for open source
Add HighFreqOpenHandler and HighFreqOpenBacktestHandler for data pipeline without paused_num
information.
* fix: position of parameter init
* style(data): 💄 rename open to general
* style(data): 💄 lint
* style: 💄 delete useless comment & fix inheritance relation
* style: 💄 lint
* style: 💄 remove duplicated function
Co-authored-by: mingzhehan <v-zhaoxing@Microsoft.com>
* My own implementation of ChangeInstrument Op. There is a newer, simpler
implemenation from remote.
On branch main
Your branch is behind 'origin/main' by 127 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
Changes to be committed:
modified: qlib/data/ops.py
Changes not staged for commit:
modified: qlib/contrib/evaluate.py
modified: qlib/contrib/strategy/signal_strategy.py
modified: qlib/utils/__init__.py
modified: qlib/workflow/cli.py
modified: qlib/workflow/expm.py
Untracked files:
.idea/
------------------------ >8 ------------------------
Do not modify or remove the line above.
Everything below it will be ignored.
diff --git a/qlib/data/ops.py b/qlib/data/ops.py
index bdc032c0..23db25cc 100644
--- a/qlib/data/ops.py
+++ b/qlib/data/ops.py
@@ -32,6 +32,90 @@ except ValueError as e:
np.seterr(invalid="ignore")
+#################### Change instrument ########################
+# In some case, one may want to change to another instrument when calculating, for example
+# calculate beta of a stock with respect to a market index
+# this would require change the calculation of features from the stock (original instrument) to
+# the index (reference instrument)
+# #############################
+
+
+class ChangeInstrument(ExpressionOps):
+ """Change Instrument Operator
+ In some case, one may want to change to another instrument when calculating, for example, to
+ calculate beta of a stock with respect to a market index.
+ This would require changing the calculation of features from the stock (original instrument) to
+ the index (reference instrument)
+ Parameters
+ ----------
+ instrument: new instrument for which the downstream operations should be performed upon.
+ i.e., SH000300 (CSI300 index), or ^GPSC (SP500 index).
+
+ feature: the feature to be calculated for the new instrument.
+ Returns
+ ----------
+ Expression
+ feature operation output
+ """
+
+ def __init__(self, instrument, feature):
+ self.instrument = instrument
+ self.feature = feature
+
+ def __str__(self):
+ return "{}({},{})".format(type(self).__name__, self.instrument, self.feature)
+
+ def load(self, instrument, start_index, end_index, freq):
+ """load feature
+
+ Parameters
+ ----------
+ instrument : str
+ instrument code, however, the actual instrument loaded is self.instrument through initialization
+ start_index : str
+ feature start index [in calendar].
+ end_index : str
+ feature end index [in calendar].
+ freq : str
+ feature frequency.
+
+ Returns
+ ----------
+ pd.Series
+ feature series: The index of the series is the calendar index
+ """
+ from .cache import H # pylint: disable=C0415
+
+ # cache
+ args = str(self), self.instrument, start_index, end_index, freq
+ if args in H["f"]:
+ return H["f"][args]
+ if start_index is not None and end_index is not None and start_index > end_index:
+ raise ValueError("Invalid index range: {} {}".format(start_index, end_index))
+ try:
+ series = self._load_internal(self.instrument, start_index, end_index, freq)
+ except Exception as e:
+ get_module_logger("data").debug(
+ f"Loading data error: instrument={instrument}, expression={str(self)}, "
+ f"start_index={start_index}, end_index={end_index}, freq={freq}. "
+ f"error info: {str(e)}"
+ )
+ raise
+ series.name = str(self)
+ H["f"][args] = series
+ return series
+
+ def _load_internal(self, instrument, start_index, end_index, freq):
+ series = self.feature.load(self.instrument, start_index, end_index, freq)
+ return series
+
+ def get_longest_back_rolling(self):
+ return self.feature.get_longest_back_rolling()
+
+ def get_extended_window_size(self):
+ return self.feature.get_extended_window_size()
+
+
#################### Element-Wise Operator ####################
@@ -1541,6 +1625,7 @@ class TResample(ElemOperator):
TOpsList = [TResample]
OpsList = [
+ ChangeInstrument,
Rolling,
Ref,
Max,
* update expm.py
* removed duplicate implementation for ChangeInstrument
* Refine previous version RL codes
* Polish utils/__init__.py
* Draft
* Use | instead of Union
* Simulator & action interpreter
* Test passed
* Migrate to SAOEState & new qlib interpreter
* Black format
* . Revert file_storage change
* Refactor file structure & renaming functions
* Enrich test cases
* Add QlibIntradayBacktestData
* Test interpreter
* Black format
* .
.
.
* Rename receive_execute_result()
* Use indicator to simplify state update
* Format code
* Modify data path
* Adjust file structure
* Minor change
* Add copyright message
* Format code
* Rename util functions
* Add CI
* Pylint issue
* Remove useless code to pass pylint
* Pass mypy
* Mypy issue
* mypy issue
* mypy issue
* Revert "mypy issue"
This reverts commit 8eb1b0174e.
* mypy issue
* mypy issue
* Fix the numpy version incompatible bug
* Fix a minor typing issue
* Try to skip python 3.7 test for qlib simulator
* Resolve PR comments by Yuge; solve several CI issues.
* Black issue
* Fix a low-level type error
* Change data name
* Resolve PR comments. Leave TODOs in the code base.
Co-authored-by: Young <afe.young@gmail.com>
* Explain data crawler structure
* Add documentation for data and feature
* Update scripts/data_collector/yahoo/README.md
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
* Remove some confusing wording
* Add third party data source
* Fix command typo
* Update commands
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
