* Update YahooNormalizeUS1dExtend(#1196)
* Prevent pandas read_csv errors while running update_data_to_bin for US region
* Fix parse_index error while running update_data_to_bin for US region
* prevent pandas.read_csv error on specific symbol names
* Reordering parameters for better rendering
* removes prefix during feature_dir existence checking
* add explanation comments
* Fix the Errors/Warnings when building Qlib's documentation
* Fix
* Fix
* Empty
* Test CI
* Add doc compiling checking to CI
* Fix
* Tries to be consistent with Makefile
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
* fix logging_level: make logging level specified in qlib.init apply to all loggers
* downgrade loglevel in expmanager __init__ to debug (it will be called in each process in multiprocessing operations such as read data)
* correct gramma error
* fix black lint
* use functor to cache loggers and set level
* correct black lint
* correct pylint
* correct pylint
* fix gramma error in doc strings
* fix typos in exchange.py
* fix typos and gramma errors
* fix typo and rename function param to avoid shading python keyword
* remove redundant parathesis; pass kwargs to parent class
* fix pyblack
* further correction
* assign -> be assigned to
* Optimize the implementation of uri
* remove redundant func
* Set the right order of _set_client_uri
* Update qlib/workflow/expm.py
* Simplify client & add test.Add docs; Fix async bug
* Fix comments & pylint
* Improve README
* bug fix: 1) 100 should be used to scale down percentileofscore return to 0-1, not length of array; 2) for (linear) weighted MA(n), weight should be n, n-1, ..., 1 instead of n-1, ..., 0
* use native pandas fucntion for rank
* remove useless import
* require pandas 1.4+
* rank for py37+pandas 1.3.5 compatibility
* lint improvement
* lint black fix
* use hasattr instead of version to check whether rolling.rank is implemented
* update TSDataSampler
* reformat code with black
* use pre-commit to reformat the code
* Add documents
* More docstring
* More Safety
Co-authored-by: Young <afe.young@gmail.com>
* Refine several todos
* CI issues
* Remove Dropna limitation of `quote_df` in Exchange (#1334)
* Remove Dropna limitation of `quote_df` of Exchange
* Impreove docstring
* Fix type error when expression is specified (#1335)
* Refine fill_missing_data()
* Remove several TODO comments
* Add back env for interpreters
* Change Literal import
* Resolve PR comments
* Move to SAOEState
* Add Trainer.get_policy_state_dict()
* Mypy issue
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
* Don't disable existing logger when initializing qlib.
* Add comma in the end of the config line.
* Add comment to the added config.
Co-authored-by: Jinge Wang <jingewang@microsoft.com>
* add missing parameters to doc string in order_generate
* fix some typos in doc strings
* reformat base on code style standard
* Update qlib/backtest/__init__.py
* Update examples/run_all_model.py
* Update examples/run_all_model.py
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
* Add REG_US and REG_TW into test case: test_utils.py.
* Fix black.
* Trigger checks.
* Add REG_US and REG_TW into test case: test_utils.py.
* Fix black.
* Trigger checks.
* RL backtest with simulator
* Minor modification in init_qlib
* Cherry pick PR 1302
* Resolve PR comments
* Fix missing data processing
* Minor bugfix
* Add TODOs and docs
* Add a comment
* feat(data): ✨ add a general highfreq data handler for open source
Add HighFreqOpenHandler and HighFreqOpenBacktestHandler for data pipeline without paused_num
information.
* fix: position of parameter init
* style(data): 💄 rename open to general
* style(data): 💄 lint
* style: 💄 delete useless comment & fix inheritance relation
* style: 💄 lint
* style: 💄 remove duplicated function
Co-authored-by: mingzhehan <v-zhaoxing@Microsoft.com>
* My own implementation of ChangeInstrument Op. There is a newer, simpler
implemenation from remote.
On branch main
Your branch is behind 'origin/main' by 127 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
Changes to be committed:
modified: qlib/data/ops.py
Changes not staged for commit:
modified: qlib/contrib/evaluate.py
modified: qlib/contrib/strategy/signal_strategy.py
modified: qlib/utils/__init__.py
modified: qlib/workflow/cli.py
modified: qlib/workflow/expm.py
Untracked files:
.idea/
------------------------ >8 ------------------------
Do not modify or remove the line above.
Everything below it will be ignored.
diff --git a/qlib/data/ops.py b/qlib/data/ops.py
index bdc032c0..23db25cc 100644
--- a/qlib/data/ops.py
+++ b/qlib/data/ops.py
@@ -32,6 +32,90 @@ except ValueError as e:
np.seterr(invalid="ignore")
+#################### Change instrument ########################
+# In some case, one may want to change to another instrument when calculating, for example
+# calculate beta of a stock with respect to a market index
+# this would require change the calculation of features from the stock (original instrument) to
+# the index (reference instrument)
+# #############################
+
+
+class ChangeInstrument(ExpressionOps):
+ """Change Instrument Operator
+ In some case, one may want to change to another instrument when calculating, for example, to
+ calculate beta of a stock with respect to a market index.
+ This would require changing the calculation of features from the stock (original instrument) to
+ the index (reference instrument)
+ Parameters
+ ----------
+ instrument: new instrument for which the downstream operations should be performed upon.
+ i.e., SH000300 (CSI300 index), or ^GPSC (SP500 index).
+
+ feature: the feature to be calculated for the new instrument.
+ Returns
+ ----------
+ Expression
+ feature operation output
+ """
+
+ def __init__(self, instrument, feature):
+ self.instrument = instrument
+ self.feature = feature
+
+ def __str__(self):
+ return "{}({},{})".format(type(self).__name__, self.instrument, self.feature)
+
+ def load(self, instrument, start_index, end_index, freq):
+ """load feature
+
+ Parameters
+ ----------
+ instrument : str
+ instrument code, however, the actual instrument loaded is self.instrument through initialization
+ start_index : str
+ feature start index [in calendar].
+ end_index : str
+ feature end index [in calendar].
+ freq : str
+ feature frequency.
+
+ Returns
+ ----------
+ pd.Series
+ feature series: The index of the series is the calendar index
+ """
+ from .cache import H # pylint: disable=C0415
+
+ # cache
+ args = str(self), self.instrument, start_index, end_index, freq
+ if args in H["f"]:
+ return H["f"][args]
+ if start_index is not None and end_index is not None and start_index > end_index:
+ raise ValueError("Invalid index range: {} {}".format(start_index, end_index))
+ try:
+ series = self._load_internal(self.instrument, start_index, end_index, freq)
+ except Exception as e:
+ get_module_logger("data").debug(
+ f"Loading data error: instrument={instrument}, expression={str(self)}, "
+ f"start_index={start_index}, end_index={end_index}, freq={freq}. "
+ f"error info: {str(e)}"
+ )
+ raise
+ series.name = str(self)
+ H["f"][args] = series
+ return series
+
+ def _load_internal(self, instrument, start_index, end_index, freq):
+ series = self.feature.load(self.instrument, start_index, end_index, freq)
+ return series
+
+ def get_longest_back_rolling(self):
+ return self.feature.get_longest_back_rolling()
+
+ def get_extended_window_size(self):
+ return self.feature.get_extended_window_size()
+
+
#################### Element-Wise Operator ####################
@@ -1541,6 +1625,7 @@ class TResample(ElemOperator):
TOpsList = [TResample]
OpsList = [
+ ChangeInstrument,
Rolling,
Ref,
Max,
* update expm.py
* removed duplicate implementation for ChangeInstrument
