mirror of
https://github.com/microsoft/qlib.git
synced 2026-07-11 06:46:56 +08:00
12317 lines
625 KiB
YAML
12317 lines
625 KiB
YAML
- docstring: null
|
|
function: setup.read
|
|
- docstring: null
|
|
function: test_structured_cov_estimator.TestStructuredCovEstimator
|
|
- docstring: null
|
|
function: test_structured_cov_estimator.test_random_covariance
|
|
- docstring: null
|
|
function: test_structured_cov_estimator.test_nan_option_covariance
|
|
- docstring: null
|
|
function: test_structured_cov_estimator.test_decompose_covariance
|
|
- docstring: null
|
|
function: test_structured_cov_estimator.test_constructed_covariance
|
|
- docstring: ' train model
|
|
|
|
Returns-------pred_score: pandas.DataFramepredict scoresperformance: dictmodel
|
|
performance'
|
|
function: test_all_pipeline.train
|
|
- docstring: ' A fake experiment workflow to test uri
|
|
|
|
Returns-------pass_or_not_for_default_uri: boolpass_or_not_for_current_uri: booltemporary_exp_dir:
|
|
str'
|
|
function: test_all_pipeline.fake_experiment
|
|
- docstring: ' backtest and analysis
|
|
|
|
Parameters----------rid : strthe id of the recorder to be used in this functionuri_path:
|
|
strmlflow uri pathReturns-------analysis : pandas.DataFramethe analysis result'
|
|
function: test_all_pipeline.backtest_analysis
|
|
- docstring: null
|
|
function: test_all_pipeline.TestAllFlow
|
|
- docstring: null
|
|
function: test_all_pipeline.tearDownClass
|
|
- docstring: null
|
|
function: test_all_pipeline.test_0_train
|
|
- docstring: null
|
|
function: test_all_pipeline.test_1_backtest
|
|
- docstring: null
|
|
function: test_all_pipeline.test_2_expmanager
|
|
- docstring: null
|
|
function: test_pit.TestPIT
|
|
- docstring: null
|
|
function: test_pit.tearDownClass
|
|
- docstring: null
|
|
function: test_pit.setUpClass
|
|
- docstring: null
|
|
function: test_pit.setUp
|
|
- docstring: null
|
|
function: test_pit.to_str
|
|
- docstring: null
|
|
function: test_pit.check_same
|
|
- docstring: " \nself.check_same(data.describe(), res)res = "
|
|
function: test_pit.test_query
|
|
- docstring: " \nself.check_same(data, expect)@pytest.mark.slow"
|
|
function: test_pit.test_no_exist_data
|
|
- docstring: " \nself.check_same(data.tail(15), expect)"
|
|
function: test_pit.test_expr
|
|
- docstring: " \nself.check_same(s[~s.duplicated().values], expect)"
|
|
function: test_pit.test_unlimit
|
|
- docstring: " \nself.check_same(data, except_data)"
|
|
function: test_pit.test_expr2
|
|
- docstring: null
|
|
function: test_get_data.TestGetData
|
|
- docstring: null
|
|
function: test_get_data.setUpClass
|
|
- docstring: null
|
|
function: test_get_data.tearDownClass
|
|
- docstring: null
|
|
function: test_get_data.test_0_qlib_data
|
|
- docstring: null
|
|
function: test_contrib_model.TestAllFlow
|
|
- docstring: null
|
|
function: test_contrib_model.test_0_initialize
|
|
- docstring: null
|
|
function: test_contrib_workflow.train_multiseg
|
|
- docstring: null
|
|
function: test_contrib_workflow.train_mse
|
|
- docstring: null
|
|
function: test_contrib_workflow.TestAllFlow
|
|
- docstring: null
|
|
function: test_contrib_workflow.tearDownClass
|
|
- docstring: null
|
|
function: test_contrib_workflow.test_0_multiseg
|
|
- docstring: null
|
|
function: test_contrib_workflow.test_1_mse
|
|
- docstring: null
|
|
function: test_workflow.WorkflowTest
|
|
- docstring: null
|
|
function: test_workflow.tearDown
|
|
- docstring: null
|
|
function: test_dump_data.TestDumpData
|
|
- docstring: null
|
|
function: test_dump_data.setUpClass
|
|
- docstring: null
|
|
function: test_dump_data.tearDownClass
|
|
- docstring: null
|
|
function: test_dump_data.test_0_dump_bin
|
|
- docstring: null
|
|
function: test_dump_data.test_1_dump_calendars
|
|
- docstring: null
|
|
function: test_dump_data.test_2_dump_instruments
|
|
- docstring: null
|
|
function: test_dump_data.test_3_dump_features
|
|
- docstring: ' Feature First Difference
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------Expressiona
|
|
feature instance with first difference'
|
|
function: test_register_ops.Diff
|
|
- docstring: null
|
|
function: test_register_ops._load_internal
|
|
- docstring: null
|
|
function: test_register_ops.get_extended_window_size
|
|
- docstring: ' Feature Distance
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------Expressiona
|
|
feature instance with distance'
|
|
function: test_register_ops.Distance
|
|
- docstring: null
|
|
function: test_register_ops._load_internal
|
|
- docstring: null
|
|
function: test_register_ops.TestRegiterCustomOps
|
|
- docstring: null
|
|
function: test_register_ops.setUpClass
|
|
- docstring: null
|
|
function: test_datalayer.TestDataset
|
|
- docstring: null
|
|
function: test_datalayer.testCSI300
|
|
- docstring: null
|
|
function: test_processor.TestProcessor
|
|
- docstring: null
|
|
function: test_processor.test_MinMaxNorm
|
|
- docstring: null
|
|
function: test_processor.normalize
|
|
- docstring: null
|
|
function: test_processor.test_ZScoreNorm
|
|
- docstring: null
|
|
function: test_processor.normalize
|
|
- docstring: null
|
|
function: test_processor.test_CSZFillna
|
|
- docstring: null
|
|
function: test_handler.HandlerTests
|
|
- docstring: null
|
|
function: test_handler.to_str
|
|
- docstring: null
|
|
function: test_handler_storage.TestHandler
|
|
- docstring: null
|
|
function: test_handler_storage.get_feature_config
|
|
- docstring: null
|
|
function: test_handler_storage.TestHandlerStorage
|
|
- docstring: null
|
|
function: test_dataset.TestDataset
|
|
- docstring: null
|
|
function: test_saoe_simple.test_pickle_data_inspect
|
|
- docstring: null
|
|
function: test_saoe_simple.test_simulator_first_step
|
|
- docstring: null
|
|
function: test_saoe_simple.test_simulator_stop_twap
|
|
- docstring: null
|
|
function: test_saoe_simple.test_simulator_stop_early
|
|
- docstring: null
|
|
function: test_saoe_simple.test_simulator_start_middle
|
|
- docstring: null
|
|
function: test_saoe_simple.test_interpreter
|
|
- docstring: null
|
|
function: test_saoe_simple.EmulateEnvWrapper
|
|
- docstring: null
|
|
function: test_saoe_simple.test_network_sanity
|
|
- docstring: null
|
|
function: test_saoe_simple.EmulateEnvWrapper
|
|
- docstring: null
|
|
function: test_saoe_simple.test_twap_strategy
|
|
- docstring: null
|
|
function: test_saoe_simple.test_cn_ppo_strategy
|
|
- docstring: null
|
|
function: test_data_queue.DummyDataset
|
|
- docstring: null
|
|
function: test_data_queue._worker
|
|
- docstring: null
|
|
function: test_data_queue._queue_to_list
|
|
- docstring: null
|
|
function: test_data_queue.test_pytorch_dataloader
|
|
- docstring: null
|
|
function: test_data_queue.test_multiprocess_shared_dataloader
|
|
- docstring: null
|
|
function: test_data_queue.test_exit_on_crash_finite
|
|
- docstring: null
|
|
function: test_data_queue._exit_finite
|
|
- docstring: null
|
|
function: test_data_queue.test_exit_on_crash_infinite
|
|
- docstring: null
|
|
function: test_qlib_simulator.is_close
|
|
- docstring: null
|
|
function: test_qlib_simulator.get_order
|
|
- docstring: null
|
|
function: test_qlib_simulator.get_configs
|
|
- docstring: null
|
|
function: test_qlib_simulator.get_simulator
|
|
- docstring: null
|
|
function: test_qlib_simulator.test_simulator_first_step
|
|
- docstring: null
|
|
function: test_qlib_simulator.test_simulator_stop_twap
|
|
- docstring: null
|
|
function: test_logger.SimpleEnv
|
|
- docstring: null
|
|
function: test_logger.reset
|
|
- docstring: null
|
|
function: test_logger.step
|
|
- docstring: null
|
|
function: test_logger.render
|
|
- docstring: null
|
|
function: test_logger.AnyPolicy
|
|
- docstring: null
|
|
function: test_logger.forward
|
|
- docstring: null
|
|
function: test_logger.learn
|
|
- docstring: null
|
|
function: test_logger.test_simple_env_logger
|
|
- docstring: null
|
|
function: test_logger.SimpleSimulator
|
|
- docstring: null
|
|
function: test_logger.step
|
|
- docstring: null
|
|
function: test_logger.get_state
|
|
- docstring: null
|
|
function: test_logger.done
|
|
- docstring: null
|
|
function: test_logger.DummyStateInterpreter
|
|
- docstring: null
|
|
function: test_logger.interpret
|
|
- docstring: null
|
|
function: test_logger.observation_space
|
|
- docstring: null
|
|
function: test_logger.DummyActionInterpreter
|
|
- docstring: null
|
|
function: test_logger.interpret
|
|
- docstring: null
|
|
function: test_logger.action_space
|
|
- docstring: null
|
|
function: test_logger.RandomFivePolicy
|
|
- docstring: null
|
|
function: test_logger.forward
|
|
- docstring: null
|
|
function: test_logger.learn
|
|
- docstring: null
|
|
function: test_trainer.ZeroSimulator
|
|
- docstring: null
|
|
function: test_trainer.step
|
|
- docstring: null
|
|
function: test_trainer.get_state
|
|
- docstring: null
|
|
function: test_trainer.done
|
|
- docstring: null
|
|
function: test_trainer.NoopStateInterpreter
|
|
- docstring: null
|
|
function: test_trainer.interpret
|
|
- docstring: null
|
|
function: test_trainer.NoopActionInterpreter
|
|
- docstring: null
|
|
function: test_trainer.interpret
|
|
- docstring: null
|
|
function: test_trainer.AccReward
|
|
- docstring: null
|
|
function: test_trainer.reward
|
|
- docstring: null
|
|
function: test_trainer.PolicyNet
|
|
- docstring: null
|
|
function: test_trainer.forward
|
|
- docstring: null
|
|
function: test_trainer._ppo_policy
|
|
- docstring: null
|
|
function: test_trainer.test_trainer
|
|
- docstring: null
|
|
function: test_trainer.test_trainer_fast_dev_run
|
|
- docstring: null
|
|
function: test_trainer.test_trainer_earlystop
|
|
- docstring: null
|
|
function: test_finite_env.FiniteEnv
|
|
- docstring: null
|
|
function: test_finite_env.reset
|
|
- docstring: null
|
|
function: test_finite_env.step
|
|
- docstring: null
|
|
function: test_finite_env.FiniteEnvWithComplexObs
|
|
- docstring: null
|
|
function: test_finite_env.reset
|
|
- docstring: null
|
|
function: test_finite_env.step
|
|
- docstring: null
|
|
function: test_finite_env.DummyDataset
|
|
- docstring: null
|
|
function: test_finite_env.AnyPolicy
|
|
- docstring: null
|
|
function: test_finite_env.forward
|
|
- docstring: null
|
|
function: test_finite_env.learn
|
|
- docstring: null
|
|
function: test_finite_env._finite_env_factory
|
|
- docstring: null
|
|
function: test_finite_env.MetricTracker
|
|
- docstring: null
|
|
function: test_finite_env.on_env_step
|
|
- docstring: null
|
|
function: test_finite_env.validate
|
|
- docstring: null
|
|
function: test_finite_env.DoNothingTracker
|
|
- docstring: null
|
|
function: test_finite_env.on_env_step
|
|
- docstring: null
|
|
function: test_finite_env.test_finite_dummy_vector_env
|
|
- docstring: null
|
|
function: test_finite_env.test_finite_shmem_vector_env
|
|
- docstring: null
|
|
function: test_finite_env.test_finite_subproc_vector_env
|
|
- docstring: null
|
|
function: test_finite_env.test_nan
|
|
- docstring: null
|
|
function: test_finite_env.test_finite_dummy_vector_env_complex
|
|
- docstring: null
|
|
function: test_update_pred.TestRolling
|
|
- docstring: " \nThis test is for testing if it will raise error if the `to_date`\
|
|
\ is out of the boundary."
|
|
function: test_update_pred.test_update_pred
|
|
- docstring: null
|
|
function: test_file_strategy.FileStrTest
|
|
- docstring: null
|
|
function: test_file_strategy._gen_orders
|
|
- docstring: null
|
|
function: test_high_freq_trading.TestHFBacktest
|
|
- docstring: null
|
|
function: test_high_freq_trading.setUpClass
|
|
- docstring: null
|
|
function: test_high_freq_trading._gen_orders
|
|
- docstring: null
|
|
function: test_utils.SingletonTest
|
|
- docstring: null
|
|
function: test_utils.test_singleton
|
|
- docstring: null
|
|
function: test_sumarize.TestSummarize
|
|
- docstring: null
|
|
function: test_sumarize.test_chat
|
|
- docstring: null
|
|
function: test_sumarize.test_execution
|
|
- docstring: null
|
|
function: test_sumarize.test_generate_batch_result
|
|
- docstring: null
|
|
function: test_sumarize.test_parse2txt
|
|
- docstring: null
|
|
function: test_cfg.FincoTpl
|
|
- docstring: ' Motivation: make sure the configuable template is consistent
|
|
with the default config
|
|
|
|
tpl_p = get_tpl_path()with (tpl_p / "sl" / "workflow_config.yaml").open("rb")
|
|
as fp:config = yaml.safe_load(fp)# init_data_handlerhd: DataHandlerLP = init_instance_by_config(config["task"]["dataset"]["kwargs"]["handler"])#
|
|
NOTE: The config in workflow_config.yaml is generated by the following code:#
|
|
dump in yaml format to file without auto linebreak# print(yaml.dump(hd.data_loader.fields,
|
|
width=10000, stream=open("_tmp", "w")))with (tpl_p / "sl-cfg" / "workflow_config.yaml").open("rb")
|
|
as fp:config = yaml.safe_load(fp)hd_ds: DataHandlerLP = init_instance_by_config(config["task"]["dataset"]["kwargs"]["handler"])self.assertEqual(hd_ds.data_loader.fields,
|
|
hd.data_loader.fields)check = hd_ds.fetch().fillna(0.0) == hd.fetch().fillna(0.0)self.assertTrue(check.all().all())'
|
|
function: test_cfg.test_tpl_consistence
|
|
- docstring: null
|
|
function: test_mlflow.MLflowTest
|
|
- docstring: null
|
|
function: test_mlflow.tearDown
|
|
- docstring: " \nPlease refer to qlib/workflow/expm.py:MLflowExpManager._clientwe\
|
|
\ don't cache _client (this is helpful to reduce maintainance work when MLflowExpManager's\
|
|
\ uri is chagned)This implementation is based on the assumption creating a client\
|
|
\ is fast"
|
|
function: test_mlflow.test_creating_client
|
|
- docstring: null
|
|
function: test_elem_operator.TestElementOperator
|
|
- docstring: null
|
|
function: test_elem_operator.setUp
|
|
- docstring: null
|
|
function: test_elem_operator.test_Abs
|
|
- docstring: null
|
|
function: test_elem_operator.test_Sign
|
|
- docstring: null
|
|
function: test_elem_operator.TestOperatorDataSetting
|
|
- docstring: null
|
|
function: test_elem_operator.test_setting
|
|
- docstring: null
|
|
function: test_elem_operator.TestInstElementOperator
|
|
- docstring: null
|
|
function: test_elem_operator.setUp
|
|
- docstring: null
|
|
function: test_special_ops.TestOperatorDataSetting
|
|
- docstring: null
|
|
function: test_special_ops.test_setting
|
|
- docstring: null
|
|
function: test_special_ops.test_case2
|
|
- docstring: " \nSample raw calendar into calendar with sam_minutes freq, shift\
|
|
\ represents the shift minute the market time- open time of stock market is [9:30\
|
|
\ - shift*pd.Timedelta(minutes=1)]- mid close time of stock market is [11:29 -\
|
|
\ shift*pd.Timedelta(minutes=1)]- mid open time of stock market is [13:00 - shift*pd.Timedelta(minutes=1)]-\
|
|
\ close time of stock market is [14:59 - shift*pd.Timedelta(minutes=1)]"
|
|
function: test_utils.cal_sam_minute
|
|
- docstring: null
|
|
function: test_utils.TimeUtils
|
|
- docstring: null
|
|
function: test_utils.setUpClass
|
|
- docstring: null
|
|
function: test_utils.test_cal_sam_minute
|
|
- docstring: null
|
|
function: test_index_data.IndexDataTest
|
|
- docstring: null
|
|
function: test_index_data.test_index_single_data
|
|
- docstring: null
|
|
function: test_index_data.test_index_multi_data
|
|
- docstring: null
|
|
function: test_index_data.test_sorting
|
|
- docstring: null
|
|
function: test_index_data.test_corner_cases
|
|
- docstring: null
|
|
function: test_index_data.test_ops
|
|
- docstring: null
|
|
function: test_index_data.test_todo
|
|
- docstring: null
|
|
function: test_get_multi_proc.get_features
|
|
- docstring: null
|
|
function: test_get_multi_proc.TestGetData
|
|
- docstring: " \nFor testing if it will raise error"
|
|
function: test_get_multi_proc.test_multi_proc
|
|
- docstring: null
|
|
function: test_sepdf.SepDF
|
|
- docstring: null
|
|
function: test_sepdf.to_str
|
|
- docstring: null
|
|
function: test_storage.TestStorage
|
|
- docstring: null
|
|
function: test_storage.test_calendar_storage
|
|
- docstring: " \nThe meaning of instrument, such as CSI500:CSI500 composition\
|
|
\ changes:date add remove2005-01-01 SH6000002005-01-01\
|
|
\ SH6000012005-01-01 SH6000022005-02-01 SH600003 SH6000002005-02-15\
|
|
\ SH600000 SH600002Calendar:pd.date_range(start=\"2020-01-01\", stop=\"\
|
|
2020-03-01\", freq=\"1D\")Instrument:symbol start_time end_timeSH600000\
|
|
\ 2005-01-01 2005-01-31 (2005-02-01 Last trading day)SH600000 2005-02-15\
|
|
\ 2005-03-01SH600001 2005-01-01 2005-03-01SH600002 2005-01-01\
|
|
\ 2005-02-14 (2005-02-15 Last trading day)SH600003 2005-02-01 2005-03-01InstrumentStorage:{\"\
|
|
SH600000\": [(2005-01-01, 2005-01-31), (2005-02-15, 2005-03-01)],\"SH600001\"\
|
|
: [(2005-01-01, 2005-03-01)],\"SH600002\": [(2005-01-01, 2005-02-14)],\"SH600003\"\
|
|
: [(2005-02-01, 2005-03-01)],}"
|
|
function: test_storage.test_instrument_storage
|
|
- docstring: " \nCalendar:pd.date_range(start=\"2005-01-01\", stop=\"2005-03-01\"\
|
|
, freq=\"1D\")Instrument:{\"SH600000\": [(2005-01-01, 2005-01-31), (2005-02-15,\
|
|
\ 2005-03-01)],\"SH600001\": [(2005-01-01, 2005-03-01)],\"SH600002\": [(2005-01-01,\
|
|
\ 2005-02-14)],\"SH600003\": [(2005-02-01, 2005-03-01)],}Feature:Stock data(close):2005-01-01\
|
|
\ ... 2005-02-01 ... 2005-02-14 2005-02-15 ... 2005-03-01SH600000 \
|
|
\ 1 ... 3 ... 4 5 6SH600001\
|
|
\ 1 ... 4 ... 5 6 7SH600002\
|
|
\ 1 ... 5 ... 6 nan nanSH600003\
|
|
\ nan ... 1 ... 2 3 4FeatureStorage(SH600000,\
|
|
\ close):[(calendar.index(\"2005-01-01\"), 1),...,(calendar.index(\"2005-03-01\"\
|
|
), 6)]====> [(0, 1), ..., (59, 6)]FeatureStorage(SH600002, close):[(calendar.index(\"\
|
|
2005-01-01\"), 1),...,(calendar.index(\"2005-02-14\"), 6)]===> [(0, 1), ..., (44,\
|
|
\ 6)]FeatureStorage(SH600003, close):[(calendar.index(\"2005-02-01\"), 1),...,(calendar.index(\"\
|
|
2005-03-01\"), 4)]===> [(31, 1), ..., (59, 4)]"
|
|
function: test_storage.test_feature_storage
|
|
- docstring: null
|
|
function: run_all_model.only_allow_defined_args
|
|
- docstring: ' Internal wrapper function.
|
|
|
|
argspec = inspect.getfullargspec(function_to_decorate)valid_names = set(argspec.args
|
|
+ argspec.kwonlyargs)if "self" in valid_names:valid_names.remove("self")for arg_name
|
|
in kwargs:if arg_name not in valid_names:raise ValueError("Unknown argument seen
|
|
''%s'', expected: [%s]" % (arg_name, ", ".join(valid_names)))return function_to_decorate(*args,
|
|
**kwargs)return _return_wrapped# function to handle ctrl z and ctrl c'
|
|
function: run_all_model._return_wrapped
|
|
- docstring: null
|
|
function: run_all_model.handler
|
|
- docstring: null
|
|
function: run_all_model.cal_mean_std
|
|
- docstring: null
|
|
function: run_all_model.create_env
|
|
- docstring: null
|
|
function: run_all_model.execute
|
|
- docstring: null
|
|
function: run_all_model.get_all_folders
|
|
- docstring: null
|
|
function: run_all_model.get_all_files
|
|
- docstring: null
|
|
function: run_all_model.get_all_results
|
|
- docstring: null
|
|
function: run_all_model.gen_and_save_md_table
|
|
- docstring: null
|
|
function: run_all_model.gen_yaml_file_without_seed_kwargs
|
|
- docstring: null
|
|
function: 'run_all_model.ModelRunner:'
|
|
- docstring: null
|
|
function: run_all_model._init_qlib
|
|
- docstring: " \nPlease be aware that this function can only work under Linux.\
|
|
\ MacOS and Windows will be supported in the future.Any PR to enhance this method\
|
|
\ is highly welcomed. Besides, this script doesn't support parallel running the\
|
|
\ same modelfor multiple times, and this will be fixed in the future development.Parameters:-----------times\
|
|
\ : intdetermines how many times the model should be running.models : str or listdetermines\
|
|
\ the specific model or list of models to run or exclude.exclude : booleandetermines\
|
|
\ whether the model being used is excluded or included.dataset : strdetermines\
|
|
\ the dataset to be used for each model.universe : strthe stock universe of the\
|
|
\ dataset.default \"\" indicates thatqlib_uri : strthe uri to install qlib with\
|
|
\ pipit could be URI on the remote or local path (NOTE: the local path must be\
|
|
\ an absolute path)exp_folder_name: strthe name of the experiment folderwait_before_rm_env\
|
|
\ : boolwait before remove environment.wait_when_err : boolwait when errors raised\
|
|
\ when executing commandsUsage:-------Here are some use cases of the function\
|
|
\ in the bash:The run_all_models will decide which config to run based no `models`\
|
|
\ `dataset` `universe`Example 1):models=\"lightgbm\", dataset=\"Alpha158\", universe=\"\
|
|
\" will result in running the following configexamples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yamlmodels=\"\
|
|
lightgbm\", dataset=\"Alpha158\", universe=\"csi500\" will result in running the\
|
|
\ following configexamples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158_csi500.yaml..\
|
|
\ code-block:: bash# Case 1 - run all models multiple timespython run_all_model.py\
|
|
\ run 3# Case 2 - run specific models multiple timespython run_all_model.py run\
|
|
\ 3 mlp# Case 3 - run specific models multiple times with specific datasetpython\
|
|
\ run_all_model.py run 3 mlp Alpha158# Case 4 - run other models except those\
|
|
\ are given as arguments for multiple timespython run_all_model.py run 3 [mlp,tft,lstm]\
|
|
\ --exclude=True# Case 5 - run specific models for one timepython run_all_model.py\
|
|
\ run --models=[mlp,lightgbm]# Case 6 - run other models except those are given\
|
|
\ as arguments for one timepython run_all_model.py run --models=[mlp,tft,sfm]\
|
|
\ --exclude=True# Case 7 - run lightgbm model on csi500.python run_all_model.py\
|
|
\ run 3 lightgbm Alpha158 csi500"
|
|
function: run_all_model.run
|
|
- docstring: null
|
|
function: create_dataset.get_library_name
|
|
- docstring: null
|
|
function: create_dataset.is_stock
|
|
- docstring: " \nexchange_place: \"SZ\" OR \"SH\"type: \"tick\", \"orderbook\"\
|
|
, ...filepath: the path of csvarc: arclink created by a process"
|
|
function: create_dataset.add_one_stock_daily_data
|
|
- docstring: null
|
|
function: create_dataset.format_time
|
|
- docstring: null
|
|
function: create_dataset.add_one_stock_daily_data_wrapper
|
|
- docstring: null
|
|
function: create_dataset.add_data
|
|
- docstring: ' Dataset creator
|
|
|
|
'
|
|
function: 'create_dataset.DSCreator:'
|
|
- docstring: null
|
|
function: create_dataset.clear
|
|
- docstring: null
|
|
function: create_dataset.initialize_library
|
|
- docstring: null
|
|
function: create_dataset._get_empty_folder
|
|
- docstring: " \nUseful commands- run all tests: pytest examples/orderbook_data/example.py-\
|
|
\ run a single test: pytest -s --pdb --disable-warnings examples/orderbook_data/example.py::TestClass::test_basic01"
|
|
function: example.TestClass
|
|
- docstring: " \nConfigure for arctic"
|
|
function: example.setUp
|
|
- docstring: null
|
|
function: example.test_basic
|
|
- docstring: null
|
|
function: example.test_basic_without_time
|
|
- docstring: null
|
|
function: example.test_basic01
|
|
- docstring: null
|
|
function: example.test_basic02
|
|
- docstring: null
|
|
function: example.test_basic03
|
|
- docstring: null
|
|
function: example.total_func
|
|
- docstring: null
|
|
function: example.test_exp_01
|
|
- docstring: null
|
|
function: example.test_exp_02
|
|
- docstring: null
|
|
function: example.test_exp_03
|
|
- docstring: null
|
|
function: example.test_exp_04
|
|
- docstring: null
|
|
function: example.test_exp_05
|
|
- docstring: null
|
|
function: example.test_exp_06
|
|
- docstring: null
|
|
function: example.expr7_init
|
|
- docstring: null
|
|
function: example.test_exp_07_1
|
|
- docstring: null
|
|
function: example.test_exp_07_2
|
|
- docstring: null
|
|
function: example.expr7_3_init
|
|
- docstring: null
|
|
function: example.test_exp_08_1
|
|
- docstring: null
|
|
function: example.test_exp_08_2
|
|
- docstring: null
|
|
function: example.test_exp_09_trans
|
|
- docstring: null
|
|
function: example.test_exp_09_order
|
|
- docstring: null
|
|
function: dataset._to_tensor
|
|
- docstring: " \ncreate time series slices from pandas indexArgs:index (pd.MultiIndex):\
|
|
\ pandas multiindex with <instrument, datetime> orderseq_len (int): sequence length"
|
|
function: dataset._create_ts_slices
|
|
- docstring: ' get date parse function
|
|
|
|
This method is used to parse date arguments as target type.Example:get_date_parse_fn(''20120101'')(''2017-01-01'')
|
|
=> ''20170101''get_date_parse_fn(20120101)(''2017-01-01'') => 20170101'
|
|
function: dataset._get_date_parse_fn
|
|
- docstring: ' Memory Augmented Time Series Dataset
|
|
|
|
Args:handler (DataHandler): data handlersegments (dict): data split segmentsseq_len
|
|
(int): time series sequence lengthhorizon (int): label horizon (to mask historical
|
|
loss for TRA)num_states (int): how many memory states to be added (for TRA)batch_size
|
|
(int): batch size (<0 means daily batch)shuffle (bool): whether shuffle datapin_memory
|
|
(bool): whether pin data to gpu memorydrop_last (bool): whether drop last batch
|
|
< batch_size'
|
|
function: dataset.MTSDatasetH
|
|
- docstring: null
|
|
function: dataset.setup_data
|
|
- docstring: null
|
|
function: dataset._prepare_seg
|
|
- docstring: null
|
|
function: dataset.restore_index
|
|
- docstring: null
|
|
function: dataset.assign_data
|
|
- docstring: null
|
|
function: dataset.clear_memory
|
|
- docstring: ' enable traning mode
|
|
|
|
self.batch_size, self.drop_last, self.shuffle = self.params'
|
|
function: dataset.train
|
|
- docstring: ' enable evaluation mode
|
|
|
|
self.batch_size = -1self.drop_last = Falseself.shuffle = False'
|
|
function: dataset.eval
|
|
- docstring: null
|
|
function: model.TRAModel
|
|
- docstring: null
|
|
function: model.train_epoch
|
|
- docstring: null
|
|
function: model.test_epoch
|
|
- docstring: null
|
|
function: model.fit
|
|
- docstring: null
|
|
function: model.predict
|
|
- docstring: ' LSTM Model
|
|
|
|
Args:input_size (int): input size (# features)hidden_size (int): hidden sizenum_layers
|
|
(int): number of hidden layersuse_attn (bool): whether use attention layer.we
|
|
use concat attention as https://github.com/fulifeng/Adv-ALSTM/dropout (float):
|
|
dropout rateinput_drop (float): input dropout for data augmentationnoise_level
|
|
(float): add gaussian noise to input for data augmentation'
|
|
function: model.LSTM
|
|
- docstring: null
|
|
function: model.forward
|
|
- docstring: null
|
|
function: model.PositionalEncoding
|
|
- docstring: null
|
|
function: model.forward
|
|
- docstring: ' Transformer Model
|
|
|
|
Args:input_size (int): input size (# features)hidden_size (int): hidden sizenum_layers
|
|
(int): number of transformer layersnum_heads (int): number of heads in transformerdropout
|
|
(float): dropout rateinput_drop (float): input dropout for data augmentationnoise_level
|
|
(float): add gaussian noise to input for data augmentation'
|
|
function: model.Transformer
|
|
- docstring: null
|
|
function: model.forward
|
|
- docstring: ' Temporal Routing Adaptor (TRA)
|
|
|
|
TRA takes historical prediction errors & latent representation as inputs,then
|
|
routes the input sample to a specific predictor for training & inference.Args:input_size
|
|
(int): input size (RNN/Transformer''s hidden size)num_states (int): number of
|
|
latent states (i.e., trading patterns)If `num_states=1`, then TRA falls back to
|
|
traditional methodshidden_size (int): hidden size of the routertau (float): gumbel
|
|
softmax temperature'
|
|
function: model.TRA
|
|
- docstring: null
|
|
function: model.forward
|
|
- docstring: null
|
|
function: model.evaluate
|
|
- docstring: null
|
|
function: model.average_params
|
|
- docstring: ' Replaces inf by maximum of tensor
|
|
|
|
mask_inf = torch.isinf(inp_tensor)ind_inf = torch.nonzero(mask_inf, as_tuple=False)if
|
|
len(ind_inf) > 0:for ind in ind_inf:if len(ind) == 2:inp_tensor[ind[0], ind[1]]
|
|
= 0elif len(ind) == 1:inp_tensor[ind[0]] = 0m = torch.max(inp_tensor)for ind in
|
|
ind_inf:if len(ind) == 2:inp_tensor[ind[0], ind[1]] = melif len(ind) == 1:inp_tensor[ind[0]]
|
|
= mreturn inp_tensor'
|
|
function: model.shoot_infs
|
|
- docstring: null
|
|
function: tft.get_shifted_label
|
|
- docstring: null
|
|
function: tft.fill_test_na
|
|
- docstring: ' Prepare data to fit the TFT model.
|
|
|
|
Args:df: Original DataFrame.fillna: Whether to fill the data with the mean values.Returns:Transformed
|
|
DataFrame.'
|
|
function: tft.process_qlib_data
|
|
- docstring: ' Transform the TFT predicted data into Qlib format.
|
|
|
|
Args:df: Original DataFrame.fillna: New column name.Returns:Transformed DataFrame.'
|
|
function: tft.process_predicted
|
|
- docstring: null
|
|
function: tft.format_score
|
|
- docstring: null
|
|
function: tft.transform_df
|
|
- docstring: ' TFT Model
|
|
|
|
self.model = Noneself.params = {"DATASET": "Alpha158", "label_shift": 5}self.params.update(kwargs)'
|
|
function: tft.TFTModel
|
|
- docstring: null
|
|
function: tft._prepare_data
|
|
- docstring: null
|
|
function: tft.fit
|
|
- docstring: ' Strips out forecast time and identifier columns.
|
|
|
|
return data[[col for col in data.columns if col not in {"forecast_time", "identifier"}]]#
|
|
p50_loss = utils.numpy_normalised_quantile_loss(# extract_numerical_data(targets),
|
|
extract_numerical_data(p50_forecast),# 0.5)# p90_loss = utils.numpy_normalised_quantile_loss(# extract_numerical_data(targets),
|
|
extract_numerical_data(p90_forecast),# 0.9)tf.keras.backend.set_session(default_keras_session)print("Training
|
|
completed at {}.".format(dte.datetime.now()))# ===========================Training
|
|
Process==========================='
|
|
function: tft.extract_numerical_data
|
|
- docstring: null
|
|
function: tft.predict
|
|
- docstring: " \nfinetune modelParameters----------dataset : DatasetHdataset\
|
|
\ for finetuning"
|
|
function: tft.finetune
|
|
- docstring: " \nTensorflow model can't be dumped directly.So the data should\
|
|
\ be save separately**TODO**: Please implement the function to load the filesParameters----------path\
|
|
\ : Union[Path, str]the target path to be dumped"
|
|
function: tft.to_pickle
|
|
- docstring: ' Returns simple Keras linear layer.
|
|
|
|
Args:size: Output sizeactivation: Activation function to apply if requireduse_time_distributed:
|
|
Whether to apply layer across timeuse_bias: Whether bias should be included in
|
|
layer'
|
|
function: tft_model.linear_layer
|
|
- docstring: ' Applies simple feed-forward network to an input.
|
|
|
|
Args:inputs: MLP inputshidden_size: Hidden state sizeoutput_size: Output size
|
|
of MLPoutput_activation: Activation function to apply on outputhidden_activation:
|
|
Activation function to apply on inputuse_time_distributed: Whether to apply across
|
|
timeReturns:Tensor for MLP outputs.'
|
|
function: tft_model.apply_mlp
|
|
- docstring: ' Applies a Gated Linear Unit (GLU) to an input.
|
|
|
|
Args:x: Input to gating layerhidden_layer_size: Dimension of GLUdropout_rate:
|
|
Dropout rate to apply if anyuse_time_distributed: Whether to apply across timeactivation:
|
|
Activation function to apply to the linear feature transform ifnecessaryReturns:Tuple
|
|
of tensors for: (GLU output, gate)'
|
|
function: tft_model.apply_gating_layer
|
|
- docstring: ' Applies skip connection followed by layer normalisation.
|
|
|
|
Args:x_list: List of inputs to sum for skip connectionReturns:Tensor output from
|
|
layer.'
|
|
function: tft_model.add_and_norm
|
|
- docstring: ' Applies the gated residual network (GRN) as defined in paper.
|
|
|
|
Args:x: Network inputshidden_layer_size: Internal state sizeoutput_size: Size
|
|
of output layerdropout_rate: Dropout rate if dropout is applieduse_time_distributed:
|
|
Whether to apply network across time dimensionadditional_context: Additional context
|
|
vector to use if relevantreturn_gate: Whether to return GLU gate for diagnostic
|
|
purposesReturns:Tuple of tensors for: (GRN output, GLU gate)'
|
|
function: tft_model.gated_residual_network
|
|
- docstring: ' Returns causal mask to apply for self-attention layer.
|
|
|
|
Args:self_attn_inputs: Inputs to self attention layer to determine mask shape'
|
|
function: tft_model.get_decoder_mask
|
|
- docstring: ' Defines scaled dot product attention layer.
|
|
|
|
Attributes:dropout: Dropout rate to useactivation: Normalisation function for
|
|
scaled dot product attention (e.g.softmax by default)'
|
|
function: 'tft_model.ScaledDotProductAttention:'
|
|
- docstring: ' Defines interpretable multi-head attention layer.
|
|
|
|
Attributes:n_head: Number of headsd_k: Key/query dimensionality per headd_v: Value
|
|
dimensionalitydropout: Dropout rate to applyqs_layers: List of queries across
|
|
headsks_layers: List of keys across headsvs_layers: List of values across headsattention:
|
|
Scaled dot product attention layerw_o: Output weight matrix to project internal
|
|
state to the original TFTstate size'
|
|
function: 'tft_model.InterpretableMultiHeadAttention:'
|
|
- docstring: ' Caches data for the TFT.
|
|
|
|
_data_cache = {}@classmethod'
|
|
function: 'tft_model.TFTDataCache:'
|
|
- docstring: ' Updates cached data.
|
|
|
|
Args:data: Source to updatekey: Key to dictionary location'
|
|
function: tft_model.update
|
|
- docstring: ' Returns data stored at key location.
|
|
|
|
return cls._data_cache[key].copy()@classmethod'
|
|
function: tft_model.get
|
|
- docstring: ' Returns boolean indicating whether key is present in cache.
|
|
|
|
return key in cls._data_cache# TFT model definitions.'
|
|
function: tft_model.contains
|
|
- docstring: ' Defines Temporal Fusion Transformer.
|
|
|
|
Attributes:name: Name of modeltime_steps: Total number of input time steps per
|
|
forecast date (i.e. Widthof Temporal fusion decoder N)input_size: Total number
|
|
of inputsoutput_size: Total number of outputscategory_counts: Number of categories
|
|
per categorical variablen_multiprocessing_workers: Number of workers to use for
|
|
parallelcomputationscolumn_definition: List of tuples of (string, DataType, InputType)
|
|
thatdefine each columnquantiles: Quantiles to forecast for TFTuse_cudnn: Whether
|
|
to use Keras CuDNNLSTM or standard LSTM layershidden_layer_size: Internal state
|
|
size of TFTdropout_rate: Dropout discard ratemax_gradient_norm: Maximum norm for
|
|
gradient clippinglearning_rate: Initial learning rate of ADAM optimizerminibatch_size:
|
|
Size of minibatches for trainingnum_epochs: Maximum number of epochs for trainingearly_stopping_patience:
|
|
Maximum number of iterations of non-improvementbefore early stopping kicks innum_encoder_steps:
|
|
Size of LSTM encoder -- i.e. number of past time stepsbefore forecast date to
|
|
usenum_stacks: Number of self-attention layers to apply (default is 1 for basicTFT)num_heads:
|
|
Number of heads for interpretable mulit-head attentionmodel: Keras model for TFT'
|
|
function: 'tft_model.TemporalFusionTransformer:'
|
|
- docstring: ' Transforms raw inputs to embeddings.
|
|
|
|
Applies linear transformation onto continuous variables and uses embeddingsfor
|
|
categorical variables.Args:all_inputs: Inputs to transformReturns:Tensors for
|
|
transformed inputs.'
|
|
function: tft_model.get_tft_embeddings
|
|
- docstring: ' Applies linear transformation for time-varying inputs.
|
|
|
|
return tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(self.hidden_layer_size))(x)#
|
|
Targetsobs_inputs = tf.keras.backend.stack([convert_real_to_embedding(regular_inputs[Ellipsis,
|
|
i : i + 1]) for i in self._input_obs_loc], axis=-1)# Observed (a prioir unknown)
|
|
inputswired_embeddings = []for i in range(num_categorical_variables):if i not
|
|
in self._known_categorical_input_idx and i + num_regular_variables not in self._input_obs_loc:e
|
|
= embeddings[i](categorical_inputs[:, :, i])wired_embeddings.append(e)unknown_inputs
|
|
= []for i in range(regular_inputs.shape[-1]):if i not in self._known_regular_input_idx
|
|
and i not in self._input_obs_loc:e = convert_real_to_embedding(regular_inputs[Ellipsis,
|
|
i : i + 1])unknown_inputs.append(e)if unknown_inputs + wired_embeddings:unknown_inputs
|
|
= tf.keras.backend.stack(unknown_inputs + wired_embeddings, axis=-1)else:unknown_inputs
|
|
= None# A priori known inputsknown_regular_inputs = [convert_real_to_embedding(regular_inputs[Ellipsis,
|
|
i : i + 1])for i in self._known_regular_input_idxif i not in self._static_input_loc]known_categorical_inputs
|
|
= [embedded_inputs[i]for i in self._known_categorical_input_idxif i + num_regular_variables
|
|
not in self._static_input_loc]known_combined_layer = tf.keras.backend.stack(known_regular_inputs
|
|
+ known_categorical_inputs, axis=-1)return unknown_inputs, known_combined_layer,
|
|
obs_inputs, static_inputs'
|
|
function: tft_model.convert_real_to_embedding
|
|
- docstring: ' Returns name of single column for input type.
|
|
|
|
return utils.get_single_col_by_input_type(input_type, self.column_definition)'
|
|
function: tft_model._get_single_col_by_type
|
|
- docstring: ' Returns boolean indicating if training data has been cached.
|
|
|
|
return TFTDataCache.contains("train") and TFTDataCache.contains("valid")'
|
|
function: tft_model.training_data_cached
|
|
- docstring: ' Batches and caches data once for using during training.
|
|
|
|
Args:data: Data to batch and cachecache_key: Key used for cachenum_samples: Maximum
|
|
number of samples to extract (-1 to use all data)'
|
|
function: tft_model.cache_batched_data
|
|
- docstring: ' Samples segments into a compatible format.
|
|
|
|
Args:data: Sources data to sample and batchmax_samples: Maximum number of samples
|
|
in batchReturns:Dictionary of batched data with the maximum samples specified.'
|
|
function: tft_model._batch_sampled_data
|
|
- docstring: ' Batches data for training.
|
|
|
|
Converts raw dataframe from a 2-D tabular format to a batched 3-D arrayto feed
|
|
into Keras model.Args:data: DataFrame to batchReturns:Batched Numpy array with
|
|
shape=(?, self.time_steps, self.input_size)'
|
|
function: tft_model._batch_data
|
|
- docstring: null
|
|
function: tft_model._batch_single_entity
|
|
- docstring: ' Formats sample weights for Keras training.
|
|
|
|
return (np.sum(x, axis=-1) > 0.0) * 1.0'
|
|
function: tft_model._get_active_locations
|
|
- docstring: ' Returns graph defining layers of the TFT.
|
|
|
|
# Size definitions.time_steps = self.time_stepscombined_input_size = self.input_sizeencoder_steps
|
|
= self.num_encoder_steps# Inputs.all_inputs = tf.keras.layers.Input(shape=(time_steps,combined_input_size,))unknown_inputs,
|
|
known_combined_layer, obs_inputs, static_inputs = self.get_tft_embeddings(all_inputs)#
|
|
Isolate known and observed historical inputs.if unknown_inputs is not None:historical_inputs
|
|
= concat([unknown_inputs[:, :encoder_steps, :],known_combined_layer[:, :encoder_steps,
|
|
:],obs_inputs[:, :encoder_steps, :],],axis=-1,)else:historical_inputs = concat([known_combined_layer[:,
|
|
:encoder_steps, :], obs_inputs[:, :encoder_steps, :]], axis=-1)# Isolate only
|
|
known future inputs.future_inputs = known_combined_layer[:, encoder_steps:, :]'
|
|
function: tft_model._build_base_graph
|
|
- docstring: ' Applies variable selection network to static inputs.
|
|
|
|
Args:embedding: Transformed static inputsReturns:Tensor output for variable selection
|
|
network'
|
|
function: tft_model.static_combine_and_mask
|
|
- docstring: ' Apply temporal variable selection networks.
|
|
|
|
Args:embedding: Transformed inputs.Returns:Processed tensor outputs.'
|
|
function: tft_model.lstm_combine_and_mask
|
|
- docstring: ' Returns LSTM cell initialized with default parameters.
|
|
|
|
if self.use_cudnn:lstm = tf.keras.layers.CuDNNLSTM(self.hidden_layer_size,return_sequences=True,return_state=return_state,stateful=False,)else:lstm
|
|
= tf.keras.layers.LSTM(self.hidden_layer_size,return_sequences=True,return_state=return_state,stateful=False,#
|
|
Additional params to ensure LSTM matches CuDNN, See TF 2.0 :# (https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM)activation="tanh",recurrent_activation="sigmoid",recurrent_dropout=0,unroll=False,use_bias=True,)return
|
|
lstmhistory_lstm, state_h, state_c = get_lstm(return_state=True)(historical_features,
|
|
initial_state=[static_context_state_h, static_context_state_c])future_lstm = get_lstm(return_state=False)(future_features,
|
|
initial_state=[state_h, state_c])lstm_layer = concat([history_lstm, future_lstm],
|
|
axis=1)# Apply gated skip connectioninput_embeddings = concat([historical_features,
|
|
future_features], axis=1)lstm_layer, _ = apply_gating_layer(lstm_layer, self.hidden_layer_size,
|
|
self.dropout_rate, activation=None)temporal_feature_layer = add_and_norm([lstm_layer,
|
|
input_embeddings])# Static enrichment layersexpanded_static_context = K.expand_dims(static_context_enrichment,
|
|
axis=1)enriched, _ = gated_residual_network(temporal_feature_layer,self.hidden_layer_size,dropout_rate=self.dropout_rate,use_time_distributed=True,additional_context=expanded_static_context,return_gate=True,)#
|
|
Decoder self attentionself_attn_layer = InterpretableMultiHeadAttention(self.num_heads,
|
|
self.hidden_layer_size, dropout=self.dropout_rate)mask = get_decoder_mask(enriched)x,
|
|
self_att = self_attn_layer(enriched, enriched, enriched, mask=mask)x, _ = apply_gating_layer(x,
|
|
self.hidden_layer_size, dropout_rate=self.dropout_rate, activation=None)x = add_and_norm([x,
|
|
enriched])# Nonlinear processing on outputsdecoder = gated_residual_network(x,
|
|
self.hidden_layer_size, dropout_rate=self.dropout_rate, use_time_distributed=True)#
|
|
Final skip connectiondecoder, _ = apply_gating_layer(decoder, self.hidden_layer_size,
|
|
activation=None)transformer_layer = add_and_norm([decoder, temporal_feature_layer])#
|
|
Attention components for explainabilityattention_components = {# Temporal attention
|
|
weights"decoder_self_attn": self_att,# Static variable selection weights"static_flags":
|
|
static_weights[Ellipsis, 0],# Variable selection weights of past inputs"historical_flags":
|
|
historical_flags[Ellipsis, 0, :],# Variable selection weights of future inputs"future_flags":
|
|
future_flags[Ellipsis, 0, :],}return transformer_layer, all_inputs, attention_components'
|
|
function: tft_model.get_lstm
|
|
- docstring: ' Build model and defines training losses.
|
|
|
|
Returns:Fully defined Keras model.'
|
|
function: tft_model.build_model
|
|
- docstring: ' Computes the combined quantile loss for prespecified
|
|
quantiles.
|
|
|
|
Attributes:quantiles: Quantiles to compute losses'
|
|
function: 'tft_model.QuantileLossCalculator:'
|
|
- docstring: ' Returns quantile loss for specified quantiles.
|
|
|
|
Args:a: Targetsb: Predictions'
|
|
function: tft_model.quantile_loss
|
|
- docstring: ' Fits deep neural network for given training and validation data.
|
|
|
|
Args:train_df: DataFrame for training datavalid_df: DataFrame for validation data'
|
|
function: tft_model.fit
|
|
- docstring: null
|
|
function: tft_model._unpack
|
|
- docstring: ' Applies evaluation metric to the training data.
|
|
|
|
Args:data: Dataframe for evaluationeval_metric: Evaluation metic to return, based
|
|
on model definition.Returns:Computed evaluation loss.'
|
|
function: tft_model.evaluate
|
|
- docstring: ' Computes predictions for a given input dataset.
|
|
|
|
Args:df: Input dataframereturn_targets: Whether to also return outputs aligned
|
|
with predictions tofacilitate evaluationReturns:Input dataframe or tuple of (input
|
|
dataframe, aligned output dataframe).'
|
|
function: tft_model.predict
|
|
- docstring: ' Returns formatted dataframes for prediction.
|
|
|
|
flat_prediction = pd.DataFrame(prediction[:, :, 0], columns=["t+{}".format(i)
|
|
for i in range(self.time_steps - self.num_encoder_steps)])cols = list(flat_prediction.columns)flat_prediction["forecast_time"]
|
|
= time[:, self.num_encoder_steps - 1, 0]flat_prediction["identifier"] = identifier[:,
|
|
0, 0]# Arrange in orderreturn flat_prediction[["forecast_time", "identifier"]
|
|
+ cols]# Extract predictions for each quantile into different entriesprocess_map
|
|
= {"p{}".format(int(q * 100)): combined[Ellipsis, i * self.output_size : (i +
|
|
1) * self.output_size]for i, q in enumerate(self.quantiles)}if return_targets:#
|
|
Add targets if relevantprocess_map["targets"] = outputsreturn {k: format_outputs(process_map[k])
|
|
for k in process_map}'
|
|
function: tft_model.format_outputs
|
|
- docstring: ' Computes TFT attention weights for a given dataset.
|
|
|
|
Args:df: Input dataframeReturns:Dictionary of numpy arrays for temporal attention
|
|
weights and variableselection weights, along with their identifiers and time indices'
|
|
function: tft_model.get_attention
|
|
- docstring: ' Returns weights for a given minibatch of data.
|
|
|
|
input_placeholder = self._input_placeholderattention_weights = {}for k in self._attention_components:attention_weight
|
|
= tf.keras.backend.get_session().run(self._attention_components[k], {input_placeholder:
|
|
input_batch.astype(np.float32)})attention_weights[k] = attention_weightreturn
|
|
attention_weights# Compute number of batchesbatch_size = self.minibatch_sizen
|
|
= inputs.shape[0]num_batches = n // batch_sizeif n - (num_batches * batch_size)
|
|
> 0:num_batches += 1# Split up inputs into batchesbatched_inputs = [inputs[i *
|
|
batch_size : (i + 1) * batch_size, Ellipsis] for i in range(num_batches)]# Get
|
|
attention weights, while avoiding large memory increasesattention_by_batch = [get_batch_attention_weights(batch)
|
|
for batch in batched_inputs]attention_weights = {}for k in self._attention_components:attention_weights[k]
|
|
= []for batch_weights in attention_by_batch:attention_weights[k].append(batch_weights[k])if
|
|
len(attention_weights[k][0].shape) == 4:tmp = np.concatenate(attention_weights[k],
|
|
axis=1)else:tmp = np.concatenate(attention_weights[k], axis=0)del attention_weights[k]gc.collect()attention_weights[k]
|
|
= tmpattention_weights["identifiers"] = identifiers[:, 0, 0]attention_weights["time"]
|
|
= time[:, :, 0]return attention_weights# Serialisation.'
|
|
function: tft_model.get_batch_attention_weights
|
|
- docstring: ' Deletes and recreates folder with temporary Keras training outputs.
|
|
|
|
print("Resetting temp folder...")utils.create_folder_if_not_exist(self._temp_folder)shutil.rmtree(self._temp_folder)os.makedirs(self._temp_folder)'
|
|
function: tft_model.reset_temp_folder
|
|
- docstring: ' Returns path to keras checkpoint.
|
|
|
|
return os.path.join(model_folder, "{}.check".format(self.name))'
|
|
function: tft_model.get_keras_saved_path
|
|
- docstring: ' Saves optimal TFT weights.
|
|
|
|
Args:model_folder: Location to serialze model.'
|
|
function: tft_model.save
|
|
- docstring: ' Loads TFT weights.
|
|
|
|
Args:model_folder: Folder containing serialized models.use_keras_loadings: Whether
|
|
to load from Keras checkpoint.Returns:'
|
|
function: tft_model.load
|
|
- docstring: ' Returns name of single column.
|
|
|
|
Args:input_type: Input type of column to extractcolumn_definition: Column definition
|
|
list for experiment'
|
|
function: utils.get_single_col_by_input_type
|
|
- docstring: ' Extracts the names of columns that correspond to a define data_type.
|
|
|
|
Args:data_type: DataType of columns to extract.column_definition: Column definition
|
|
to use.excluded_input_types: Set of input types to excludeReturns:List of names
|
|
for columns with data type specified.'
|
|
function: utils.extract_cols_from_data_type
|
|
- docstring: ' Computes quantile loss for tensorflow.
|
|
|
|
Standard quantile loss as defined in the "Training Procedure" section ofthe main
|
|
TFT paperArgs:y: Targetsy_pred: Predictionsquantile: Quantile to use for loss
|
|
calculations (between 0 & 1)Returns:Tensor for quantile loss.'
|
|
function: utils.tensorflow_quantile_loss
|
|
- docstring: ' Computes normalised quantile loss for numpy arrays.
|
|
|
|
Uses the q-Risk metric as defined in the "Training Procedure" section of themain
|
|
TFT paper.Args:y: Targetsy_pred: Predictionsquantile: Quantile to use for loss
|
|
calculations (between 0 & 1)Returns:Float for normalised quantile loss.'
|
|
function: utils.numpy_normalised_quantile_loss
|
|
- docstring: ' Creates folder if it doesn''t exist.
|
|
|
|
Args:directory: Folder path to create.'
|
|
function: utils.create_folder_if_not_exist
|
|
- docstring: ' Creates tensorflow config for graphs to run on CPU or GPU.
|
|
|
|
Specifies whether to run graph on gpu or cpu and which GPU ID to use for multiGPU
|
|
machines.Args:tf_device: ''cpu'' or ''gpu''gpu_id: GPU ID to use if relevantReturns:Tensorflow
|
|
config.'
|
|
function: utils.get_default_tensorflow_config
|
|
- docstring: ' Saves Tensorflow graph to checkpoint.
|
|
|
|
Saves all trainiable variables under a given variable scope to checkpoint.Args:tf_session:
|
|
Session containing graphmodel_folder: Folder to save modelscp_name: Name of Tensorflow
|
|
checkpointscope: Variable scope containing variables to save'
|
|
function: utils.save
|
|
- docstring: ' Loads Tensorflow graph from checkpoint.
|
|
|
|
Args:tf_session: Session to load graph intomodel_folder: Folder containing serialised
|
|
modelcp_name: Name of Tensorflow checkpointscope: Variable scope to use.verbose:
|
|
Whether to print additional debugging information.'
|
|
function: utils.load
|
|
- docstring: ' Prints all weights in Tensorflow checkpoint.
|
|
|
|
Args:model_folder: Folder containing checkpointcp_name: Name of checkpointReturns:'
|
|
function: utils.print_weights_in_checkpoint
|
|
- docstring: ' Manages hyperparameter optimisation using random search for a single
|
|
GPU.
|
|
|
|
Attributes:param_ranges: Discrete hyperparameter range for random search.results:
|
|
Dataframe of validation results.fixed_params: Fixed model parameters per experiment.saved_params:
|
|
Dataframe of parameters trained.best_score: Minimum validation loss observed thus
|
|
far.optimal_name: Key to best configuration.hyperparam_folder: Where to save optimisation
|
|
outputs.'
|
|
function: 'hyperparam_opt.HyperparamOptManager:'
|
|
- docstring: ' Loads results from previous hyperparameter optimisation.
|
|
|
|
Returns:A boolean indicating if previous results can be loaded.'
|
|
function: hyperparam_opt.load_results
|
|
- docstring: ' Returns previously saved parameters given a key.
|
|
|
|
params = self.saved_paramsselected_params = dict(params[name])if self._override_w_fixed_params:for
|
|
k in self.fixed_params:selected_params[k] = self.fixed_params[k]return selected_params'
|
|
function: hyperparam_opt._get_params_from_name
|
|
- docstring: ' Returns the optimal hyperparameters thus far.
|
|
|
|
optimal_name = self.optimal_namereturn self._get_params_from_name(optimal_name)'
|
|
function: hyperparam_opt.get_best_params
|
|
- docstring: ' Clears all previous results and saved parameters.
|
|
|
|
shutil.rmtree(self.hyperparam_folder)os.makedirs(self.hyperparam_folder)self.results
|
|
= pd.DataFrame()self.saved_params = pd.DataFrame()'
|
|
function: hyperparam_opt.clear
|
|
- docstring: ' Checks that parameter map is properly defined.
|
|
|
|
valid_fields = list(self.param_ranges.keys()) + list(self.fixed_params.keys())invalid_fields
|
|
= [k for k in params if k not in valid_fields]missing_fields = [k for k in valid_fields
|
|
if k not in params]if invalid_fields:raise ValueError("Invalid Fields Found {}
|
|
- Valid ones are {}".format(invalid_fields, valid_fields))if missing_fields:raise
|
|
ValueError("Missing Fields Found {} - Valid ones are {}".format(missing_fields,
|
|
valid_fields))'
|
|
function: hyperparam_opt._check_params
|
|
- docstring: ' Returns a unique key for the supplied set of params.
|
|
|
|
self._check_params(params)fields = list(params.keys())fields.sort()return "_".join([str(params[k])
|
|
for k in fields])'
|
|
function: hyperparam_opt._get_name
|
|
- docstring: ' Returns the next set of parameters to optimise.
|
|
|
|
Args:ranges_to_skip: Explicitly defines a set of keys to skip.'
|
|
function: hyperparam_opt.get_next_parameters
|
|
- docstring: ' Returns next hyperparameter set per try.
|
|
|
|
parameters = {k: np.random.choice(self.param_ranges[k]) for k in param_range_keys}#
|
|
Adds fixed paramsfor k in self.fixed_params:parameters[k] = self.fixed_params[k]return
|
|
parametersfor _ in range(self._max_tries):parameters = _get_next()name = self._get_name(parameters)if
|
|
name not in ranges_to_skip:return parametersraise ValueError("Exceeded max number
|
|
of hyperparameter searches!!")'
|
|
function: hyperparam_opt._get_next
|
|
- docstring: ' Updates the results from last optimisation run.
|
|
|
|
Args:parameters: Hyperparameters used in optimisation.loss: Validation loss obtained.model:
|
|
Model to serialised if required.info: Any ancillary information to tag on to results.Returns:Boolean
|
|
flag indicating if the model is the best seen so far.'
|
|
function: hyperparam_opt.update_score
|
|
- docstring: ' Manages distributed hyperparameter optimisation across many gpus.
|
|
|
|
self,param_ranges,fixed_params,root_model_folder,worker_number,search_iterations=1000,num_iterations_per_worker=5,clear_serialised_params=False,):Instantiates
|
|
optimisation manager.This hyperparameter optimisation pre-generates #search_iterationshyperparameter
|
|
combinations and serialises themat the start. At runtime, each worker goes through
|
|
their own set ofparameter ranges. The pregenerationallows for multiple workers
|
|
to run in parallel on different machines withoutresulting in parameter overlaps.Args:param_ranges:
|
|
Discrete hyperparameter range for random search.fixed_params: Fixed model parameters
|
|
per experiment.root_model_folder: Folder to store optimisation artifacts.worker_number:
|
|
Worker index defining which set of hyperparameters totest.search_iterations: Maximum
|
|
number of random search iterations.num_iterations_per_worker: How many iterations
|
|
are handled per worker.clear_serialised_params: Whether to regenerate hyperparametercombinations.'
|
|
function: hyperparam_opt.DistributedHyperparamOptManager
|
|
- docstring: null
|
|
function: hyperparam_opt.optimisation_completed
|
|
- docstring: ' Returns next dictionary of hyperparameters to optimise.
|
|
|
|
param_name = self.worker_search_queue.pop()params = self.global_hyperparam_df.loc[param_name,
|
|
:].to_dict()# Always override!for k in self.fixed_params:print("Overriding saved
|
|
{}: {}".format(k, self.fixed_params[k]))params[k] = self.fixed_params[k]return
|
|
params'
|
|
function: hyperparam_opt.get_next_parameters
|
|
- docstring: ' Loads serialsed hyperparameter ranges from file.
|
|
|
|
Returns:DataFrame containing hyperparameter combinations.'
|
|
function: hyperparam_opt.load_serialised_hyperparam_df
|
|
- docstring: ' Regenerates hyperparameter combinations and saves to file.
|
|
|
|
Returns:DataFrame containing hyperparameter combinations.'
|
|
function: hyperparam_opt.update_serialised_hyperparam_df
|
|
- docstring: ' Generates actual hyperparameter combinations.
|
|
|
|
Returns:DataFrame containing hyperparameter combinations.'
|
|
function: hyperparam_opt._generate_full_hyperparam_df
|
|
- docstring: ' Clears results for hyperparameter manager and resets.
|
|
|
|
super().clear()self.worker_search_queue = self._get_worker_search_queue()'
|
|
function: hyperparam_opt.clear
|
|
- docstring: ' Load results from file and queue parameter combinations to try.
|
|
|
|
Returns:Boolean indicating if results were successfully loaded.'
|
|
function: hyperparam_opt.load_results
|
|
- docstring: ' Generates the queue of param combinations for current worker.
|
|
|
|
Returns:Queue of hyperparameter combinations outstanding.'
|
|
function: hyperparam_opt._get_worker_search_queue
|
|
- docstring: ' Updates parameter combinations with the index of the worker
|
|
used.
|
|
|
|
Args:df: DataFrame of parameter combinations.Returns:Updated DataFrame with worker
|
|
number.'
|
|
function: hyperparam_opt.assign_worker_numbers
|
|
- docstring: ' Defines experiment configs and paths to outputs.
|
|
|
|
Attributes:root_folder: Root folder to contain all experimental outputs.experiment:
|
|
Name of experiment to run.data_folder: Folder to store data for experiment.model_folder:
|
|
Folder to store serialised models.results_folder: Folder to store results.data_csv_path:
|
|
Path to primary data csv file used in experiment.hyperparam_iterations: Default
|
|
number of random search iterations forexperiment.'
|
|
function: 'configs.ExperimentConfig:'
|
|
- docstring: null
|
|
function: configs.data_csv_path
|
|
- docstring: null
|
|
function: configs.hyperparam_iterations
|
|
- docstring: ' Gets a data formatter object for experiment.
|
|
|
|
Returns:Default DataFormatter per experiment.'
|
|
function: configs.make_data_formatter
|
|
- docstring: ' Defines and formats data for the Alpha158 dataset.
|
|
|
|
Attributes:column_definition: Defines input and data type of column used in theexperiment.identifiers:
|
|
Entity identifiers used in experiments.'
|
|
function: qlib_Alpha158.Alpha158Formatter
|
|
- docstring: ' Splits data frame into training-validation-test data frames.
|
|
|
|
This also calibrates scaling object, and transforms data for each split.Args:df:
|
|
Source data frame to split.valid_boundary: Starting year for validation datatest_boundary:
|
|
Starting year for test dataReturns:Tuple of transformed (train, valid, test) data.'
|
|
function: qlib_Alpha158.split_data
|
|
- docstring: ' Calibrates scalers using the data supplied.
|
|
|
|
Args:df: Data to use to calibrate scalers.'
|
|
function: qlib_Alpha158.set_scalers
|
|
- docstring: ' Performs feature transformations.
|
|
|
|
This includes both feature engineering, preprocessing and normalisation.Args:df:
|
|
Data frame to transform.Returns:Transformed data frame.'
|
|
function: qlib_Alpha158.transform_inputs
|
|
- docstring: ' Reverts any normalisation to give predictions in original scale.
|
|
|
|
Args:predictions: Dataframe of model predictions.Returns:Data frame of unnormalised
|
|
predictions.'
|
|
function: qlib_Alpha158.format_predictions
|
|
- docstring: ' Returns fixed model parameters for experiments.
|
|
|
|
fixed_params = {"total_time_steps": 6 + 6,"num_encoder_steps": 6,"num_epochs":
|
|
100,"early_stopping_patience": 10,"multiprocessing_workers": 5,}return fixed_params'
|
|
function: qlib_Alpha158.get_fixed_params
|
|
- docstring: ' Defines numerical types of each column.
|
|
|
|
REAL_VALUED = 0CATEGORICAL = 1DATE = 2'
|
|
function: base.DataTypes
|
|
- docstring: ' Defines input types of each column.
|
|
|
|
TARGET = 0OBSERVED_INPUT = 1KNOWN_INPUT = 2STATIC_INPUT = 3ID = 4 # Single column
|
|
used as an entity identifierTIME = 5 # Single column exclusively used as a time
|
|
index'
|
|
function: base.InputTypes
|
|
- docstring: ' Abstract base class for all data formatters.
|
|
|
|
User can implement the abstract methods below to perform dataset-specificmanipulations.'
|
|
function: base.GenericDataFormatter
|
|
- docstring: ' Calibrates scalers using the data supplied.
|
|
|
|
raise NotImplementedError()@abc.abstractmethod'
|
|
function: base.set_scalers
|
|
- docstring: ' Performs feature transformation.
|
|
|
|
raise NotImplementedError()@abc.abstractmethod'
|
|
function: base.transform_inputs
|
|
- docstring: ' Reverts any normalisation to give predictions in original scale.
|
|
|
|
raise NotImplementedError()@abc.abstractmethod'
|
|
function: base.format_predictions
|
|
- docstring: ' Performs the default train, validation and test splits.
|
|
|
|
raise NotImplementedError()@property@abc.abstractmethod'
|
|
function: base.split_data
|
|
- docstring: ' Defines order, input type and data type of each column.
|
|
|
|
raise NotImplementedError()@abc.abstractmethod'
|
|
function: base._column_definition
|
|
- docstring: ' Defines the fixed parameters used by the model for training.
|
|
|
|
Requires the following keys:''total_time_steps'': Defines the total number of
|
|
time steps used by TFT''num_encoder_steps'': Determines length of LSTM encoder
|
|
(i.e. history)''num_epochs'': Maximum number of epochs for training''early_stopping_patience'':
|
|
Early stopping param for keras''multiprocessing_workers'': # of cpus for data
|
|
processingReturns:A dictionary of fixed parameters, e.g.:fixed_params = {''total_time_steps'':
|
|
252 + 5,''num_encoder_steps'': 252,''num_epochs'': 100,''early_stopping_patience'':
|
|
5,''multiprocessing_workers'': 5,}'
|
|
function: base.get_fixed_params
|
|
- docstring: ' Returns number of categories per relevant input.
|
|
|
|
This is seqeuently required for keras embedding layers.'
|
|
function: base.num_classes_per_cat_input
|
|
- docstring: ' Gets the default number of training and validation samples.
|
|
|
|
Use to sub-sample the data for network calibration and a value of -1 usesall available
|
|
samples.Returns:Tuple of (training samples, validation samples)'
|
|
function: base.get_num_samples_for_calibration
|
|
- docstring: ' Returns formatted column definition in order expected by the
|
|
TFT.
|
|
|
|
column_definition = self._column_definition# Sanity checks first.# Ensure only
|
|
one ID and time column exist'
|
|
function: base.get_column_definition
|
|
- docstring: null
|
|
function: base._check_single_column
|
|
- docstring: ' Returns names of all input columns.
|
|
|
|
return [tup[0] for tup in self.get_column_definition() if tup[2] not in {InputTypes.ID,
|
|
InputTypes.TIME}]'
|
|
function: base._get_input_columns
|
|
- docstring: ' Returns the relevant indexes and input sizes required by TFT.
|
|
|
|
# Functions'
|
|
function: base._get_tft_input_indices
|
|
- docstring: null
|
|
function: base._extract_tuples_from_data_type
|
|
- docstring: null
|
|
function: base._get_locations
|
|
- docstring: null
|
|
function: multi_freq_handler.Avg15minLoader
|
|
- docstring: null
|
|
function: multi_freq_handler.load
|
|
- docstring: null
|
|
function: multi_freq_handler.Avg15minHandler
|
|
- docstring: null
|
|
function: 'workflow.NestedDecisionExecutionWorkflow:'
|
|
- docstring: ' initialize qlib
|
|
|
|
provider_uri_day = "~/.qlib/qlib_data/cn_data" # target_dirGetData().qlib_data(target_dir=provider_uri_day,
|
|
region=REG_CN, version="v2", exists_skip=True)provider_uri_1min = HIGH_FREQ_CONFIG.get("provider_uri")GetData().qlib_data(target_dir=provider_uri_1min,
|
|
interval="1min", region=REG_CN, version="v2", exists_skip=True)provider_uri_map
|
|
= {"1min": provider_uri_1min, "day": provider_uri_day}qlib.init(provider_uri=provider_uri_map,
|
|
dataset_cache=None, expression_cache=None)'
|
|
function: workflow._init_qlib
|
|
- docstring: null
|
|
function: workflow._train_model
|
|
- docstring: null
|
|
function: workflow.backtest
|
|
- docstring: null
|
|
function: workflow.collect_data
|
|
- docstring: null
|
|
function: workflow.check_diff_freq
|
|
- docstring: " \nThis backtest is used for comparing the nested execution and\
|
|
\ single layer executionDue to the low quality daily-level and miniute-level data,\
|
|
\ they are hardly comparable.So it is used for detecting serious bugs which make\
|
|
\ the results different greatly... code-block:: shell[1724971:MainThread](2021-12-07\
|
|
\ 16:24:31,156) INFO - qlib.workflow - [record_temp.py:441] - Portfolio analysis\
|
|
\ record 'port_analysis_1day.pkl'has been saved as the artifact of the Experiment\
|
|
\ 2'The following are analysis results of benchmark return(1day).'riskmean \
|
|
\ 0.000651std 0.012472annualized_return 0.154967information_ratio\
|
|
\ 0.805422max_drawdown -0.160445'The following are analysis results of the\
|
|
\ excess return without cost(1day).'riskmean 0.001375std \
|
|
\ 0.006103annualized_return 0.327204information_ratio 3.475016max_drawdown\
|
|
\ -0.024927'The following are analysis results of the excess return with\
|
|
\ cost(1day).'riskmean 0.001184std 0.006091annualized_return\
|
|
\ 0.281801information_ratio 2.998749max_drawdown -0.029568[1724971:MainThread](2021-12-07\
|
|
\ 16:24:31,170) INFO - qlib.workflow - [record_temp.py:466] - Indicator analysis\
|
|
\ record 'indicator_analysis_1day.pkl' has been saved as the artifact of the Experiment\
|
|
\ 2'The following are analysis results of indicators(1day).'valueffr 1.0pa\
|
|
\ 0.0pos 0.0[1724971:MainThread](2021-12-07 16:24:31,188) INFO - qlib.timer\
|
|
\ - [log.py:113] - Time cost: 0.007s | waiting `async_log` Done"
|
|
function: workflow.backtest_only_daily
|
|
- docstring: null
|
|
function: 'task_manager_rolling.RollingTaskExample:'
|
|
- docstring: null
|
|
function: task_manager_rolling.reset
|
|
- docstring: null
|
|
function: task_manager_rolling.task_generating
|
|
- docstring: null
|
|
function: task_manager_rolling.task_training
|
|
- docstring: null
|
|
function: task_manager_rolling.worker
|
|
- docstring: null
|
|
function: task_manager_rolling.task_collecting
|
|
- docstring: null
|
|
function: task_manager_rolling.rec_key
|
|
- docstring: null
|
|
function: task_manager_rolling.my_filter
|
|
- docstring: ' DayLast Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:a
|
|
series of that each value equals the last value of its day'
|
|
function: highfreq_ops.DayLast
|
|
- docstring: null
|
|
function: highfreq_ops._load_internal
|
|
- docstring: ' FFillNan Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:a
|
|
forward fill nan feature'
|
|
function: highfreq_ops.FFillNan
|
|
- docstring: null
|
|
function: highfreq_ops._load_internal
|
|
- docstring: ' BFillNan Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:a
|
|
backfoward fill nan feature'
|
|
function: highfreq_ops.BFillNan
|
|
- docstring: null
|
|
function: highfreq_ops._load_internal
|
|
- docstring: ' Date Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:a
|
|
series of that each value is the date corresponding to feature.index'
|
|
function: highfreq_ops.Date
|
|
- docstring: null
|
|
function: highfreq_ops._load_internal
|
|
- docstring: ' Select Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instance, select conditionfeature_right
|
|
: Expressionfeature instance, select valueReturns----------feature:value(feature_right)
|
|
that meets the condition(feature_left)'
|
|
function: highfreq_ops.Select
|
|
- docstring: null
|
|
function: highfreq_ops._load_internal
|
|
- docstring: ' IsNull Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:A
|
|
series indicating whether the feature is nan'
|
|
function: highfreq_ops.IsNull
|
|
- docstring: null
|
|
function: highfreq_ops._load_internal
|
|
- docstring: ' Cut Operator
|
|
|
|
Parameters----------feature : Expressionfeature instancel : intl > 0, delete the
|
|
first l elements of feature (default is None, which means 0)r : intr < 0, delete
|
|
the last -r elements of feature (default is None, which means 0)Returns----------feature:A
|
|
series with the first l and last -r elements deleted from the feature.Note: It
|
|
is deleted from the raw data, not the sliced data'
|
|
function: highfreq_ops.Cut
|
|
- docstring: null
|
|
function: highfreq_ops._load_internal
|
|
- docstring: null
|
|
function: 'workflow.HighfreqWorkflow:'
|
|
- docstring: ' initialize qlib
|
|
|
|
# use cn_data_1min dataQLIB_INIT_CONFIG = {**HIGH_FREQ_CONFIG, **self.SPEC_CONF}provider_uri
|
|
= QLIB_INIT_CONFIG.get("provider_uri")GetData().qlib_data(target_dir=provider_uri,
|
|
interval="1min", region=REG_CN, exists_skip=True)qlib.init(**QLIB_INIT_CONFIG)'
|
|
function: workflow._init_qlib
|
|
- docstring: ' preload the calendar for cache
|
|
|
|
# This code used the copy-on-write feature of Linux to avoid calculating the calendar
|
|
multiple times in the subprocess# This code may accelerate, but may be not useful
|
|
on Windows and Mac OsCal.calendar(freq="1min")get_calendar_day(freq="1min")'
|
|
function: workflow._prepare_calender_cache
|
|
- docstring: ' use dataset to get highreq data
|
|
|
|
self._init_qlib()self._prepare_calender_cache()dataset = init_instance_by_config(self.task["dataset"])xtrain,
|
|
xtest = dataset.prepare(["train", "test"])print(xtrain, xtest)dataset_backtest
|
|
= init_instance_by_config(self.task["dataset_backtest"])backtest_train, backtest_test
|
|
= dataset_backtest.prepare(["train", "test"])print(backtest_train, backtest_test)return'
|
|
function: workflow.get_data
|
|
- docstring: null
|
|
function: highfreq_handler.HighFreqHandler
|
|
- docstring: null
|
|
function: highfreq_handler.get_feature_config
|
|
- docstring: ' Get normalized price feature ops
|
|
|
|
if shift == 0:template_norm = "Cut({0}/Ref(DayLast({1}), 240), 240, None)"else:template_norm
|
|
= "Cut(Ref({0}, " + str(shift) + ")/Ref(DayLast({1}), 240), 240, None)"feature_ops
|
|
= template_norm.format(template_if.format(template_fillnan.format(template_paused.format("$close")),template_paused.format(price_field),),template_fillnan.format(template_paused.format("$close")),)return
|
|
feature_opsfields += [get_normalized_price_feature("$open", 0)]fields += [get_normalized_price_feature("$high",
|
|
0)]fields += [get_normalized_price_feature("$low", 0)]fields += [get_normalized_price_feature("$close",
|
|
0)]fields += [get_normalized_price_feature(simpson_vwap, 0)]names += ["$open",
|
|
"$high", "$low", "$close", "$vwap"]fields += [get_normalized_price_feature("$open",
|
|
240)]fields += [get_normalized_price_feature("$high", 240)]fields += [get_normalized_price_feature("$low",
|
|
240)]fields += [get_normalized_price_feature("$close", 240)]fields += [get_normalized_price_feature(simpson_vwap,
|
|
240)]names += ["$open_1", "$high_1", "$low_1", "$close_1", "$vwap_1"]fields +=
|
|
["Cut({0}/Ref(DayLast(Mean({0}, 7200)), 240), 240, None)".format("If(IsNull({0}),
|
|
0, If(Or(Gt({1}, Mul(1.001, {3})), Lt({1}, Mul(0.999, {2}))), 0, {0}))".format(template_paused.format("$volume"),template_paused.format(simpson_vwap),template_paused.format("$low"),template_paused.format("$high"),))]names
|
|
+= ["$volume"]fields += ["Cut(Ref({0}, 240)/Ref(DayLast(Mean({0}, 7200)), 240),
|
|
240, None)".format("If(IsNull({0}), 0, If(Or(Gt({1}, Mul(1.001, {3})), Lt({1},
|
|
Mul(0.999, {2}))), 0, {0}))".format(template_paused.format("$volume"),template_paused.format(simpson_vwap),template_paused.format("$low"),template_paused.format("$high"),))]names
|
|
+= ["$volume_1"]return fields, names'
|
|
function: highfreq_handler.get_normalized_price_feature
|
|
- docstring: null
|
|
function: highfreq_handler.HighFreqBacktestHandler
|
|
- docstring: null
|
|
function: highfreq_processor.HighFreqNorm
|
|
- docstring: null
|
|
function: 'update_online_pred.UpdatePredExample:'
|
|
- docstring: null
|
|
function: update_online_pred.first_train
|
|
- docstring: null
|
|
function: update_online_pred.update_online_pred
|
|
- docstring: null
|
|
function: 'rolling_online_management.RollingOnlineExample:'
|
|
- docstring: null
|
|
function: rolling_online_management.worker
|
|
- docstring: null
|
|
function: rolling_online_management.reset
|
|
- docstring: null
|
|
function: rolling_online_management.first_run
|
|
- docstring: null
|
|
function: rolling_online_management.routine
|
|
- docstring: null
|
|
function: rolling_online_management.add_strategy
|
|
- docstring: " \nInit OnlineManagerExample.Args:provider_uri (str, optional):\
|
|
\ the provider uri. Defaults to \"~/.qlib/qlib_data/cn_data\".region (str, optional):\
|
|
\ the stock region. Defaults to \"cn\".exp_name (str, optional): the experiment\
|
|
\ name. Defaults to \"rolling_exp\".task_url (str, optional): your MongoDB url.\
|
|
\ Defaults to \"mongodb://10.0.0.4:27017/\".task_db_name (str, optional): database\
|
|
\ name. Defaults to \"rolling_db\".task_pool (str, optional): the task pool name\
|
|
\ (a task pool is a collection in MongoDB). Defaults to \"rolling_task\".rolling_step\
|
|
\ (int, optional): the step for rolling. Defaults to 80.start_time (str, optional):\
|
|
\ the start time of simulating. Defaults to \"2018-09-10\".end_time (str, optional):\
|
|
\ the end time of simulating. Defaults to \"2018-10-31\".tasks (dict or list[dict]):\
|
|
\ a set of the task config waiting for rolling and training"
|
|
function: 'online_management_simulate.OnlineSimulationExample:'
|
|
- docstring: null
|
|
function: online_management_simulate.reset
|
|
- docstring: null
|
|
function: online_management_simulate.main
|
|
- docstring: null
|
|
function: 'workflow.RollingDataWorkflow:'
|
|
- docstring: ' initialize qlib
|
|
|
|
provider_uri = "~/.qlib/qlib_data/cn_data" # target_dirGetData().qlib_data(target_dir=provider_uri,
|
|
region=REG_CN, exists_skip=True)qlib.init(provider_uri=provider_uri, region=REG_CN)'
|
|
function: workflow._init_qlib
|
|
- docstring: null
|
|
function: workflow._dump_pre_handler
|
|
- docstring: null
|
|
function: workflow._load_pre_handler
|
|
- docstring: " \nUser could collect system info by following commands`cd scripts\
|
|
\ && python collect_info.py all`- NOTE: please avoid running this script in the\
|
|
\ project folder which contains `qlib`"
|
|
function: 'collect_info.InfoCollector:'
|
|
- docstring: ' collect system related info
|
|
|
|
for method in ["system", "machine", "platform", "version"]:print(getattr(platform,
|
|
method)())'
|
|
function: collect_info.sys
|
|
- docstring: ' collect Python related info
|
|
|
|
print("Python version: {}".format(sys.version.replace("\n", " ")))'
|
|
function: collect_info.py
|
|
- docstring: ' collect qlib related info
|
|
|
|
print("Qlib version: {}".format(qlib.__version__))REQUIRED = ["numpy","pandas","scipy","requests","sacred","python-socketio","redis","python-redis-lock","schedule","cvxpy","hyperopt","fire","statsmodels","xlrd","plotly","matplotlib","tables","pyyaml","mlflow","tqdm","loguru","lightgbm","tornado","joblib","fire","ruamel.yaml",]for
|
|
package in REQUIRED:version = pkg_resources.get_distribution(package).versionprint(f"{package}=={version}")'
|
|
function: collect_info.qlib
|
|
- docstring: " \nParameters----------csv_path: strstock data path or directoryqlib_dir:\
|
|
\ strqlib(dump) data directorbackup_dir: str, default Noneif backup_dir is not\
|
|
\ None, backup qlib_dir to backup_dirfreq: str, default \"day\"transaction frequencymax_workers:\
|
|
\ int, default Nonenumber of threadsdate_field_name: str, default \"date\"the\
|
|
\ name of the date field in the csvfile_suffix: str, default \".csv\"file suffixsymbol_field_name:\
|
|
\ str, default \"symbol\"symbol field nameinclude_fields: tupledump fieldsexclude_fields:\
|
|
\ tuplefields not dumpedlimit_nums: intUse when debugging, default None"
|
|
function: 'dump_bin.DumpDataBase:'
|
|
- docstring: null
|
|
function: dump_bin._backup_qlib_dir
|
|
- docstring: null
|
|
function: dump_bin._format_datetime
|
|
- docstring: null
|
|
function: dump_bin._get_date
|
|
- docstring: null
|
|
function: dump_bin._get_source_data
|
|
- docstring: null
|
|
function: dump_bin.get_symbol_from_file
|
|
- docstring: null
|
|
function: dump_bin.get_dump_fields
|
|
- docstring: null
|
|
function: dump_bin._read_calendars
|
|
- docstring: null
|
|
function: dump_bin._read_instruments
|
|
- docstring: null
|
|
function: dump_bin.save_calendars
|
|
- docstring: null
|
|
function: dump_bin.save_instruments
|
|
- docstring: null
|
|
function: dump_bin.data_merge_calendar
|
|
- docstring: null
|
|
function: dump_bin.get_datetime_index
|
|
- docstring: null
|
|
function: dump_bin._data_to_bin
|
|
- docstring: null
|
|
function: dump_bin._dump_bin
|
|
- docstring: null
|
|
function: dump_bin.dump
|
|
- docstring: null
|
|
function: dump_bin.DumpDataAll
|
|
- docstring: null
|
|
function: dump_bin._get_all_date
|
|
- docstring: null
|
|
function: dump_bin._dump_calendars
|
|
- docstring: null
|
|
function: dump_bin._dump_instruments
|
|
- docstring: null
|
|
function: dump_bin._dump_features
|
|
- docstring: null
|
|
function: dump_bin.dump
|
|
- docstring: null
|
|
function: dump_bin.DumpDataFix
|
|
- docstring: null
|
|
function: dump_bin._dump_instruments
|
|
- docstring: null
|
|
function: dump_bin.dump
|
|
- docstring: " \nParameters----------csv_path: strstock data path or directoryqlib_dir:\
|
|
\ strqlib(dump) data directorbackup_dir: str, default Noneif backup_dir is not\
|
|
\ None, backup qlib_dir to backup_dirfreq: str, default \"day\"transaction frequencymax_workers:\
|
|
\ int, default Nonenumber of threadsdate_field_name: str, default \"date\"the\
|
|
\ name of the date field in the csvfile_suffix: str, default \".csv\"file suffixsymbol_field_name:\
|
|
\ str, default \"symbol\"symbol field nameinclude_fields: tupledump fieldsexclude_fields:\
|
|
\ tuplefields not dumpedlimit_nums: intUse when debugging, default None"
|
|
function: dump_bin.DumpDataUpdate
|
|
- docstring: null
|
|
function: dump_bin._load_all_source_data
|
|
- docstring: null
|
|
function: dump_bin._read_csv
|
|
- docstring: null
|
|
function: dump_bin._dump_calendars
|
|
- docstring: null
|
|
function: dump_bin._dump_instruments
|
|
- docstring: null
|
|
function: dump_bin._dump_features
|
|
- docstring: " \nParameters----------qlib_dir : strqlib dircsv_path : strorigin\
|
|
\ csv pathcheck_fields : str, optionalcheck fields, by default None, check qlib_dir/features/<first_dir>/*.<freq>.binfreq\
|
|
\ : str, optionalfreq, value from [\"day\", \"1m\"]symbol_field_name: str, optionalsymbol\
|
|
\ field name, by default \"symbol\"date_field_name: str, optionaldate field name,\
|
|
\ by default \"date\"file_suffix: str, optionalcsv file suffix, by default \"\
|
|
.csv\"max_workers: int, optionalmax workers, by default 16"
|
|
function: 'check_dump_bin.CheckBin:'
|
|
- docstring: null
|
|
function: check_dump_bin._compare
|
|
- docstring: " \nParameters----------csv_path: strstock data path or directoryqlib_dir:\
|
|
\ strqlib(dump) data directorbackup_dir: str, default Noneif backup_dir is not\
|
|
\ None, backup qlib_dir to backup_dirfreq: str, default \"quarterly\"data frequencymax_workers:\
|
|
\ int, default Nonenumber of threadsdate_column_name: str, default \"date\"the\
|
|
\ name of the date field in the csvfile_suffix: str, default \".csv\"file suffixinclude_fields:\
|
|
\ tupledump fieldsexclude_fields: tuplefields not dumpedlimit_nums: intUse when\
|
|
\ debugging, default None"
|
|
function: 'dump_pit.DumpPitData:'
|
|
- docstring: null
|
|
function: dump_pit._backup_qlib_dir
|
|
- docstring: null
|
|
function: dump_pit.get_source_data
|
|
- docstring: null
|
|
function: dump_pit.get_symbol_from_file
|
|
- docstring: null
|
|
function: dump_pit.get_dump_fields
|
|
- docstring: null
|
|
function: dump_pit.get_filenames
|
|
- docstring: " \ndump data as the following format:`/path/to/<field>.data`[date,\
|
|
\ period, value, _next][date, period, value, _next][...]`/path/to/<field>.index`[first_year,\
|
|
\ index, index, ...]`<field.data>` contains the data as the point-in-time (PIT)\
|
|
\ order: `value` of `period`is published at `date`, and its successive revised\
|
|
\ value can be found at `_next` (linked list).`<field>.index` contains the index\
|
|
\ of value for each period (quarter or year). To savedisk space, we only store\
|
|
\ the `first_year` as its followings periods can be easily infered.Parameters----------symbol:\
|
|
\ strstock symbolinterval: strdata intervaloverwrite: boolwhether overwrite existing\
|
|
\ data or update only"
|
|
function: dump_pit._dump_pit
|
|
- docstring: ' get SH/SZ history calendar list
|
|
|
|
Parameters----------bench_code: strvalue from ["CSI300", "CSI500", "ALL", "US_ALL"]Returns-------history
|
|
calendar list'
|
|
function: utils.get_calendar_list
|
|
- docstring: null
|
|
function: utils._get_calendar
|
|
- docstring: null
|
|
function: utils._get_calendar
|
|
- docstring: null
|
|
function: utils.return_date_list
|
|
- docstring: ' get calendar list by selecting the date when few funds trade in
|
|
this day
|
|
|
|
Parameters----------source_dir: str or PathThe directory where the raw data collected
|
|
from the Internet is saveddate_field_name: strdate field name, default is datethreshold:
|
|
floatthreshold to exclude some days when few funds trade in this day, default
|
|
0.5minimum_count: intminimum count of funds should trade in one daymax_workers:
|
|
intConcurrent number, default is 16Returns-------history calendar list'
|
|
function: utils.get_calendar_list_by_ratio
|
|
- docstring: ' get SH/SZ stock symbols
|
|
|
|
Returns-------stock symbols'
|
|
function: utils.get_hs_stock_symbols
|
|
- docstring: null
|
|
function: utils._get_symbol
|
|
- docstring: ' get US stock symbols
|
|
|
|
Returns-------stock symbols'
|
|
function: utils.get_us_stock_symbols
|
|
- docstring: null
|
|
function: utils._get_eastmoney
|
|
- docstring: null
|
|
function: utils._get_nasdaq
|
|
- docstring: null
|
|
function: utils._get_nyse
|
|
- docstring: null
|
|
function: utils._format
|
|
- docstring: ' get IN stock symbols
|
|
|
|
Returns-------stock symbols'
|
|
function: utils.get_in_stock_symbols
|
|
- docstring: null
|
|
function: utils._get_nifty
|
|
- docstring: null
|
|
function: utils._format
|
|
- docstring: ' get Brazil(B3) stock symbols
|
|
|
|
Returns-------B3 stock symbols'
|
|
function: utils.get_br_stock_symbols
|
|
- docstring: null
|
|
function: utils._get_ibovespa
|
|
- docstring: null
|
|
function: utils._format
|
|
- docstring: ' get en fund symbols
|
|
|
|
Returns-------fund symbols in China'
|
|
function: utils.get_en_fund_symbols
|
|
- docstring: null
|
|
function: utils._get_eastmoney
|
|
- docstring: ' symbol suffix to prefix
|
|
|
|
Parameters----------symbol: strsymbolcapital : boolby default TrueReturns-------'
|
|
function: utils.symbol_suffix_to_prefix
|
|
- docstring: ' symbol prefix to sufix
|
|
|
|
Parameters----------symbol: strsymbolcapital : boolby default TrueReturns-------'
|
|
function: utils.symbol_prefix_to_sufix
|
|
- docstring: null
|
|
function: utils.deco_retry
|
|
- docstring: null
|
|
function: utils.deco_func
|
|
- docstring: null
|
|
function: utils.wrapper
|
|
- docstring: ' get trading date by shift
|
|
|
|
Parameters----------trading_list: listtrading calendar listshift : intshift, default
|
|
is 1trading_date : pd.Timestamptrading dateReturns-------'
|
|
function: utils.get_trading_date_by_shift
|
|
- docstring: ' generate minutes calendar
|
|
|
|
Parameters----------calendars: Iterabledaily calendarfreq: strby default 1minam_range:
|
|
Tuple[str, str]AM Time Range, by default China-Stock: ("09:30:00", "11:29:00")pm_range:
|
|
Tuple[str, str]PM Time Range, by default China-Stock: ("13:00:00", "14:59:00")'
|
|
function: utils.generate_minutes_calendar_from_daily
|
|
- docstring: " \nParameters----------qlib_dir: strqlib data dir, default \"Path(__file__).parent/qlib_data\"\
|
|
index_name: strindex name, value from [\"csi100\", \"csi300\"]method: strmethod,\
|
|
\ value from [\"parse_instruments\", \"save_new_companies\"]freq: strfreq, value\
|
|
\ from [\"day\", \"1min\"]request_retry: intrequest retry, by default 5retry_sleep:\
|
|
\ intrequest sleep, by default 3market_index: strWhere the files to obtain the\
|
|
\ index are located,for example data_collector.cn_index.collectorExamples-------#\
|
|
\ parse instruments$ python collector.py --index_name CSI300 --qlib_dir ~/.qlib/qlib_data/cn_data\
|
|
\ --method parse_instruments# parse new companies$ python collector.py --index_name\
|
|
\ CSI300 --qlib_dir ~/.qlib/qlib_data/cn_data --method save_new_companies"
|
|
function: utils.get_instruments
|
|
- docstring: " \nParameters----------save_dir: strinstrument save dirmax_workers:\
|
|
\ intworkers, default 1; Concurrent number, default is 1; when collecting data,\
|
|
\ it is recommended that max_workers be set to 1max_collector_count: intdefault\
|
|
\ 2delay: floattime.sleep(delay), default 0interval: strfreq, value from [1min,\
|
|
\ 1d], default 1dstart: strstart datetime, default Noneend: strend datetime, default\
|
|
\ Nonecheck_data_length: intcheck data length, if not None and greater than 0,\
|
|
\ each symbol will be considered complete if its data length is greater than or\
|
|
\ equal to this value, otherwise it will be fetched again, the maximum number\
|
|
\ of fetches being (max_collector_count). By default None.limit_nums: intusing\
|
|
\ for debug, by default None"
|
|
function: base.BaseCollector
|
|
- docstring: null
|
|
function: base.normalize_start_datetime
|
|
- docstring: null
|
|
function: base.normalize_end_datetime
|
|
- docstring: null
|
|
function: base.get_instrument_list
|
|
- docstring: ' normalize symbol
|
|
|
|
raise NotImplementedError("rewrite normalize_symbol")@abc.abstractmethod'
|
|
function: base.normalize_symbol
|
|
- docstring: ' get data with symbol
|
|
|
|
Parameters----------symbol: strinterval: strvalue from [1min, 1d]start_datetime:
|
|
pd.Timestampend_datetime: pd.TimestampReturns---------pd.DataFrame, "symbol" and
|
|
"date"in pd.columns'
|
|
function: base.get_data
|
|
- docstring: null
|
|
function: base.sleep
|
|
- docstring: " \nParameters----------symbol: str"
|
|
function: base._simple_collector
|
|
- docstring: ' save instrument data to file
|
|
|
|
Parameters----------symbol: strinstrument codedf : pd.DataFramedf.columns must
|
|
contain "symbol" and "datetime"'
|
|
function: base.save_instrument
|
|
- docstring: null
|
|
function: base.cache_small_data
|
|
- docstring: null
|
|
function: base._collector
|
|
- docstring: ' collector data
|
|
|
|
logger.info("start collector data......")instrument_list = self.instrument_listfor
|
|
i in range(self.max_collector_count):if not instrument_list:breaklogger.info(f"getting
|
|
data: {i+1}")instrument_list = self._collector(instrument_list)logger.info(f"{i+1}
|
|
finish.")for _symbol, _df_list in self.mini_symbol_map.items():_df = pd.concat(_df_list,
|
|
sort=False)if not _df.empty:self.save_instrument(_symbol, _df.drop_duplicates(["date"]).sort_values(["date"]))if
|
|
self.mini_symbol_map:logger.warning(f"less than {self.check_data_length} instrument
|
|
list: {list(self.mini_symbol_map.keys())}")logger.info(f"total {len(self.instrument_list)},
|
|
error: {len(set(instrument_list))}")'
|
|
function: base.collector_data
|
|
- docstring: " \nParameters----------date_field_name: strdate field name, default\
|
|
\ is datesymbol_field_name: strsymbol field name, default is symbol"
|
|
function: base.BaseNormalize
|
|
- docstring: null
|
|
function: base.normalize
|
|
- docstring: ' Get benchmark calendar
|
|
|
|
raise NotImplementedError("")'
|
|
function: base._get_calendar_list
|
|
- docstring: " \nParameters----------source_dir: str or PathThe directory where\
|
|
\ the raw data collected from the Internet is savedtarget_dir: str or PathDirectory\
|
|
\ for normalize datanormalize_class: Type[YahooNormalize]normalize classmax_workers:\
|
|
\ intConcurrent number, default is 16date_field_name: strdate field name, default\
|
|
\ is datesymbol_field_name: strsymbol field name, default is symbol"
|
|
function: 'base.Normalize:'
|
|
- docstring: null
|
|
function: base._executor
|
|
- docstring: null
|
|
function: base.normalize
|
|
- docstring: " \nParameters----------source_dir: strThe directory where the\
|
|
\ raw data collected from the Internet is saved, default \"Path(__file__).parent/source\"\
|
|
normalize_dir: strDirectory for normalize data, default \"Path(__file__).parent/normalize\"\
|
|
max_workers: intConcurrent number, default is 1; Concurrent number, default is\
|
|
\ 1; when collecting data, it is recommended that max_workers be set to 1interval:\
|
|
\ strfreq, value from [1min, 1d], default 1d"
|
|
function: base.BaseRun
|
|
- docstring: null
|
|
function: base.collector_class_name
|
|
- docstring: null
|
|
function: base.normalize_class_name
|
|
- docstring: null
|
|
function: base.default_base_dir
|
|
- docstring: ' download data from Internet
|
|
|
|
Parameters----------max_collector_count: intdefault 2delay: floattime.sleep(delay),
|
|
default 0start: strstart datetime, default "2000-01-01"end: strend datetime, default
|
|
``pd.Timestamp(datetime.datetime.now() + pd.Timedelta(days=1))``check_data_length:
|
|
intcheck data length, if not None and greater than 0, each symbol will be considered
|
|
complete if its data length is greater than or equal to this value, otherwise
|
|
it will be fetched again, the maximum number of fetches being (max_collector_count).
|
|
By default None.limit_nums: intusing for debug, by default NoneExamples---------#
|
|
get daily data$ python collector.py download_data --source_dir ~/.qlib/instrument_data/source
|
|
--region CN --start 2020-11-01 --end 2020-11-10 --delay 0.1 --interval 1d# get
|
|
1m data$ python collector.py download_data --source_dir ~/.qlib/instrument_data/source
|
|
--region CN --start 2020-11-01 --end 2020-11-10 --delay 0.1 --interval 1m'
|
|
function: base.download_data
|
|
- docstring: ' normalize data
|
|
|
|
Parameters----------date_field_name: strdate field name, default datesymbol_field_name:
|
|
strsymbol field name, default symbolExamples---------$ python collector.py normalize_data
|
|
--source_dir ~/.qlib/instrument_data/source --normalize_dir ~/.qlib/instrument_data/normalize
|
|
--region CN --interval 1d'
|
|
function: base.normalize_data
|
|
- docstring: " \nParameters----------index_name: strindex nameqlib_dir: strqlib\
|
|
\ directory, by default Path(__file__).resolve().parent.joinpath(\"qlib_data\"\
|
|
)freq: strfreq, value from [\"day\", \"1min\"]request_retry: intrequest retry,\
|
|
\ by default 5retry_sleep: intrequest sleep, by default 3"
|
|
function: 'index.IndexBase:'
|
|
- docstring: " \nReturns-------index start date"
|
|
function: index.bench_start_date
|
|
- docstring: ' get history trading date
|
|
|
|
Returns-------calendar list'
|
|
function: index.calendar_list
|
|
- docstring: " \nReturns-------pd.DataFrame:symbol start_date end_dateSH600000\
|
|
\ 2000-01-01 2099-12-31dtypes:symbol: strstart_date: pd.Timestampend_date:\
|
|
\ pd.Timestamp"
|
|
function: index.get_new_companies
|
|
- docstring: ' get companies changes
|
|
|
|
Returns-------pd.DataFrame:symbol date typeSH600000 2019-11-11 addSH600000 2020-11-10 removedtypes:symbol:
|
|
strdate: pd.Timestamptype: str, value from ["add", "remove"]'
|
|
function: index.get_changes
|
|
- docstring: ' formatting the datetime in an instrument
|
|
|
|
Parameters----------inst_df: pd.DataFrameinst_df.columns = [self.SYMBOL_FIELD_NAME,
|
|
self.START_DATE_FIELD, self.END_DATE_FIELD]Returns-------'
|
|
function: index.format_datetime
|
|
- docstring: ' save new companies
|
|
|
|
Examples-------$ python collector.py save_new_companies --index_name CSI300 --qlib_dir
|
|
~/.qlib/qlib_data/cn_data'
|
|
function: index.save_new_companies
|
|
- docstring: ' get changes with history companies
|
|
|
|
Parameters----------history_companies : pd.DataFramesymbol dateSH600000 2020-11-11dtypes:symbol:
|
|
strdate: pd.TimestampReturn--------pd.DataFrame:symbol date typeSH600000 2019-11-11 addSH600000 2020-11-10 removedtypes:symbol:
|
|
strdate: pd.Timestamptype: str, value from ["add", "remove"]'
|
|
function: index.get_changes_with_history_companies
|
|
- docstring: ' parse instruments, eg: csi300.txt
|
|
|
|
Examples-------$ python collector.py parse_instruments --index_name CSI300 --qlib_dir
|
|
~/.qlib/qlib_data/cn_data'
|
|
function: index.parse_instruments
|
|
- docstring: " \nParameters----------qlib_dir:qlib data directorystart_datestart\
|
|
\ dateend_dateend date"
|
|
function: 'future_calendar_collector.CollectorFutureCalendar:'
|
|
- docstring: null
|
|
function: future_calendar_collector.calendar_list
|
|
- docstring: null
|
|
function: future_calendar_collector._format_datetime
|
|
- docstring: null
|
|
function: future_calendar_collector.write_calendar
|
|
- docstring: " \nReturns-------"
|
|
function: future_calendar_collector.collector
|
|
- docstring: null
|
|
function: future_calendar_collector.CollectorFutureCalendarCN
|
|
- docstring: null
|
|
function: future_calendar_collector.collector
|
|
- docstring: null
|
|
function: future_calendar_collector.CollectorFutureCalendarUS
|
|
- docstring: null
|
|
function: future_calendar_collector.collector
|
|
- docstring: ' Collect future calendar(day)
|
|
|
|
Parameters----------qlib_dir:qlib data directoryregion:cn/CN or us/USstart_datestart
|
|
dateend_dateend dateExamples-------# get cn future calendar$ python future_calendar_collector.py
|
|
--qlib_data_1d_dir <user data dir> --region cn'
|
|
function: future_calendar_collector.run
|
|
- docstring: ' get crypto symbols in coingecko
|
|
|
|
Returns-------crypto symbols in given exchanges list of coingecko'
|
|
function: collector.get_cg_crypto_symbols
|
|
- docstring: null
|
|
function: collector._get_coingecko
|
|
- docstring: " \nParameters----------save_dir: strcrypto save dirmax_workers:\
|
|
\ intworkers, default 4max_collector_count: intdefault 2delay: floattime.sleep(delay),\
|
|
\ default 0interval: strfreq, value from [1min, 1d], default 1minstart: strstart\
|
|
\ datetime, default Noneend: strend datetime, default Nonecheck_data_length: intcheck\
|
|
\ data length, if not None and greater than 0, each symbol will be considered\
|
|
\ complete if its data length is greater than or equal to this value, otherwise\
|
|
\ it will be fetched again, the maximum number of fetches being (max_collector_count).\
|
|
\ By default None.limit_nums: intusing for debug, by default None"
|
|
function: collector.CryptoCollector
|
|
- docstring: null
|
|
function: collector.init_datetime
|
|
- docstring: null
|
|
function: collector.convert_datetime
|
|
- docstring: null
|
|
function: collector._timezone
|
|
- docstring: null
|
|
function: collector.get_data_from_remote
|
|
- docstring: null
|
|
function: collector.get_data
|
|
- docstring: null
|
|
function: collector._get_simple
|
|
- docstring: null
|
|
function: collector.CryptoCollector1d
|
|
- docstring: null
|
|
function: collector.get_instrument_list
|
|
- docstring: null
|
|
function: collector.normalize_symbol
|
|
- docstring: null
|
|
function: collector._timezone
|
|
- docstring: null
|
|
function: collector.CryptoNormalize
|
|
- docstring: null
|
|
function: collector.normalize_crypto
|
|
- docstring: null
|
|
function: collector.normalize
|
|
- docstring: null
|
|
function: collector.CryptoNormalize1d
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: " \nParameters----------source_dir: strThe directory where the\
|
|
\ raw data collected from the Internet is saved, default \"Path(__file__).parent/source\"\
|
|
normalize_dir: strDirectory for normalize data, default \"Path(__file__).parent/normalize\"\
|
|
max_workers: intConcurrent number, default is 1interval: strfreq, value from [1min,\
|
|
\ 1d], default 1d"
|
|
function: collector.Run
|
|
- docstring: null
|
|
function: collector.collector_class_name
|
|
- docstring: null
|
|
function: collector.normalize_class_name
|
|
- docstring: null
|
|
function: collector.default_base_dir
|
|
- docstring: ' download data from Internet
|
|
|
|
Parameters----------max_collector_count: intdefault 2delay: floattime.sleep(delay),
|
|
default 0interval: strfreq, value from [1min, 1d], default 1d, currently only
|
|
supprot 1dstart: strstart datetime, default "2000-01-01"end: strend datetime,
|
|
default ``pd.Timestamp(datetime.datetime.now() + pd.Timedelta(days=1))``check_data_length:
|
|
int # if this param useful?check data length, if not None and greater than 0,
|
|
each symbol will be considered complete if its data length is greater than or
|
|
equal to this value, otherwise it will be fetched again, the maximum number of
|
|
fetches being (max_collector_count). By default None.limit_nums: intusing for
|
|
debug, by default NoneExamples---------# get daily data$ python collector.py download_data
|
|
--source_dir ~/.qlib/crypto_data/source/1d --start 2015-01-01 --end 2021-11-30
|
|
--delay 1 --interval 1d'
|
|
function: collector.download_data
|
|
- docstring: ' normalize data
|
|
|
|
Parameters----------date_field_name: strdate field name, default datesymbol_field_name:
|
|
strsymbol field name, default symbolExamples---------$ python collector.py normalize_data
|
|
--source_dir ~/.qlib/crypto_data/source/1d --normalize_dir ~/.qlib/crypto_data/source/1d_nor
|
|
--interval 1d --date_field_name date'
|
|
function: collector.normalize_data
|
|
- docstring: null
|
|
function: collector.WIKIIndex
|
|
- docstring: " \nReturns-------index start date"
|
|
function: collector.bench_start_date
|
|
- docstring: ' get companies changes
|
|
|
|
Returns-------pd.DataFrame:symbol date typeSH600000 2019-11-11 addSH600000 2020-11-10 removedtypes:symbol:
|
|
strdate: pd.Timestamptype: str, value from ["add", "remove"]'
|
|
function: collector.get_changes
|
|
- docstring: ' formatting the datetime in an instrument
|
|
|
|
Parameters----------inst_df: pd.DataFrameinst_df.columns = [self.SYMBOL_FIELD_NAME,
|
|
self.START_DATE_FIELD, self.END_DATE_FIELD]Returns-------'
|
|
function: collector.format_datetime
|
|
- docstring: ' get history trading date
|
|
|
|
Returns-------calendar list'
|
|
function: collector.calendar_list
|
|
- docstring: null
|
|
function: collector._request_new_companies
|
|
- docstring: null
|
|
function: collector.set_default_date_range
|
|
- docstring: null
|
|
function: collector.get_new_companies
|
|
- docstring: null
|
|
function: collector.filter_df
|
|
- docstring: null
|
|
function: collector.NASDAQ100Index
|
|
- docstring: null
|
|
function: collector.filter_df
|
|
- docstring: null
|
|
function: collector.bench_start_date
|
|
- docstring: null
|
|
function: collector._request_history_companies
|
|
- docstring: null
|
|
function: collector.get_history_companies
|
|
- docstring: null
|
|
function: collector.get_changes
|
|
- docstring: null
|
|
function: collector.DJIAIndex
|
|
- docstring: null
|
|
function: collector.bench_start_date
|
|
- docstring: null
|
|
function: collector.get_changes
|
|
- docstring: null
|
|
function: collector.filter_df
|
|
- docstring: null
|
|
function: collector.parse_instruments
|
|
- docstring: null
|
|
function: collector.SP500Index
|
|
- docstring: null
|
|
function: collector.bench_start_date
|
|
- docstring: null
|
|
function: collector.get_changes
|
|
- docstring: null
|
|
function: collector.filter_df
|
|
- docstring: null
|
|
function: collector.SP400Index
|
|
- docstring: null
|
|
function: collector.bench_start_date
|
|
- docstring: null
|
|
function: collector.get_changes
|
|
- docstring: null
|
|
function: collector.filter_df
|
|
- docstring: " \nParameters----------save_dir: strinstrument save dirmax_workers:\
|
|
\ intworkers, default 1; Concurrent number, default is 1; when collecting data,\
|
|
\ it is recommended that max_workers be set to 1max_collector_count: intdefault\
|
|
\ 2delay: floattime.sleep(delay), default 0interval: strfreq, value from [1min,\
|
|
\ 1d], default 1dstart: strstart datetime, default Noneend: strend datetime, default\
|
|
\ Nonecheck_data_length: intcheck data length, if not None and greater than 0,\
|
|
\ each symbol will be considered complete if its data length is greater than or\
|
|
\ equal to this value, otherwise it will be fetched again, the maximum number\
|
|
\ of fetches being (max_collector_count). By default None.limit_nums: intusing\
|
|
\ for debug, by default Nonesymbol_regex: strsymbol regular expression, by default\
|
|
\ None."
|
|
function: collector.PitCollector
|
|
- docstring: null
|
|
function: collector.get_instrument_list
|
|
- docstring: null
|
|
function: collector.normalize_symbol
|
|
- docstring: null
|
|
function: collector.get_performance_express_report_df
|
|
- docstring: null
|
|
function: collector.get_profit_df
|
|
- docstring: null
|
|
function: collector.get_forecast_report_df
|
|
- docstring: null
|
|
function: collector.get_growth_df
|
|
- docstring: null
|
|
function: collector.get_data
|
|
- docstring: null
|
|
function: collector.PitNormalize
|
|
- docstring: null
|
|
function: collector.normalize
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector.Run
|
|
- docstring: null
|
|
function: collector.collector_class_name
|
|
- docstring: null
|
|
function: collector.normalize_class_name
|
|
- docstring: null
|
|
function: fill_cn_1min_data.get_date_range
|
|
- docstring: null
|
|
function: fill_cn_1min_data.get_symbols
|
|
- docstring: ' Use 1d data to fill in the missing symbols relative to 1min
|
|
|
|
Parameters----------data_1min_dir: str1min data dirqlib_data_1d_dir: str1d qlib
|
|
data(bin data) dir, from: https://qlib.readthedocs.io/en/latest/component/data.html#converting-csv-format-into-qlib-formatmax_workers:
|
|
intThreadPoolExecutor(max_workers), by default 16date_field_name: strdate field
|
|
name, by default datesymbol_field_name: strsymbol field name, by default symbol'
|
|
function: fill_cn_1min_data.fill_1min_using_1d
|
|
- docstring: null
|
|
function: future_trading_date_collector.read_calendar_from_qlib
|
|
- docstring: null
|
|
function: future_trading_date_collector.write_calendar_to_qlib
|
|
- docstring: null
|
|
function: future_trading_date_collector.generate_qlib_calendar
|
|
- docstring: ' get future calendar
|
|
|
|
Parameters----------qlib_dir: str or Pathqlib data directoryfreq: strvalue from
|
|
["day", "1min"], by default day'
|
|
function: future_trading_date_collector.future_calendar_collector
|
|
- docstring: null
|
|
function: collector.retry_request
|
|
- docstring: null
|
|
function: collector.CSIIndex
|
|
- docstring: ' get history trading date
|
|
|
|
Returns-------calendar list'
|
|
function: collector.calendar_list
|
|
- docstring: null
|
|
function: collector.new_companies_url
|
|
- docstring: null
|
|
function: collector.changes_url
|
|
- docstring: " \nReturns-------index start date"
|
|
function: collector.bench_start_date
|
|
- docstring: " \nReturns-------index code"
|
|
function: collector.index_code
|
|
- docstring: ' Which table of changes in html
|
|
|
|
CSI300: 0CSI100: 1:return:'
|
|
function: collector.html_table_index
|
|
- docstring: ' formatting the datetime in an instrument
|
|
|
|
Parameters----------inst_df: pd.DataFrameinst_df.columns = [self.SYMBOL_FIELD_NAME,
|
|
self.START_DATE_FIELD, self.END_DATE_FIELD]Returns-------'
|
|
function: collector.format_datetime
|
|
- docstring: ' get companies changes
|
|
|
|
Returns-------pd.DataFrame:symbol date typeSH600000 2019-11-11 addSH600000 2020-11-10 removedtypes:symbol:
|
|
strdate: pd.Timestamptype: str, value from ["add", "remove"]'
|
|
function: collector.get_changes
|
|
- docstring: " \nParameters----------symbol: strsymbolReturns-------symbol"
|
|
function: collector.normalize_symbol
|
|
- docstring: null
|
|
function: collector._parse_excel
|
|
- docstring: null
|
|
function: collector._parse_table
|
|
- docstring: ' read change from url
|
|
|
|
The parameter url is from the _get_change_notices_url method.Determine the stock
|
|
add_date/remove_date based on the title.The response contains three cases:1.Only
|
|
excel_url(extract data from excel_url)2.Both the excel_url and the body text(try
|
|
to extract data from excel_url first, and then try to extract data from body text)3.Only
|
|
body text(extract data from body text)Parameters----------url : strchange urlReturns-------pd.DataFrame:symbol date typeSH600000 2019-11-11 addSH600000 2020-11-10 removedtypes:symbol:
|
|
strdate: pd.Timestamptype: str, value from ["add", "remove"]'
|
|
function: collector._read_change_from_url
|
|
- docstring: ' get change notices url
|
|
|
|
Returns-------[url1, url2]'
|
|
function: collector._get_change_notices_url
|
|
- docstring: " \nReturns-------pd.DataFrame:symbol start_date end_dateSH600000\
|
|
\ 2000-01-01 2099-12-31dtypes:symbol: strstart_date: pd.Timestampend_date:\
|
|
\ pd.Timestamp"
|
|
function: collector.get_new_companies
|
|
- docstring: null
|
|
function: collector.CSI300Index
|
|
- docstring: null
|
|
function: collector.index_code
|
|
- docstring: null
|
|
function: collector.bench_start_date
|
|
- docstring: null
|
|
function: collector.html_table_index
|
|
- docstring: null
|
|
function: collector.CSI100Index
|
|
- docstring: null
|
|
function: collector.index_code
|
|
- docstring: null
|
|
function: collector.bench_start_date
|
|
- docstring: null
|
|
function: collector.html_table_index
|
|
- docstring: null
|
|
function: collector.CSI500Index
|
|
- docstring: null
|
|
function: collector.index_code
|
|
- docstring: null
|
|
function: collector.bench_start_date
|
|
- docstring: ' get companies changes
|
|
|
|
Return--------pd.DataFrame:symbol date typeSH600000 2019-11-11 addSH600000 2020-11-10 removedtypes:symbol:
|
|
strdate: pd.Timestamptype: str, value from ["add", "remove"]'
|
|
function: collector.get_changes
|
|
- docstring: " \nReturns-------pd.DataFrame:symbol date typeSH600000\
|
|
\ 2019-11-11 addSH600000 2020-11-10 removedtypes:symbol: strdate: pd.Timestamptype:\
|
|
\ str, value from [\"add\", \"remove\"]"
|
|
function: collector.get_history_companies
|
|
- docstring: " \nData source: http://baostock.com/baostock/index.php/%E4%B8%AD%E8%AF%81500%E6%88%90%E5%88%86%E8%82%A1Avoid\
|
|
\ a large number of parallel data acquisition,such as 1000 times of concurrent\
|
|
\ data acquisition, because IP will be blockedReturns-------pd.DataFrame:date\
|
|
\ symbol code_nameSH600039 2007-01-15 \u56DB\u5DDD\u8DEF\u6865\
|
|
SH600051 2020-01-15 \u5B81\u6CE2\u8054\u5408dtypes:date: pd.Timestampsymbol:\
|
|
\ strcode_name: str"
|
|
function: collector.get_data_from_baostock
|
|
- docstring: " \nReturns-------pd.DataFrame:symbol start_date end_dateSH600000\
|
|
\ 2000-01-01 2099-12-31dtypes:symbol: strstart_date: pd.Timestampend_date:\
|
|
\ pd.Timestamp"
|
|
function: collector.get_new_companies
|
|
- docstring: null
|
|
function: collector.IBOVIndex
|
|
- docstring: " \nThe ibovespa index started on 2 January 1968 (wiki), however,no\
|
|
\ suitable data source that keeps track of ibovespa's historystocks composition\
|
|
\ has been found. Except from the repo indicatedin README. Which keeps track of\
|
|
\ such information starting fromthe first quarter of 2003"
|
|
function: collector.bench_start_date
|
|
- docstring: " \nThis function is used to calculated what is the currentfour\
|
|
\ month period for the current month. For example,If the current month is August\
|
|
\ 8, its four month periodis 2Q.OBS: In english Q is used to represent *quarter*which\
|
|
\ means a three month period. However, inportuguese we use Q to represent a four\
|
|
\ month period.In other words,Jan, Feb, Mar, Apr: 1QMay, Jun, Jul, Aug: 2QSep,\
|
|
\ Oct, Nov, Dez: 3QParameters----------month : intCurrent month (1 <= month <=\
|
|
\ 12)Returns-------current_4m_period:strCurrent Four Month Period (1Q or 2Q or\
|
|
\ 3Q)"
|
|
function: collector.get_current_4_month_period
|
|
- docstring: " \nThe ibovespa index is updated every four months.Therefore,\
|
|
\ we will represent each time period as 2003_1Qwhich means 2003 first four mount\
|
|
\ period (Jan, Feb, Mar, Apr)"
|
|
function: collector.get_four_month_period
|
|
- docstring: ' formatting the datetime in an instrument
|
|
|
|
Parameters----------inst_df: pd.DataFrameinst_df.columns = [self.SYMBOL_FIELD_NAME,
|
|
self.START_DATE_FIELD, self.END_DATE_FIELD]Returns-------inst_df: pd.DataFrame'
|
|
function: collector.format_datetime
|
|
- docstring: " \nParameters----------cell: strIt must be on the format 2003_1Q\
|
|
\ --> years_4_month_periodsReturns----------date: strReturns date in format 2003-03-01"
|
|
function: collector.format_quarter
|
|
- docstring: " \nAccess the index historic composition and compare it quarterby\
|
|
\ quarter and year by year in order to generate a file thatkeeps track of which\
|
|
\ stocks have been removed and which havebeen added.The Dataframe used as reference\
|
|
\ will provided the indexcomposition for each year an quarter:pd.DataFrame:symbolSH600000SH600001...Parameters----------self:\
|
|
\ is used to represent the instance of the class.Returns----------pd.DataFrame:symbol\
|
|
\ date typeSH600000 2019-11-11 addSH600001 2020-11-10 removedtypes:symbol:\
|
|
\ strdate: pd.Timestamptype: str, value from [\"add\", \"remove\"]"
|
|
function: collector.get_changes
|
|
- docstring: " \nGet latest index composition.The repo indicated on README\
|
|
\ has implemented a scriptto get the latest index composition from B3 website\
|
|
\ usingselenium. Therefore, this method will download the filecontaining such\
|
|
\ compositionParameters----------self: is used to represent the instance of the\
|
|
\ class.Returns----------pd.DataFrame:symbol start_date end_dateRRRP3\t\
|
|
\ 2020-11-13\t2022-03-02ALPA4\t 2008-01-02\t2022-03-02dtypes:symbol: strstart_date:\
|
|
\ pd.Timestampend_date: pd.Timestamp"
|
|
function: collector.get_new_companies
|
|
- docstring: " \nParameters----------save_dir: strstock save dirmax_workers:\
|
|
\ intworkers, default 4max_collector_count: intdefault 2delay: floattime.sleep(delay),\
|
|
\ default 0interval: strfreq, value from [1min, 1d], default 1minstart: strstart\
|
|
\ datetime, default Noneend: strend datetime, default Nonecheck_data_length: intcheck\
|
|
\ data length, by default Nonelimit_nums: intusing for debug, by default None"
|
|
function: collector.YahooCollector
|
|
- docstring: null
|
|
function: collector.init_datetime
|
|
- docstring: null
|
|
function: collector.convert_datetime
|
|
- docstring: null
|
|
function: collector._timezone
|
|
- docstring: null
|
|
function: collector.get_data_from_remote
|
|
- docstring: null
|
|
function: collector._show_logging_func
|
|
- docstring: null
|
|
function: collector.get_data
|
|
- docstring: null
|
|
function: collector._get_simple
|
|
- docstring: ' collector data
|
|
|
|
super(YahooCollector, self).collector_data()self.download_index_data()@abc.abstractmethod'
|
|
function: collector.collector_data
|
|
- docstring: ' download index data
|
|
|
|
raise NotImplementedError("rewrite download_index_data")'
|
|
function: collector.download_index_data
|
|
- docstring: null
|
|
function: collector.YahooCollectorCN
|
|
- docstring: null
|
|
function: collector.get_instrument_list
|
|
- docstring: null
|
|
function: collector.normalize_symbol
|
|
- docstring: null
|
|
function: collector._timezone
|
|
- docstring: null
|
|
function: collector.YahooCollectorCN1d
|
|
- docstring: null
|
|
function: collector.download_index_data
|
|
- docstring: null
|
|
function: collector.YahooCollectorCN1min
|
|
- docstring: null
|
|
function: collector.get_instrument_list
|
|
- docstring: null
|
|
function: collector.download_index_data
|
|
- docstring: null
|
|
function: collector.YahooCollectorUS
|
|
- docstring: null
|
|
function: collector.get_instrument_list
|
|
- docstring: null
|
|
function: collector.download_index_data
|
|
- docstring: null
|
|
function: collector.normalize_symbol
|
|
- docstring: null
|
|
function: collector._timezone
|
|
- docstring: null
|
|
function: collector.YahooCollectorUS1d
|
|
- docstring: null
|
|
function: collector.YahooCollectorUS1min
|
|
- docstring: null
|
|
function: collector.YahooCollectorIN
|
|
- docstring: null
|
|
function: collector.get_instrument_list
|
|
- docstring: null
|
|
function: collector.download_index_data
|
|
- docstring: null
|
|
function: collector.normalize_symbol
|
|
- docstring: null
|
|
function: collector._timezone
|
|
- docstring: null
|
|
function: collector.YahooCollectorIN1d
|
|
- docstring: null
|
|
function: collector.YahooCollectorIN1min
|
|
- docstring: null
|
|
function: collector.YahooCollectorBR
|
|
- docstring: " \nThe reason to use retry=2 is due to the fact thatYahoo Finance\
|
|
\ unfortunately does not keep track of someBrazilian stocks.Therefore, the decorator\
|
|
\ deco_retry with retry argumentset to 5 will keep trying to get the stock data\
|
|
\ up to 5 times,which makes the code to download Brazilians stocks very slow.In\
|
|
\ future, this may change, but for nowI suggest to leave retry argument to 1 or\
|
|
\ 2 inorder to improve download speed.To achieve this goal an abstract attribute\
|
|
\ (retry)was added into YahooCollectorBR base class"
|
|
function: collector.retry
|
|
- docstring: null
|
|
function: collector.get_instrument_list
|
|
- docstring: null
|
|
function: collector.download_index_data
|
|
- docstring: null
|
|
function: collector.normalize_symbol
|
|
- docstring: null
|
|
function: collector._timezone
|
|
- docstring: null
|
|
function: collector.YahooCollectorBR1d
|
|
- docstring: null
|
|
function: collector.YahooCollectorBR1min
|
|
- docstring: null
|
|
function: collector.YahooNormalize
|
|
- docstring: null
|
|
function: collector.calc_change
|
|
- docstring: null
|
|
function: collector.normalize_yahoo
|
|
- docstring: null
|
|
function: collector.normalize
|
|
- docstring: ' adjusted price
|
|
|
|
raise NotImplementedError("rewrite adjusted_price")'
|
|
function: collector.adjusted_price
|
|
- docstring: null
|
|
function: collector.YahooNormalize1d
|
|
- docstring: null
|
|
function: collector.adjusted_price
|
|
- docstring: null
|
|
function: collector.normalize
|
|
- docstring: ' get first close value
|
|
|
|
Notes-----For incremental updates(append) to Yahoo 1D data, user need to use a
|
|
close that is not 0 on the first trading day of the existing data'
|
|
function: collector._get_first_close
|
|
- docstring: ' manual adjust data: All fields (except change) are standardized
|
|
according to the close of the first day
|
|
|
|
if df.empty:return dfdf = df.copy()df.sort_values(self._date_field_name, inplace=True)df
|
|
= df.set_index(self._date_field_name)_close = self._get_first_close(df)for _col
|
|
in df.columns:# NOTE: retain original adjclose, required for incremental updatesif
|
|
_col in [self._symbol_field_name, "adjclose", "change"]:continueif _col == "volume":df[_col]
|
|
= df[_col] * _closeelse:df[_col] = df[_col] / _closereturn df.reset_index()'
|
|
function: collector._manual_adj_data
|
|
- docstring: " \nParameters----------old_qlib_data_dir: str, Paththe qlib data\
|
|
\ to be updated for yahoo, usually from: https://github.com/microsoft/qlib/tree/main/scripts#download-cn-datadate_field_name:\
|
|
\ strdate field name, default is datesymbol_field_name: strsymbol field name,\
|
|
\ default is symbol"
|
|
function: collector.YahooNormalize1dExtend
|
|
- docstring: null
|
|
function: collector._get_old_data
|
|
- docstring: null
|
|
function: collector._get_close
|
|
- docstring: null
|
|
function: collector._get_first_close
|
|
- docstring: null
|
|
function: collector._get_last_close
|
|
- docstring: null
|
|
function: collector._get_last_date
|
|
- docstring: null
|
|
function: collector.normalize
|
|
- docstring: null
|
|
function: collector.YahooNormalize1min
|
|
- docstring: null
|
|
function: collector.calendar_list_1d
|
|
- docstring: null
|
|
function: collector.generate_1min_from_daily
|
|
- docstring: ' get 1d data
|
|
|
|
Returns------data_1d: pd.DataFramedata_1d.columns = [self._date_field_name, self._symbol_field_name,
|
|
"paused", "volume", "factor", "close"]'
|
|
function: collector.get_1d_data
|
|
- docstring: null
|
|
function: collector.adjusted_price
|
|
- docstring: null
|
|
function: collector._calc_factor
|
|
- docstring: null
|
|
function: collector.calc_paused_num
|
|
- docstring: null
|
|
function: collector.symbol_to_yahoo
|
|
- docstring: null
|
|
function: collector._get_1d_calendar_list
|
|
- docstring: ' Normalised to 1min using local 1d data
|
|
|
|
self, qlib_data_1d_dir: [str, Path], date_field_name: str = "date", symbol_field_name:
|
|
str = "symbol", **kwargs):'
|
|
function: collector.YahooNormalize1minOffline
|
|
- docstring: null
|
|
function: collector._get_1d_calendar_list
|
|
- docstring: null
|
|
function: collector._get_all_1d_data
|
|
- docstring: ' get 1d data
|
|
|
|
Returns------data_1d: pd.DataFramedata_1d.columns = [self._date_field_name, self._symbol_field_name,
|
|
"paused", "volume", "factor", "close"]'
|
|
function: collector.get_1d_data
|
|
- docstring: null
|
|
function: 'collector.YahooNormalizeUS:'
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector.YahooNormalizeUS1d
|
|
- docstring: null
|
|
function: collector.YahooNormalizeUS1dExtend
|
|
- docstring: null
|
|
function: collector.YahooNormalizeUS1min
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector._get_1d_calendar_list
|
|
- docstring: null
|
|
function: collector.symbol_to_yahoo
|
|
- docstring: null
|
|
function: 'collector.YahooNormalizeIN:'
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector.YahooNormalizeIN1d
|
|
- docstring: null
|
|
function: collector.YahooNormalizeIN1min
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector._get_1d_calendar_list
|
|
- docstring: null
|
|
function: collector.symbol_to_yahoo
|
|
- docstring: null
|
|
function: 'collector.YahooNormalizeCN:'
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector.YahooNormalizeCN1d
|
|
- docstring: null
|
|
function: collector.YahooNormalizeCN1dExtend
|
|
- docstring: null
|
|
function: collector.YahooNormalizeCN1min
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector.symbol_to_yahoo
|
|
- docstring: null
|
|
function: collector._get_1d_calendar_list
|
|
- docstring: null
|
|
function: 'collector.YahooNormalizeBR:'
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector.YahooNormalizeBR1d
|
|
- docstring: null
|
|
function: collector.YahooNormalizeBR1min
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector._get_1d_calendar_list
|
|
- docstring: null
|
|
function: collector.symbol_to_yahoo
|
|
- docstring: " \nParameters----------source_dir: strThe directory where the\
|
|
\ raw data collected from the Internet is saved, default \"Path(__file__).parent/source\"\
|
|
normalize_dir: strDirectory for normalize data, default \"Path(__file__).parent/normalize\"\
|
|
max_workers: intConcurrent number, default is 1; when collecting data, it is recommended\
|
|
\ that max_workers be set to 1interval: strfreq, value from [1min, 1d], default\
|
|
\ 1dregion: strregion, value from [\"CN\", \"US\", \"BR\"], default \"CN\""
|
|
function: collector.Run
|
|
- docstring: null
|
|
function: collector.collector_class_name
|
|
- docstring: null
|
|
function: collector.normalize_class_name
|
|
- docstring: null
|
|
function: collector.default_base_dir
|
|
- docstring: ' download data from Internet
|
|
|
|
Parameters----------max_collector_count: intdefault 2delay: floattime.sleep(delay),
|
|
default 0.5start: strstart datetime, default "2000-01-01"; closed interval(including
|
|
start)end: strend datetime, default ``pd.Timestamp(datetime.datetime.now() + pd.Timedelta(days=1))``;
|
|
open interval(excluding end)check_data_length: intcheck data length, if not None
|
|
and greater than 0, each symbol will be considered complete if its data length
|
|
is greater than or equal to this value, otherwise it will be fetched again, the
|
|
maximum number of fetches being (max_collector_count). By default None.limit_nums:
|
|
intusing for debug, by default NoneNotes-----check_data_length, example:daily,
|
|
one year: 252 // 4us 1min, a week: 6.5 * 60 * 5cn 1min, a week: 4 * 60 * 5Examples---------#
|
|
get daily data$ python collector.py download_data --source_dir ~/.qlib/stock_data/source
|
|
--region CN --start 2020-11-01 --end 2020-11-10 --delay 0.1 --interval 1d# get
|
|
1m data$ python collector.py download_data --source_dir ~/.qlib/stock_data/source
|
|
--region CN --start 2020-11-01 --end 2020-11-10 --delay 0.1 --interval 1m'
|
|
function: collector.download_data
|
|
- docstring: ' normalize data
|
|
|
|
Parameters----------date_field_name: strdate field name, default datesymbol_field_name:
|
|
strsymbol field name, default symbolend_date: strif not None, normalize the last
|
|
date saved (including end_date); if None, it will ignore this parameter; by default
|
|
Noneqlib_data_1d_dir: strif interval==1min, qlib_data_1d_dir cannot be None, normalize
|
|
1min needs to use 1d data;qlib_data_1d can be obtained like this:$ python scripts/get_data.py
|
|
qlib_data --target_dir <qlib_data_1d_dir> --interval 1d$ python scripts/data_collector/yahoo/collector.py
|
|
update_data_to_bin --qlib_data_1d_dir <qlib_data_1d_dir> --trading_date 2021-06-01or:download
|
|
1d data, reference: https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo#1d-from-yahooExamples---------$
|
|
python collector.py normalize_data --source_dir ~/.qlib/stock_data/source --normalize_dir
|
|
~/.qlib/stock_data/normalize --region cn --interval 1d$ python collector.py normalize_data
|
|
--qlib_data_1d_dir ~/.qlib/qlib_data/cn_data --source_dir ~/.qlib/stock_data/source_cn_1min
|
|
--normalize_dir ~/.qlib/stock_data/normalize_cn_1min --region CN --interval 1min'
|
|
function: collector.normalize_data
|
|
- docstring: ' normalize data extend; extending yahoo qlib data(from: https://github.com/microsoft/qlib/tree/main/scripts#download-cn-data)
|
|
|
|
Notes-----Steps to extend yahoo qlib data:1. download qlib data: https://github.com/microsoft/qlib/tree/main/scripts#download-cn-data;
|
|
save to <dir1>2. collector source data: https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo#collector-data;
|
|
save to <dir2>3. normalize new source data(from step 2): python scripts/data_collector/yahoo/collector.py
|
|
normalize_data_1d_extend --old_qlib_dir <dir1> --source_dir <dir2> --normalize_dir
|
|
<dir3> --region CN --interval 1d4. dump data: python scripts/dump_bin.py dump_update
|
|
--csv_path <dir3> --qlib_dir <dir1> --freq day --date_field_name date --symbol_field_name
|
|
symbol --exclude_fields symbol,date5. update instrument(eg. csi300): python python
|
|
scripts/data_collector/cn_index/collector.py --index_name CSI300 --qlib_dir <dir1>
|
|
--method parse_instrumentsParameters----------old_qlib_data_dir: strthe qlib data
|
|
to be updated for yahoo, usually from: https://github.com/microsoft/qlib/tree/main/scripts#download-cn-datadate_field_name:
|
|
strdate field name, default datesymbol_field_name: strsymbol field name, default
|
|
symbolExamples---------$ python collector.py normalize_data_1d_extend --old_qlib_dir
|
|
~/.qlib/qlib_data/cn_data --source_dir ~/.qlib/stock_data/source --normalize_dir
|
|
~/.qlib/stock_data/normalize --region CN --interval 1d'
|
|
function: collector.normalize_data_1d_extend
|
|
- docstring: ' download today data from Internet
|
|
|
|
Parameters----------max_collector_count: intdefault 2delay: floattime.sleep(delay),
|
|
default 0.5check_data_length: intcheck data length, if not None and greater than
|
|
0, each symbol will be considered complete if its data length is greater than
|
|
or equal to this value, otherwise it will be fetched again, the maximum number
|
|
of fetches being (max_collector_count). By default None.limit_nums: intusing for
|
|
debug, by default NoneNotes-----Download today''s data:start_time = datetime.datetime.now().date();
|
|
closed interval(including start)end_time = pd.Timestamp(start_time + pd.Timedelta(days=1)).date();
|
|
open interval(excluding end)check_data_length, example:daily, one year: 252 //
|
|
4us 1min, a week: 6.5 * 60 * 5cn 1min, a week: 4 * 60 * 5Examples---------# get
|
|
daily data$ python collector.py download_today_data --source_dir ~/.qlib/stock_data/source
|
|
--region CN --delay 0.1 --interval 1d# get 1m data$ python collector.py download_today_data
|
|
--source_dir ~/.qlib/stock_data/source --region CN --delay 0.1 --interval 1m'
|
|
function: collector.download_today_data
|
|
- docstring: ' update yahoo data to bin
|
|
|
|
Parameters----------qlib_data_1d_dir: strthe qlib data to be updated for yahoo,
|
|
usually from: https://github.com/microsoft/qlib/tree/main/scripts#download-cn-datatrading_date:
|
|
strtrading days to be updated, by default ``datetime.datetime.now().strftime("%Y-%m-%d")``end_date:
|
|
strend datetime, default ``pd.Timestamp(trading_date + pd.Timedelta(days=1))``;
|
|
open interval(excluding end)check_data_length: intcheck data length, if not None
|
|
and greater than 0, each symbol will be considered complete if its data length
|
|
is greater than or equal to this value, otherwise it will be fetched again, the
|
|
maximum number of fetches being (max_collector_count). By default None.delay:
|
|
floattime.sleep(delay), default 1Notes-----If the data in qlib_data_dir is incomplete,
|
|
np.nan will be populated to trading_date for the previous trading dayExamples-------$
|
|
python collector.py update_data_to_bin --qlib_data_1d_dir <user data dir> --trading_date
|
|
<start date> --end_date <end date># get 1m data'
|
|
function: collector.update_data_to_bin
|
|
- docstring: " \nParameters----------save_dir: strfund save dirmax_workers:\
|
|
\ intworkers, default 4max_collector_count: intdefault 2delay: floattime.sleep(delay),\
|
|
\ default 0interval: strfreq, value from [1min, 1d], default 1minstart: strstart\
|
|
\ datetime, default Noneend: strend datetime, default Nonecheck_data_length: intcheck\
|
|
\ data length, if not None and greater than 0, each symbol will be considered\
|
|
\ complete if its data length is greater than or equal to this value, otherwise\
|
|
\ it will be fetched again, the maximum number of fetches being (max_collector_count).\
|
|
\ By default None.limit_nums: intusing for debug, by default None"
|
|
function: collector.FundCollector
|
|
- docstring: null
|
|
function: collector.init_datetime
|
|
- docstring: null
|
|
function: collector.convert_datetime
|
|
- docstring: null
|
|
function: collector._timezone
|
|
- docstring: null
|
|
function: collector.get_data_from_remote
|
|
- docstring: null
|
|
function: collector.get_data
|
|
- docstring: null
|
|
function: collector._get_simple
|
|
- docstring: null
|
|
function: collector.FundollectorCN
|
|
- docstring: null
|
|
function: collector.get_instrument_list
|
|
- docstring: null
|
|
function: collector.normalize_symbol
|
|
- docstring: null
|
|
function: collector._timezone
|
|
- docstring: null
|
|
function: collector.FundCollectorCN1d
|
|
- docstring: null
|
|
function: collector.FundNormalize
|
|
- docstring: null
|
|
function: collector.normalize_fund
|
|
- docstring: null
|
|
function: collector.normalize
|
|
- docstring: null
|
|
function: collector.FundNormalize1d
|
|
- docstring: null
|
|
function: 'collector.FundNormalizeCN:'
|
|
- docstring: null
|
|
function: collector._get_calendar_list
|
|
- docstring: null
|
|
function: collector.FundNormalizeCN1d
|
|
- docstring: " \nParameters----------source_dir: strThe directory where the\
|
|
\ raw data collected from the Internet is saved, default \"Path(__file__).parent/source\"\
|
|
normalize_dir: strDirectory for normalize data, default \"Path(__file__).parent/normalize\"\
|
|
max_workers: intConcurrent number, default is 4interval: strfreq, value from [1min,\
|
|
\ 1d], default 1dregion: strregion, value from [\"CN\"], default \"CN\""
|
|
function: collector.Run
|
|
- docstring: null
|
|
function: collector.collector_class_name
|
|
- docstring: null
|
|
function: collector.normalize_class_name
|
|
- docstring: null
|
|
function: collector.default_base_dir
|
|
- docstring: ' download data from Internet
|
|
|
|
Parameters----------max_collector_count: intdefault 2delay: floattime.sleep(delay),
|
|
default 0interval: strfreq, value from [1min, 1d], default 1dstart: strstart datetime,
|
|
default "2000-01-01"end: strend datetime, default ``pd.Timestamp(datetime.datetime.now()
|
|
+ pd.Timedelta(days=1))``check_data_length: int # if this param useful?check data
|
|
length, if not None and greater than 0, each symbol will be considered complete
|
|
if its data length is greater than or equal to this value, otherwise it will be
|
|
fetched again, the maximum number of fetches being (max_collector_count). By default
|
|
None.limit_nums: intusing for debug, by default NoneExamples---------# get daily
|
|
data$ python collector.py download_data --source_dir ~/.qlib/fund_data/source/cn_data
|
|
--region CN --start 2020-11-01 --end 2020-11-10 --delay 0.1 --interval 1d'
|
|
function: collector.download_data
|
|
- docstring: ' normalize data
|
|
|
|
Parameters----------date_field_name: strdate field name, default datesymbol_field_name:
|
|
strsymbol field name, default symbolExamples---------$ python collector.py normalize_data
|
|
--source_dir ~/.qlib/fund_data/source/cn_data --normalize_dir ~/.qlib/fund_data/source/cn_1d_nor
|
|
--region CN --interval 1d --date_field_name FSRQ'
|
|
function: collector.normalize_data
|
|
- docstring: null
|
|
function: 'config.Config:'
|
|
- docstring: null
|
|
function: config.get
|
|
- docstring: null
|
|
function: config.reset
|
|
- docstring: null
|
|
function: config.update
|
|
- docstring: null
|
|
function: config.set_conf_from_C
|
|
- docstring: null
|
|
function: config.register_from_C
|
|
- docstring: null
|
|
function: config.QlibConfig
|
|
- docstring: " \nMotivation:- get the right path (e.g. data uri) for accessing\
|
|
\ data based on given information(e.g. provider_uri, mount_path and frequency)-\
|
|
\ some helper functions to process uri."
|
|
function: 'config.DataPathManager:'
|
|
- docstring: null
|
|
function: config.format_provider_uri
|
|
- docstring: null
|
|
function: config.get_uri_type
|
|
- docstring: " \nplease refer DataPathManager's __init__ and class doc"
|
|
function: config.get_data_uri
|
|
- docstring: null
|
|
function: config.set_mode
|
|
- docstring: null
|
|
function: config.set_region
|
|
- docstring: null
|
|
function: config.is_depend_redis
|
|
- docstring: null
|
|
function: config.dpm
|
|
- docstring: null
|
|
function: config.resolve_path
|
|
- docstring: " \nconfigure qlib based on the input parametersThe configuration\
|
|
\ will act like a dictionary.Normally, it literally is replaced the value according\
|
|
\ to the keys.However, sometimes it is hard for users to set the config when the\
|
|
\ configuration is nested and complicatedSo this API provides some special parameters\
|
|
\ for users to set the keys in a more convenient way.- region: REG_CN, REG_US-\
|
|
\ several region-related config will be changedParameters----------default_conf\
|
|
\ : strthe default config template chosen by user: \"server\", \"client\""
|
|
function: config.set
|
|
- docstring: null
|
|
function: config.register
|
|
- docstring: null
|
|
function: config.reset_qlib_version
|
|
- docstring: ' get number of processors given frequency
|
|
|
|
if isinstance(self["kernels"], Callable):return self["kernels"](freq)return self["kernels"]@property'
|
|
function: config.get_kernels
|
|
- docstring: " \nInstDictConf is a Dict-based config to describe an instancecase\
|
|
\ 1){'class': 'ClassName','kwargs': dict, # It is optional. {} will be used if\
|
|
\ not given'model_path': path, # It is optional if module is given in the class}case\
|
|
\ 2){'class': <The class it self>,'kwargs': dict, # It is optional. {} will be\
|
|
\ used if not given}"
|
|
function: typehint.InstDictConf
|
|
- docstring: " \nParameters----------default_conf: strthe default value is client.\
|
|
\ Accepted values: client/server.**kwargs :clear_mem_cache: strthe default value\
|
|
\ is True;Will the memory cache be clear.It is often used to improve performance\
|
|
\ when init will be called for multiple timesskip_if_reg: bool: strthe default\
|
|
\ value is True;When using the recorder, skip_if_reg can set to True to avoid\
|
|
\ loss of recorder."
|
|
function: __init__.init
|
|
- docstring: null
|
|
function: __init__._mount_nfs_uri
|
|
- docstring: ' init_from_yaml_conf
|
|
|
|
:param conf_path: A path to the qlib config in yml format'
|
|
function: __init__.init_from_yaml_conf
|
|
- docstring: " \nIf users are building a project follow the following pattern.-\
|
|
\ Qlib is a sub folder in project path- There is a file named `config.yaml` in\
|
|
\ qlib.For example:If your project file system structure follows such a pattern<project_path>/-\
|
|
\ config.yaml- ...some folders...- qlib/This folder will return <project_path>NOTE:\
|
|
\ link is not supported here.This method is often used when- user want to use\
|
|
\ a relative config path instead of hard-coding qlib config path in codeRaises------FileNotFoundError:If\
|
|
\ project path is not found"
|
|
function: __init__.get_project_path
|
|
- docstring: " \nThis function will init qlib automatically with following priority-\
|
|
\ Find the project configuration and init qlib- The parsing process will be affected\
|
|
\ by the `conf_type` of the configuration file- Init qlib with default config-\
|
|
\ Skip initialization if already initialized:**kwargs: it may contain following\
|
|
\ parameterscur_path: the start path to find the project pathHere are two examples\
|
|
\ of the configurationExample 1)If you want to create a new project-specific config\
|
|
\ based on a shared configure, you can use `conf_type: ref`.. code-block:: yamlconf_type:\
|
|
\ refqlib_cfg: '<shared_yaml_config_path>' # this could be null reference no\
|
|
\ config from other files# following configs in `qlib_cfg_update` is project=specificqlib_cfg_update:exp_manager:class:\
|
|
\ \"MLflowExpManager\"module_path: \"qlib.workflow.expm\"kwargs:uri: \"file://<your\
|
|
\ mlflow experiment path>\"default_exp_name: \"Experiment\"Example 2)If you want\
|
|
\ to create simple a standalone config, you can use following config(a.k.a. `conf_type:\
|
|
\ origin`).. code-block:: pythonexp_manager:class: \"MLflowExpManager\"module_path:\
|
|
\ \"qlib.workflow.expm\"kwargs:uri: \"file://<your mlflow experiment path>\"default_exp_name:\
|
|
\ \"Experiment\""
|
|
function: __init__.auto_init
|
|
- docstring: null
|
|
function: log.MetaLogger
|
|
- docstring: " \nCustomized logger for Qlib."
|
|
function: log.QlibLogger
|
|
- docstring: null
|
|
function: log.logger
|
|
- docstring: null
|
|
function: log.setLevel
|
|
- docstring: null
|
|
function: 'log._QLibLoggerManager:'
|
|
- docstring: " \nGet a logger for a specific module.:param module_name: strLogic\
|
|
\ module name.:param level: int:return: LoggerLogger object."
|
|
function: log.setLevel
|
|
- docstring: null
|
|
function: 'log.TimeInspector:'
|
|
- docstring: " \nSet a time mark with current time, and this time mark will\
|
|
\ push into a stack.:return: floatA timestamp for current time."
|
|
function: log.set_time_mark
|
|
- docstring: " \nPop last time mark from stack."
|
|
function: log.pop_time_mark
|
|
- docstring: " \nGet last time mark from stack, calculate time diff with current\
|
|
\ time.:return: floatTime diff calculated by last time mark with current time."
|
|
function: log.get_cost_time
|
|
- docstring: " \nGet last time mark from stack, calculate time diff with current\
|
|
\ time, and log time diff and info.:param info: strInfo that will be logged into\
|
|
\ stdout."
|
|
function: log.log_cost_time
|
|
- docstring: ' logt.
|
|
|
|
Log the time of the inside codeParameters----------name :nameshow_start :show_start'
|
|
function: log.logt
|
|
- docstring: ' set log with config
|
|
|
|
:param log_config::return:'
|
|
function: log.set_log_with_config
|
|
- docstring: null
|
|
function: log.LogFilter
|
|
- docstring: null
|
|
function: log.match_msg
|
|
- docstring: null
|
|
function: log.filter
|
|
- docstring: ' set qlib.xxx logger handlers level
|
|
|
|
Parameters----------level: intlogger levelreturn_orig_handler_level: boolreturn
|
|
origin handler level mapExamples---------.. code-block:: pythonimport qlibimport
|
|
loggingfrom qlib.log import get_module_logger, set_global_logger_levelqlib.init()tmp_logger_01
|
|
= get_module_logger("tmp_logger_01", level=logging.INFO)tmp_logger_01.info("1.
|
|
tmp_logger_01 info show")global_level = logging.WARNING + 1set_global_logger_level(global_level)tmp_logger_02
|
|
= get_module_logger("tmp_logger_02", level=logging.INFO)tmp_logger_02.log(msg="2.
|
|
tmp_logger_02 log show", level=global_level)tmp_logger_01.info("3. tmp_logger_01
|
|
info do not show")'
|
|
function: log.set_global_logger_level
|
|
- docstring: ' set qlib.xxx logger handlers level to use contextmanager
|
|
|
|
Parameters----------level: intlogger levelExamples---------.. code-block:: pythonimport
|
|
qlibimport loggingfrom qlib.log import get_module_logger, set_global_logger_level_cmqlib.init()tmp_logger_01
|
|
= get_module_logger("tmp_logger_01", level=logging.INFO)tmp_logger_01.info("1.
|
|
tmp_logger_01 info show")global_level = logging.WARNING + 1with set_global_logger_level_cm(global_level):tmp_logger_02
|
|
= get_module_logger("tmp_logger_02", level=logging.INFO)tmp_logger_02.log(msg="2.
|
|
tmp_logger_02 log show", level=global_level)tmp_logger_01.info("3. tmp_logger_01
|
|
info do not show")tmp_logger_01.info("4. tmp_logger_01 info show")'
|
|
function: log.set_global_logger_level_cm
|
|
- docstring: " \nParameters----------delete_zip_file : bool, optionalWhether\
|
|
\ to delete the zip file, value from True or False, by default False"
|
|
function: 'data.GetData:'
|
|
- docstring: null
|
|
function: data.merge_remote_url
|
|
- docstring: " \nDownload the specified file to the target folder.Parameters----------target_dir:\
|
|
\ strdata save directoryfile_name: strdataset name, needs to endwith .zip, value\
|
|
\ from [rl_data.zip, csv_data_cn.zip, ...]may contain folder names, for example:\
|
|
\ v2/qlib_data_simple_cn_1d_latest.zipdelete_old: booldelete an existing directory,\
|
|
\ by default TrueExamples---------# get rl datapython get_data.py download_data\
|
|
\ --file_name rl_data.zip --target_dir ~/.qlib/qlib_data/rl_dataWhen this command\
|
|
\ is run, the data will be downloaded from this link: https://qlibpublic.blob.core.windows.net/data/default/stock_data/rl_data.zip?{token}#\
|
|
\ get cn csv datapython get_data.py download_data --file_name csv_data_cn.zip\
|
|
\ --target_dir ~/.qlib/csv_data/cn_dataWhen this command is run, the data will\
|
|
\ be downloaded from this link: https://qlibpublic.blob.core.windows.net/data/default/stock_data/csv_data_cn.zip?{token}-------"
|
|
function: data.download_data
|
|
- docstring: null
|
|
function: data.check_dataset
|
|
- docstring: null
|
|
function: data._unzip
|
|
- docstring: null
|
|
function: data._delete_qlib_data
|
|
- docstring: ' download cn qlib data from remote
|
|
|
|
Parameters----------target_dir: strdata save directoryname: strdataset name, value
|
|
from [qlib_data, qlib_data_simple], by default qlib_dataversion: strdata version,
|
|
value from [v1, ...], by default None(use script to specify version)interval:
|
|
strdata freq, value from [1d], by default 1dregion: strdata region, value from
|
|
[cn, us], by default cndelete_old: booldelete an existing directory, by default
|
|
Trueexists_skip: boolexists skip, by default FalseExamples---------# get 1d datapython
|
|
get_data.py qlib_data --name qlib_data --target_dir ~/.qlib/qlib_data/cn_data
|
|
--interval 1d --region cnWhen this command is run, the data will be downloaded
|
|
from this link: https://qlibpublic.blob.core.windows.net/data/default/stock_data/v2/qlib_data_cn_1d_latest.zip?{token}#
|
|
get 1min datapython get_data.py qlib_data --name qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min
|
|
--interval 1min --region cnWhen this command is run, the data will be downloaded
|
|
from this link: https://qlibpublic.blob.core.windows.net/data/default/stock_data/v2/qlib_data_cn_1min_latest.zip?{token}-------'
|
|
function: data.qlib_data
|
|
- docstring: null
|
|
function: config.get_data_handler_config
|
|
- docstring: null
|
|
function: config.get_dataset_config
|
|
- docstring: null
|
|
function: config.get_gbdt_task
|
|
- docstring: null
|
|
function: config.get_record_lgb_config
|
|
- docstring: null
|
|
function: __init__.TestAutoData
|
|
- docstring: null
|
|
function: __init__.setUpClass
|
|
- docstring: null
|
|
function: __init__.TestOperatorData
|
|
- docstring: '
|
|
|
|
MOCK_DF = pd.read_csv(io.StringIO(MOCK_DATA), header=0, dtype={"symbol": str})'
|
|
function: __init__.setUpClass
|
|
- docstring: null
|
|
function: '__init__.MockStorageBase:'
|
|
- docstring: null
|
|
function: __init__.MockCalendarStorage
|
|
- docstring: null
|
|
function: __init__.data
|
|
- docstring: null
|
|
function: __init__.MockInstrumentStorage
|
|
- docstring: null
|
|
function: __init__.data
|
|
- docstring: null
|
|
function: __init__.MockFeatureStorage
|
|
- docstring: null
|
|
function: __init__.data
|
|
- docstring: null
|
|
function: __init__.start_index
|
|
- docstring: null
|
|
function: __init__.end_index
|
|
- docstring: null
|
|
function: __init__.TestMockData
|
|
- docstring: ' Base strategy for trading
|
|
|
|
self,outer_trade_decision: BaseTradeDecision = None,level_infra: LevelInfrastructure
|
|
= None,common_infra: CommonInfrastructure = None,trade_exchange: Exchange = None,)
|
|
-> None:'
|
|
function: 'base.BaseStrategy:'
|
|
- docstring: null
|
|
function: base.executor
|
|
- docstring: null
|
|
function: base.trade_calendar
|
|
- docstring: null
|
|
function: base.trade_position
|
|
- docstring: ' get trade exchange in a prioritized order
|
|
|
|
return getattr(self, "_trade_exchange", None) or self.common_infra.get("trade_exchange")'
|
|
function: base.trade_exchange
|
|
- docstring: null
|
|
function: base.reset_level_infra
|
|
- docstring: null
|
|
function: base.reset_common_infra
|
|
- docstring: " \n- reset `level_infra`, used to reset trade calendar, .etc-\
|
|
\ reset `common_infra`, used to reset `trade_account`, `trade_exchange`, .etc-\
|
|
\ reset `outer_trade_decision`, used to make split decision**NOTE**:split this\
|
|
\ function into `reset` and `_reset` will make following cases more convenient1.\
|
|
\ Users want to initialize his strategy by overriding `reset`, but they don't\
|
|
\ want to affect the `_reset`called when initialization"
|
|
function: base.reset
|
|
- docstring: " \nPlease refer to the docs of `reset`"
|
|
function: base._reset
|
|
- docstring: ' Generate trade decision in each trading bar
|
|
|
|
Parameters----------execute_result : List[object], optionalthe executed result
|
|
for trade decision, by default None- When call the generate_trade_decision firstly,
|
|
`execute_result` could be None'
|
|
function: base.generate_trade_decision
|
|
- docstring: " \nreturn data calendar's available decision range for `self`\
|
|
\ strategythe range consider following factors- data calendar in the charge of\
|
|
\ `self` strategy- trading range limitation from the decision of outer strategyrelated\
|
|
\ methods- TradeCalendarManager.get_data_cal_range- BaseTradeDecision.get_data_cal_range_limitParameters----------rtype:\
|
|
\ str- \"full\": return the available data index range of the strategy from `start_time`\
|
|
\ to `end_time`- \"step\": return the available data index range of the strategy\
|
|
\ of current stepReturns-------Tuple[int, int]:the available range both sides\
|
|
\ are closed"
|
|
function: base.get_data_cal_avail_range
|
|
- docstring: " \nupdate trade decision in each step of inner execution, this\
|
|
\ method enable all orderParameters----------trade_decision : BaseTradeDecisionthe\
|
|
\ trade decision that will be updatedtrade_calendar : TradeCalendarManagerThe\
|
|
\ calendar of the **inner strategy**!!!!!Returns-------BaseTradeDecision:"
|
|
function: base.update_trade_decision
|
|
- docstring: " \nA method for updating the outer_trade_decision.The outer strategy\
|
|
\ may change its decision during updating.Parameters----------outer_trade_decision\
|
|
\ : BaseTradeDecisionthe decision updated by the outer strategyReturns-------BaseTradeDecision"
|
|
function: base.alter_outer_trade_decision
|
|
- docstring: " \nA hook for doing sth after the upper level executor finished\
|
|
\ its execution (for example, finalizethe metrics collection)."
|
|
function: base.post_upper_level_exe_step
|
|
- docstring: " \nA hook for doing sth after the corresponding executor finished\
|
|
\ its execution.Parameters----------execute_result :the execution result"
|
|
function: base.post_exe_step
|
|
- docstring: ' RL-based strategy
|
|
|
|
self,policy,outer_trade_decision: BaseTradeDecision = None,level_infra: LevelInfrastructure
|
|
= None,common_infra: CommonInfrastructure = None,**kwargs,) -> None:'
|
|
function: base.RLStrategy
|
|
- docstring: ' (RL)-based (Strategy) with (Int)erpreter
|
|
|
|
self,policy,state_interpreter: dict | StateInterpreter,action_interpreter: dict
|
|
| ActionInterpreter,outer_trade_decision: BaseTradeDecision = None,level_infra:
|
|
LevelInfrastructure = None,common_infra: CommonInfrastructure = None,**kwargs,)
|
|
-> None:'
|
|
function: base.RLIntStrategy
|
|
- docstring: null
|
|
function: 'objm.ObjManager:'
|
|
- docstring: " \nsave obj as nameParameters----------obj : objectobject to\
|
|
\ be savedname : strname of the object"
|
|
function: objm.save_obj
|
|
- docstring: " \nsave objectsParameters----------obj_name_l : list of <obj,\
|
|
\ name>"
|
|
function: objm.save_objs
|
|
- docstring: " \nload object by nameParameters----------name : strthe name\
|
|
\ of the objectReturns-------object:loaded object"
|
|
function: objm.load_obj
|
|
- docstring: " \nif the object named `name` existsParameters----------name\
|
|
\ : strname of the objecTReturns-------bool:If the object exists"
|
|
function: objm.exists
|
|
- docstring: " \nlist the objectsReturns-------list:the list of returned objects"
|
|
function: objm.list
|
|
- docstring: ' remove.
|
|
|
|
Parameters----------fname :if file name is provided. specific file is removedotherwise,
|
|
The all the objects will be removed.'
|
|
function: objm.remove
|
|
- docstring: " \nUse file system to manage objects"
|
|
function: objm.FileManager
|
|
- docstring: null
|
|
function: objm.create_path
|
|
- docstring: null
|
|
function: objm.save_obj
|
|
- docstring: null
|
|
function: objm.save_objs
|
|
- docstring: null
|
|
function: objm.load_obj
|
|
- docstring: null
|
|
function: objm.exists
|
|
- docstring: null
|
|
function: objm.list
|
|
- docstring: " \nResample the calendar with frequency freq_raw into the calendar\
|
|
\ with frequency freq_samAssumption:- Fix length (240) of the calendar in each\
|
|
\ day.Parameters----------calendar_raw : np.ndarrayThe calendar with frequency\
|
|
\ freq_rawfreq_raw : strFrequency of the raw calendarfreq_sam : strSample frequencyregion:\
|
|
\ strRegion, for example, \"cn\", \"us\"Returns-------np.ndarrayThe calendar with\
|
|
\ frequency freq_sam"
|
|
function: resam.resam_calendar
|
|
- docstring: ' get the feature with higher or equal frequency than `freq`.
|
|
|
|
Returns-------pd.DataFramethe feature with higher or equal frequency'
|
|
function: resam.get_higher_eq_freq_feature
|
|
- docstring: " \nResample value from time-series data- If `feature` has MultiIndex[instrument,\
|
|
\ datetime], apply the `method` to each instruemnt data with datetime in [start_time,\
|
|
\ end_time]Example:.. code-block::print(feature)$close $volumeinstrument\
|
|
\ datetimeSH600000 2010-01-04 86.778313 16162960.02010-01-05 87.433578\
|
|
\ 28117442.02010-01-06 85.713585 23632884.02010-01-07 83.788803 20813402.02010-01-08\
|
|
\ 84.730675 16044853.0SH600655 2010-01-04 2699.567383 158193.3281252010-01-08\
|
|
\ 2612.359619 77501.4062502010-01-11 2712.982422 160852.3906252010-01-12\
|
|
\ 2788.688232 164587.9375002010-01-13 2790.604004 145460.453125print(resam_ts_data(feature,\
|
|
\ start_time=\"2010-01-04\", end_time=\"2010-01-05\", fields=[\"$close\", \"$volume\"\
|
|
], method=\"last\"))$close $volumeinstrumentSH600000 87.433578 28117442.0SH600655\
|
|
\ 2699.567383 158193.328125- Else, the `feature` should have Index[datetime],\
|
|
\ just apply the `method` to `feature` directlyExample:.. code-block::print(feature)$close\
|
|
\ $volumedatetime2010-01-04 86.778313 16162960.02010-01-05 87.433578\
|
|
\ 28117442.02010-01-06 85.713585 23632884.02010-01-07 83.788803 20813402.02010-01-08\
|
|
\ 84.730675 16044853.0print(resam_ts_data(feature, start_time=\"2010-01-04\"\
|
|
, end_time=\"2010-01-05\", method=\"last\"))$close 87.433578$volume 28117442.0print(resam_ts_data(feature['$close'],\
|
|
\ start_time=\"2010-01-04\", end_time=\"2010-01-05\", method=\"last\"))87.433578Parameters----------ts_feature\
|
|
\ : Union[pd.DataFrame, pd.Series]Raw time-series feature to be resampledstart_time\
|
|
\ : Union[str, pd.Timestamp], optionalstart sampling time, by default Noneend_time\
|
|
\ : Union[str, pd.Timestamp], optionalend sampling time, by default Nonemethod\
|
|
\ : Union[str, Callable], optionalsample method, apply method function to each\
|
|
\ stock series data, by default \"last\"- If type(method) is str or callable function,\
|
|
\ it should be an attribute of SeriesGroupBy or DataFrameGroupby, and applies\
|
|
\ groupy.method for the sliced time-series data- If method is None, do nothing\
|
|
\ for the sliced time-series data.method_kwargs : dict, optionalarguments of method,\
|
|
\ by default {}Returns-------The resampled DataFrame/Series/value, return None\
|
|
\ when the resampled data is empty."
|
|
function: resam.resam_ts_data
|
|
- docstring: ' get the first/last not nan value of pd.Series with single level
|
|
index
|
|
|
|
Parameters----------series : pd.Seriesseries should not be emptylast : bool, optionalwhether
|
|
to get the last valid value, by default True- if last is True, get the last valid
|
|
value- else, get the first valid valueReturns-------Nan | floatthe first/last
|
|
valid value'
|
|
function: resam.get_valid_value
|
|
- docstring: ' Robust ZScore Normalization
|
|
|
|
Use robust statistics for Z-Score normalization:mean(x) = median(x)std(x) = MAD(x)
|
|
* 1.4826Reference:https://en.wikipedia.org/wiki/Median_absolute_deviation.'
|
|
function: data.robust_zscore
|
|
- docstring: null
|
|
function: data.zscore
|
|
- docstring: " \ndeepcopy an object without copy the complicated objects.This is\
|
|
\ useful when you want to generate Qlib tasks and share the handlerNOTE:- This\
|
|
\ function can't handle recursive objects!!!!!Parameters----------obj : objectthe\
|
|
\ object to be copiedReturns-------object:The copied object"
|
|
function: data.deepcopy_basic_type
|
|
- docstring: " \nsupporting adding base config based on the ext_config>>> bc =\
|
|
\ {\"a\": \"xixi\"}>>> ec = {\"b\": \"haha\"}>>> new_bc = update_config(bc, ec)>>>\
|
|
\ print(new_bc){'a': 'xixi', 'b': 'haha'}>>> print(bc) # base config should not\
|
|
\ be changed{'a': 'xixi'}>>> print(update_config(bc, {\"b\": S_DROP})){'a': 'xixi'}>>>\
|
|
\ print(update_config(new_bc, {\"b\": S_DROP})){'a': 'xixi'}"
|
|
function: data.update_config
|
|
- docstring: ' Create or get a file or directory given the path and return_dir.
|
|
|
|
Parameters----------path: a string indicates the path or None indicates creating
|
|
a temporary path.return_dir: if True, create and return a directory; otherwise
|
|
c&r a file.'
|
|
function: file.get_or_create_path
|
|
- docstring: ' Save multiple parts file
|
|
|
|
Implementation process:1. get the absolute path to ''filename''2. create a ''filename''
|
|
directory3. user does something with file_path(''filename/'')4. remove ''filename''
|
|
directory5. make_archive ''filename'' directory, and rename ''archive file'' to
|
|
filename:param filename: result model path:param format: archive format: one of
|
|
"zip", "tar", "gztar", "bztar", or "xztar":return: real model pathUsage::>>> #
|
|
The following code will create an archive file(''~/tmp/test_file'') containing
|
|
''test_doc_i''(i is 0-10) files.>>> with save_multiple_parts_file(''~/tmp/test_file'')
|
|
as filename_dir:... for i in range(10):... temp_path = os.path.join(filename_dir,
|
|
''test_doc_{}''.format(str(i)))... with open(temp_path) as fp:... fp.write(str(i))...'
|
|
function: file.save_multiple_parts_file
|
|
- docstring: ' Unpack archive with archive buffer
|
|
|
|
After the call is finished, the archive file and directory will be deleted.Implementation
|
|
process:1. create ''tempfile'' in ''~/tmp/'' and directory2. ''buffer'' write
|
|
to ''tempfile''3. unpack archive file(''tempfile'')4. user does something with
|
|
file_path(''tempfile/'')5. remove ''tempfile'' and ''tempfile directory'':param
|
|
buffer: bytes:param format: archive format: one of "zip", "tar", "gztar", "bztar",
|
|
or "xztar":return: unpack archive directory pathUsage::>>> # The following code
|
|
is to print all the file names in ''test_unpack.tar.gz''>>> with open(''test_unpack.tar.gz'')
|
|
as fp:... buffer = fp.read()...>>> with unpack_archive_with_buffer(buffer)
|
|
as temp_dir:... for f_n in os.listdir(temp_dir):... print(f_n)...'
|
|
function: file.unpack_archive_with_buffer
|
|
- docstring: null
|
|
function: file.get_tmp_file_with_buffer
|
|
- docstring: " \nproviding a easy interface to get an IO objectParameters----------file\
|
|
\ : Union[IO, str, Path]a object representing the fileReturns-------IO:a IO-like\
|
|
\ objectRaises------NotImplementedError:"
|
|
function: file.get_io_object
|
|
- docstring: null
|
|
function: paral.ParallelExt
|
|
- docstring: ' datetime_groupby_apply
|
|
|
|
This function will apply the `apply_func` on the datetime level index.Parameters----------df
|
|
:DataFrame for processingapply_func : Union[Callable, Text]apply_func for processing
|
|
the dataif a string is given, then it is treated as naive pandas functionaxis
|
|
:which axis is the datetime level locatedlevel :which level is the datetime levelresample_rule
|
|
:How to resample the data to calculating paralleln_jobs :n_jobs for joblibReturns:pd.DataFrame'
|
|
function: paral.datetime_groupby_apply
|
|
- docstring: null
|
|
function: paral._naive_group_apply
|
|
- docstring: " \nThis AsyncCaller tries to make it easier to async callCurrently,\
|
|
\ it is used in MLflowRecorder to make functions like `log_params` asyncNOTE:-\
|
|
\ This caller didn't consider the return value"
|
|
function: 'paral.AsyncCaller:'
|
|
- docstring: null
|
|
function: paral.close
|
|
- docstring: null
|
|
function: paral.run
|
|
- docstring: null
|
|
function: paral.wait
|
|
- docstring: null
|
|
function: paral.async_dec
|
|
- docstring: null
|
|
function: paral.decorator_func
|
|
- docstring: null
|
|
function: paral.wrapper
|
|
- docstring: null
|
|
function: 'paral.DelayedTask:'
|
|
- docstring: ' get_delayed_tuple.
|
|
|
|
Return the delayed_tuple created by joblib.delayed'
|
|
function: paral.get_delayed_tuple
|
|
- docstring: ' set_res.
|
|
|
|
Parameters----------res :the executed result of the delayed tuple'
|
|
function: paral.set_res
|
|
- docstring: ' return the object to replace the delayed task
|
|
|
|
raise NotImplementedError("NotImplemented")'
|
|
function: paral.get_replacement
|
|
- docstring: null
|
|
function: paral.DelayedTuple
|
|
- docstring: null
|
|
function: paral.get_delayed_tuple
|
|
- docstring: null
|
|
function: paral.get_replacement
|
|
- docstring: ' DelayedDict.
|
|
|
|
It is designed for following feature:Converting following existing code to parallel-
|
|
constructing a dict- key can be gotten instantly- computation of values tasks
|
|
a lot of time.- AND ALL the values are calculated in a SINGLE function'
|
|
function: paral.DelayedDict
|
|
- docstring: null
|
|
function: paral.get_delayed_tuple
|
|
- docstring: null
|
|
function: paral.get_replacement
|
|
- docstring: ' is_delayed_tuple.
|
|
|
|
Parameters----------obj : objectReturns-------boolis `obj` joblib.delayed tuple'
|
|
function: paral.is_delayed_tuple
|
|
- docstring: ' _replace_and_get_dt.
|
|
|
|
FIXME: this function may cause infinite loop when the complex data-structure contains
|
|
loop-referenceParameters----------complex_iter :complex_iter'
|
|
function: paral._replace_and_get_dt
|
|
- docstring: ' _recover_dt.
|
|
|
|
replace all the DelayedTask in the `complex_iter` with its `.res` valueFIXME:
|
|
this function may cause infinite loop when the complex data-structure contains
|
|
loop-referenceParameters----------complex_iter :complex_iter'
|
|
function: paral._recover_dt
|
|
- docstring: ' complex_parallel.
|
|
|
|
Find all the delayed function created by delayed in complex_iter, run them parallelly
|
|
and then replace it with the result>>> from qlib.utils.paral import complex_parallel>>>
|
|
from joblib import Parallel, delayed>>> complex_iter = {"a": delayed(sum)([1,2,3]),
|
|
"b": [1, 2, delayed(sum)([10, 1])]}>>> complex_parallel(Parallel(), complex_iter){''a'':
|
|
6, ''b'': [1, 2, 11]}Parameters----------paral : Parallelparalcomplex_iter :NOTE:
|
|
only list, tuple and dict will be explored!!!!Returns-------complex_iter whose
|
|
delayed joblib tasks are replaced with its execution results.'
|
|
function: paral.complex_parallel
|
|
- docstring: " \nWhen we repeatedly run functions, it is hard to avoid memory leakage.So\
|
|
\ we run it in the subprocess to ensure it is OK.NOTE: Because local object can't\
|
|
\ be pickled. So we can't implement it via closure.We have to implement it via\
|
|
\ callable Class"
|
|
function: 'paral.call_in_subproc:'
|
|
- docstring: null
|
|
function: exceptions.QlibException
|
|
- docstring: ' Error type for re-initialization when starting an experiment
|
|
|
|
'
|
|
function: exceptions.RecorderInitializationError
|
|
- docstring: ' Error type for Recorder when can not load object
|
|
|
|
'
|
|
function: exceptions.LoadObjectError
|
|
- docstring: ' Load module path
|
|
|
|
:param module_path::return::raises: ModuleNotFoundError'
|
|
function: mod.get_module_by_module_path
|
|
- docstring: " \nParameters----------module_path : stre.g. \"a.b.c.ClassName\"\
|
|
Returns-------Tuple[str, str]e.g. (\"a.b.c\", \"ClassName\")"
|
|
function: mod.split_module_path
|
|
- docstring: " \nextract class/func and kwargs from config infoParameters----------config\
|
|
\ : [dict, str]similar to configplease refer to the doc of init_instance_by_configdefault_module\
|
|
\ : Python module or strIt should be a python module to load the class typeThis\
|
|
\ function will load class from the config['module_path'] first.If config['module_path']\
|
|
\ doesn't exists, it will load the class from default_module.Returns-------(type,\
|
|
\ dict):the class/func object and it's arguments.Raises------ModuleNotFoundError"
|
|
function: mod.get_callable_kwargs
|
|
- docstring: " \nget initialized instance with configParameters----------config\
|
|
\ : InstConfdefault_module : Python moduleOptional. It should be a python module.NOTE:\
|
|
\ the \"module_path\" will be override by `module` argumentsThis function will\
|
|
\ load class from the config['module_path'] first.If config['module_path'] doesn't\
|
|
\ exists, it will load the class from default_module.accept_types: Union[type,\
|
|
\ Tuple[type]]Optional. If the config is a instance of specific type, return the\
|
|
\ config directly.This will be passed into the second parameter of isinstance.try_kwargs:\
|
|
\ DictTry to pass in kwargs in `try_kwargs` when initialized the instanceIf error\
|
|
\ occurred, it will fail back to initialization without try_kwargs.Returns-------object:An\
|
|
\ initialized object based on the config info"
|
|
function: mod.init_instance_by_config
|
|
- docstring: " \nPython doesn't provide the downcasting mechanism.We use the trick\
|
|
\ here to downcast the classParameters----------obj : objectthe object to be castcls\
|
|
\ : typethe target class type"
|
|
function: mod.class_casting
|
|
- docstring: " \nFind all the classes recursively that inherit from `cls` in a\
|
|
\ given module.- `cls` itself is also included>>> from qlib.data.dataset.handler\
|
|
\ import DataHandler>>> find_all_classes(\"qlib.contrib.data.handler\", DataHandler)[<class\
|
|
\ 'qlib.contrib.data.handler.Alpha158'>, <class 'qlib.contrib.data.handler.Alpha158vwap'>,\
|
|
\ <class 'qlib.contrib.data.handler.Alpha360'>, <class 'qlib.contrib.data.handler.Alpha360vwap'>,\
|
|
\ <class 'qlib.data.dataset.handler.DataHandlerLP'>]>>> from qlib.contrib.rolling.base\
|
|
\ import Rolling>>> find_all_classes(\"qlib.contrib.rolling\", Rolling)[<class\
|
|
\ 'qlib.contrib.rolling.base.Rolling'>, <class 'qlib.contrib.rolling.ddgda.DDGDA'>]TODO:-\
|
|
\ skip import error"
|
|
function: mod.find_all_classes
|
|
- docstring: " \nSerializable will change the behaviors of pickle.The rule to tell\
|
|
\ if a attribute will be kept or dropped when dumping.The rule with higher priorities\
|
|
\ is on the top- in the config attribute list -> always dropped- in the include\
|
|
\ attribute list -> always kept- in the exclude attribute list -> always dropped-\
|
|
\ name not starts with `_` -> kept- name starts with `_` -> kept if `dump_all`\
|
|
\ is true else droppedIt provides a syntactic sugar for distinguish the attributes\
|
|
\ which user doesn't want.- For examples, a learnable Datahandler just wants to\
|
|
\ save the parameters without data when dumping to disk"
|
|
function: 'serial.Serializable:'
|
|
- docstring: null
|
|
function: serial._is_kept
|
|
- docstring: " \nwill the object dump all object"
|
|
function: serial.dump_all
|
|
- docstring: " \nWhat attribute will not be in specific listParameters----------attr_type\
|
|
\ : str\"include\" or \"exclude\"Returns-------list:"
|
|
function: serial._get_attr_list
|
|
- docstring: " \nconfigure the serializable objectParameters----------kwargs\
|
|
\ may include following keysdump_all : boolwill the object dump all objectexclude\
|
|
\ : listWhat attribute will not be dumpedinclude : listWhat attribute will be\
|
|
\ dumpedrecursive : boolwill the configuration be recursive"
|
|
function: serial.config
|
|
- docstring: " \nDump self to a pickle file.path (Union[Path, str]): the path\
|
|
\ to dumpkwargs may include following keysdump_all : boolwill the object dump\
|
|
\ all objectexclude : listWhat attribute will not be dumpedinclude : listWhat\
|
|
\ attribute will be dumped"
|
|
function: serial.to_pickle
|
|
- docstring: " \nLoad the serializable class from a filepath.Args:filepath\
|
|
\ (str): the path of fileRaises:TypeError: the pickled file must be `type(cls)`Returns:`type(cls)`:\
|
|
\ the instance of `type(cls)`"
|
|
function: serial.load
|
|
- docstring: " \nReturn the real backend of a Serializable class. The pickle_backend\
|
|
\ value can be \"pickle\" or \"dill\".Returns:module: pickle or dill module based\
|
|
\ on pickle_backend"
|
|
function: serial.get_backend
|
|
- docstring: " \nA general dumping method for objectParameters----------obj\
|
|
\ : objectthe object to be dumpedpath : Union[Path, str]the target path the data\
|
|
\ will be dumped"
|
|
function: serial.general_dump
|
|
- docstring: ' concat all SingleData by index.
|
|
|
|
TODO: now just for SingleData.Parameters----------data_list : List[SingleData]the
|
|
list of all SingleData to concat.Returns-------MultiDatathe MultiData with ndim
|
|
== 2'
|
|
function: index_data.concat
|
|
- docstring: ' concat all SingleData by new index.
|
|
|
|
Parameters----------data_list : List[SingleData]the list of all SingleData to
|
|
sum.new_index : listthe new_index of new SingleData.fill_value : floatfill the
|
|
missing values or replace np.NaN.Returns-------SingleDatathe SingleData with new_index
|
|
and values after sum.'
|
|
function: index_data.sum_by_index
|
|
- docstring: " \nThis is for indexing(rows or columns)Read-only operations has\
|
|
\ higher priorities than others.So this class is designed in a **read-only** way\
|
|
\ to shared data for queries.Modifications will results in new Index.NOTE: the\
|
|
\ indexing has following flaws- duplicated index value is not well supported (only\
|
|
\ the first appearance will be considered)- The order of the index is not considered!!!!\
|
|
\ So the slicing will not behave like pandas when indexings are ordered"
|
|
function: 'index_data.Index:'
|
|
- docstring: " \nAfter user creates indices with Type A, user may query data\
|
|
\ with other types with the same info.This method try to make type conversion\
|
|
\ and make query sane rather than raising KeyError strictlyParameters----------item\
|
|
\ :The item to query index"
|
|
function: index_data._convert_type
|
|
- docstring: " \nGiven the index value, get the integer indexParameters----------item\
|
|
\ :The item to queryReturns-------int:The index of the itemRaises------KeyError:If\
|
|
\ the query item does not exist"
|
|
function: index_data.index
|
|
- docstring: null
|
|
function: index_data.is_sorted
|
|
- docstring: " \nsort the indexReturns-------Tuple[\"Index\", np.ndarray]:the\
|
|
\ sorted Index and the changed index"
|
|
function: index_data.sort
|
|
- docstring: ' return the index with the format of list.
|
|
|
|
return self.idx_list.tolist()'
|
|
function: index_data.tolist
|
|
- docstring: " \n`Indexer` will behave like the `LocIndexer` in PandasRead-only\
|
|
\ operations has higher priorities than others.So this class is designed in a\
|
|
\ read-only way to shared data for queries.Modifications will results in new Index."
|
|
function: 'index_data.LocIndexer:'
|
|
- docstring: ' process the indices from user and output a list of `Index`
|
|
|
|
res = []for i, idx in enumerate(indices):res.append(Index(data_shape[i] if len(idx)
|
|
== 0 else idx))return res'
|
|
function: index_data.proc_idx_l
|
|
- docstring: " \nconvert value-based indexing to integer-based indexing.Parameters----------index\
|
|
\ : Indexindex data.indexing : slicevalue based indexing data with slice type\
|
|
\ for indexing.Returns-------slice:the integer based slicing"
|
|
function: index_data._slc_convert
|
|
- docstring: null
|
|
function: 'index_data.BinaryOps:'
|
|
- docstring: " \nmeta class for auto generating operations for index data."
|
|
function: index_data.index_data_ops_creator
|
|
- docstring: " \nBase data structure of SingleData and MultiData.NOTE:- For performance\
|
|
\ issue, only **np.floating** is supported in the underlayer data !!!- Boolean\
|
|
\ based on np.floating is also supported. Here are some examples.. code-block::\
|
|
\ pythonnp.array([ np.nan]).any() -> Truenp.array([ np.nan]).all() -> Truenp.array([1.\
|
|
\ , 0.]).any() -> Truenp.array([1. , 0.]).all() -> False"
|
|
function: index_data.IndexData
|
|
- docstring: null
|
|
function: index_data.loc
|
|
- docstring: null
|
|
function: index_data.iloc
|
|
- docstring: null
|
|
function: index_data.index
|
|
- docstring: null
|
|
function: index_data.columns
|
|
- docstring: " \nAlign all indices of `other` to `self` before performing the\
|
|
\ arithmetic operations.This function will return a new IndexData rather than\
|
|
\ changing data in `other` inplaceParameters----------other : \"IndexData\"the\
|
|
\ index in `other` is to be changedReturns-------IndexData:the data in `other`\
|
|
\ with index aligned to `self`"
|
|
function: index_data._align_indices
|
|
- docstring: null
|
|
function: index_data.sort_index
|
|
- docstring: ' get the abs of data except np.NaN.
|
|
|
|
tmp_data = np.absolute(self.data)return self.__class__(tmp_data, *self.indices)'
|
|
function: index_data.abs
|
|
- docstring: null
|
|
function: index_data.replace
|
|
- docstring: ' apply a function to data.
|
|
|
|
tmp_data = func(self.data)return self.__class__(tmp_data, *self.indices)the length
|
|
of the data.Returns-------intthe length of the data.'
|
|
function: index_data.apply
|
|
- docstring: null
|
|
function: index_data.sum
|
|
- docstring: null
|
|
function: index_data.mean
|
|
- docstring: null
|
|
function: index_data.isna
|
|
- docstring: null
|
|
function: index_data.fillna
|
|
- docstring: null
|
|
function: index_data.count
|
|
- docstring: null
|
|
function: index_data.all
|
|
- docstring: null
|
|
function: index_data.empty
|
|
- docstring: null
|
|
function: index_data.values
|
|
- docstring: ' A data structure of index and numpy data.
|
|
|
|
It''s used to replace pd.Series due to high-speed.Parameters----------data : Union[int,
|
|
float, np.number, list, dict, pd.Series]the input dataindex : Union[list, pd.Index]the
|
|
index of data.empty list indicates that auto filling the index to the length of
|
|
data'
|
|
function: index_data.SingleData
|
|
- docstring: null
|
|
function: index_data._align_indices
|
|
- docstring: ' reindex data and fill the missing value with np.NaN.
|
|
|
|
Parameters----------new_index : listnew indexfill_value:what value to fill if
|
|
index is missingReturns-------SingleDatareindex data'
|
|
function: index_data.reindex
|
|
- docstring: null
|
|
function: index_data.add
|
|
- docstring: ' convert SingleData to dict.
|
|
|
|
Returns-------dictdata with the dict format.'
|
|
function: index_data.to_dict
|
|
- docstring: null
|
|
function: index_data.to_series
|
|
- docstring: ' A data structure of index and numpy data.
|
|
|
|
It''s used to replace pd.DataFrame due to high-speed.Parameters----------data
|
|
: Union[list, np.ndarray]the dim of data must be 2.index : Union[List, pd.Index,
|
|
Index]the index of data.columns: Union[List, pd.Index, Index]the columns of data.'
|
|
function: index_data.MultiData
|
|
- docstring: " \nget the minute level calendar in day periodParameters----------shift\
|
|
\ : intthe shift direction would be like pandas shift.series.shift(1) will replace\
|
|
\ the value at `i`-th with the one at `i-1`-thregion: strRegion, for example,\
|
|
\ \"cn\", \"us\"Returns-------List[time]:"
|
|
function: time.get_min_cal
|
|
- docstring: ' Is there only one piece of data for stock market.
|
|
|
|
Parameters----------start_time : Union[pd.Timestamp, str]closed start time for
|
|
data.end_time : Union[pd.Timestamp, str]closed end time for data.freq :region:
|
|
strRegion, for example, "cn", "us"Returns-------boolTrue means one piece of data
|
|
to obtain.'
|
|
function: time.is_single_value
|
|
- docstring: null
|
|
function: 'time.Freq:'
|
|
- docstring: " \nParse freq into a unified formatParameters----------freq :\
|
|
\ strRaw freq, supported freq should match the re '^([0-9]*)(month|mon|week|w|day|d|minute|min)$'Returns-------freq:\
|
|
\ Tuple[int, str]Unified freq, including freq count and unified freq unit. The\
|
|
\ freq unit should be '[month|week|day|minute]'.Example:.. code-block::print(Freq.parse(\"\
|
|
day\"))(1, \"day\" )print(Freq.parse(\"2mon\"))(2, \"month\")print(Freq.parse(\"\
|
|
10w\"))(10, \"week\")"
|
|
function: time.parse
|
|
- docstring: " \nget pd.Timedeta objectParameters----------n : intfreq : strTypically,\
|
|
\ they are the return value of Freq.parseReturns-------pd.Timedelta:"
|
|
function: time.get_timedelta
|
|
- docstring: ' Calculate freq delta
|
|
|
|
Parameters----------left_frq: strright_freq: strReturns-------'
|
|
function: time.get_min_delta
|
|
- docstring: ' Get the closest freq to base_freq from freq_list
|
|
|
|
Parameters----------base_freqfreq_listReturns-------if the recent frequency is
|
|
foundFreqelse:None'
|
|
function: time.get_recent_freq
|
|
- docstring: null
|
|
function: time.time_to_day_index
|
|
- docstring: " \nget the min-bar index in a day for a time range (both left and\
|
|
\ right is closed) given a fixed frequencyParameters----------start : stre.g.\
|
|
\ \"9:30\"end : stre.g. \"14:30\"freq : str\"1min\"Returns-------Tuple[int, int]:The\
|
|
\ index of start and end in the calendar. Both left and right are **closed**"
|
|
function: time.get_day_min_idx_range
|
|
- docstring: null
|
|
function: time.concat_date_time
|
|
- docstring: " \nalign the minute-level data to a down sampled calendare.g. align\
|
|
\ 10:38 to 10:35 in 5 minute-level(10:30 in 10 minute-level)Parameters----------x\
|
|
\ : pd.Timestampdatetime to be alignedsam_minutes : intalign to `sam_minutes`\
|
|
\ minute-level calendarregion: strRegion, for example, \"cn\", \"us\"Returns-------pd.Timestamp:the\
|
|
\ datetime after aligned"
|
|
function: time.cal_sam_minute
|
|
- docstring: " \nchange the time by infinitely small quantity.Parameters----------date_time\
|
|
\ : pd.Timestampthe original timedirection : strthe direction the time are going\
|
|
\ to- \"backward\" for going to history- \"forward\" for going to the futureReturns-------pd.Timestamp:the\
|
|
\ shifted time"
|
|
function: time.epsilon_change
|
|
- docstring: ' get redis connection instance.
|
|
|
|
return redis.StrictRedis(host=C.redis_host, port=C.redis_port, db=C.redis_task_db,
|
|
password=C.redis_password)#################### Data ####################'
|
|
function: __init__.get_redis_connection
|
|
- docstring: null
|
|
function: __init__.read_bin
|
|
- docstring: " \nThis method will be used in PIT database.It return all the possible\
|
|
\ values between `first` and `end` (first and end is included)Parameters----------quarterly\
|
|
\ : boolwill it return quarterly index or yearly index.Returns-------List[int]the\
|
|
\ possible index between [first, last]"
|
|
function: __init__.get_period_list
|
|
- docstring: null
|
|
function: __init__.get_period_offset
|
|
- docstring: " \nAt `cur_date`(e.g. 20190102), read the information at `period`(e.g.\
|
|
\ 201803).Only the updating info before cur_date or at cur_date will be used.Parameters----------period:\
|
|
\ intdate period represented by interger, e.g. 201901 corresponds to the first\
|
|
\ quarter in 2019cur_date_int: intdate which represented by interger, e.g. 20190102last_period_index:\
|
|
\ intit is a optional parameter; it is designed to avoid repeatedly access the\
|
|
\ .index data of PIT database whensequentially observing the data (Because the\
|
|
\ latest index of a specific period of data certainly appear in after the one\
|
|
\ in last observation).Returns-------the query value and byte index the index\
|
|
\ value"
|
|
function: __init__.read_period_data
|
|
- docstring: " \nforward fill a 1D numpy arrayParameters----------arr : np.arrayInput\
|
|
\ numpy 1D array"
|
|
function: __init__.np_ffill
|
|
- docstring: ' multi fields list lower bound.
|
|
|
|
for single field list use `bisect.bisect_left` instead'
|
|
function: __init__.lower_bound
|
|
- docstring: ' multi fields list upper bound.
|
|
|
|
for single field list use `bisect.bisect_right` instead'
|
|
function: __init__.upper_bound
|
|
- docstring: null
|
|
function: __init__.requests_with_retry
|
|
- docstring: null
|
|
function: __init__.parse_config
|
|
- docstring: null
|
|
function: __init__.drop_nan_by_y_index
|
|
- docstring: null
|
|
function: __init__.hash_args
|
|
- docstring: null
|
|
function: __init__.parse_field
|
|
- docstring: ' Compare dict value
|
|
|
|
:param src_data::param dst_data::return:'
|
|
function: __init__.compare_dict_value
|
|
- docstring: null
|
|
function: __init__.DateEncoder
|
|
- docstring: null
|
|
function: __init__.default
|
|
- docstring: ' remove repeat field
|
|
|
|
:param fields: list; features fields:return: list'
|
|
function: __init__.remove_repeat_field
|
|
- docstring: ' remove fields space
|
|
|
|
:param fields: features fields:return: list or str'
|
|
function: __init__.remove_fields_space
|
|
- docstring: ' normalize cache fields
|
|
|
|
:param fields: features fields:return: list'
|
|
function: __init__.normalize_cache_fields
|
|
- docstring: ' normalize cache instruments
|
|
|
|
:return: list or dict'
|
|
function: __init__.normalize_cache_instruments
|
|
- docstring: ' judgy whether date is a tradable date
|
|
|
|
----------date : pandas.Timestampcurrent date'
|
|
function: __init__.is_tradable_date
|
|
- docstring: ' get trading date range by shift
|
|
|
|
Parameters----------trading_date: pd.Timestampleft_shift: intright_shift: intfuture:
|
|
bool'
|
|
function: __init__.get_date_range
|
|
- docstring: ' get trading date with shift bias will cur_date
|
|
|
|
e.g. : shift == 1, return next trading dateshift == -1, return previous trading
|
|
date----------trading_date : pandas.Timestampcurrent dateshift : intclip_shift:
|
|
boolalign : Optional[str]When align is None, this function will raise ValueError
|
|
if `trading_date` is not a trading datewhen align is "left"/"right", it will try
|
|
to align to left/right nearest trading date before shifting when `trading_date`
|
|
is not a trading date'
|
|
function: __init__.get_date_by_shift
|
|
- docstring: ' get next trading date
|
|
|
|
----------cur_date : pandas.Timestampcurrent date'
|
|
function: __init__.get_next_trading_date
|
|
- docstring: ' get previous trading date
|
|
|
|
----------date : pandas.Timestampcurrent date'
|
|
function: __init__.get_pre_trading_date
|
|
- docstring: ' handle the end date with various format
|
|
|
|
If end_date is -1, None, or end_date is greater than the maximum trading day,
|
|
the last trading date is returned.Otherwise, returns the end_date----------end_date:
|
|
strend trading datedate : pandas.Timestampcurrent date'
|
|
function: __init__.transform_end_date
|
|
- docstring: ' Get the date(YYYY-MM-DD) written in file name
|
|
|
|
Parameterfile_name : str:returndate : str''YYYY-MM-DD'''
|
|
function: __init__.get_date_in_file_name
|
|
- docstring: ' split the score file into two part
|
|
|
|
Parameter---------pred : pd.DataFrame (index:<instrument, datetime>)A score file
|
|
of stocksnumber: the number of dates for pred_leftsplit_date: the last date of
|
|
the pred_leftReturn-------pred_left : pd.DataFrame (index:<instrument, datetime>)The
|
|
first part of original score filepred_right : pd.DataFrame (index:<instrument,
|
|
datetime>)The second part of original score file'
|
|
function: __init__.split_pred
|
|
- docstring: " \nTime slicing in Qlib or Pandas is a frequently-used action.However,\
|
|
\ user often input all kinds of data format to represent time.This function will\
|
|
\ help user to convert these inputs into a uniform format which is friendly to\
|
|
\ time slicing.Parameters----------t : Union[None, str, pd.Timestamp]original\
|
|
\ timeReturns-------Union[None, pd.Timestamp]:"
|
|
function: __init__.time_to_slc_point
|
|
- docstring: null
|
|
function: __init__.can_use_cache
|
|
- docstring: null
|
|
function: __init__.exists_qlib_data
|
|
- docstring: null
|
|
function: __init__.check_qlib_data
|
|
- docstring: " \nmake the df index sorteddf.sort_index() will take a lot of time\
|
|
\ even when `df.is_lexsorted() == True`This function could avoid such caseParameters----------df\
|
|
\ : pd.DataFrameReturns-------pd.DataFrame:sorted dataframe"
|
|
function: __init__.lazy_sort_index
|
|
- docstring: " \nFlatten a nested dict.>>> flatten_dict({'a': 1, 'c': {'a': 2,\
|
|
\ 'b': {'x': 5, 'y' : 10}}, 'd': [1, 2, 3]})>>> {'a': 1, 'c.a': 2, 'c.b.x': 5,\
|
|
\ 'd': [1, 2, 3], 'c.b.y': 10}>>> flatten_dict({'a': 1, 'c': {'a': 2, 'b': {'x':\
|
|
\ 5, 'y' : 10}}, 'd': [1, 2, 3]}, sep=FLATTEN_TUPLE)>>> {'a': 1, ('c','a'): 2,\
|
|
\ ('c','b','x'): 5, 'd': [1, 2, 3], ('c','b','y'): 10}Args:d (dict): the dict\
|
|
\ waiting for flattingparent_key (str, optional): the parent key, will be a prefix\
|
|
\ in new key. Defaults to \"\".sep (str, optional): the separator for string connecting.\
|
|
\ FLATTEN_TUPLE for tuple connecting.Returns:dict: flatten dict"
|
|
function: __init__.flatten_dict
|
|
- docstring: " \nFollow the name_path to get values from configFor example:If we\
|
|
\ follow the example in in the Parameters section,Timestamp('2008-01-02 00:00:00')\
|
|
\ will be returnedParameters----------config : dicte.g.{'dataset': {'class': 'DatasetH','kwargs':\
|
|
\ {'handler': {'class': 'Alpha158','kwargs': {'end_time': '2020-08-01','fit_end_time':\
|
|
\ '<dataset.kwargs.segments.train.1>','fit_start_time': '<dataset.kwargs.segments.train.0>','instruments':\
|
|
\ 'csi100','start_time': '2008-01-01'},'module_path': 'qlib.contrib.data.handler'},'segments':\
|
|
\ {'test': (Timestamp('2017-01-03 00:00:00'),Timestamp('2019-04-08 00:00:00')),'train':\
|
|
\ (Timestamp('2008-01-02 00:00:00'),Timestamp('2014-12-31 00:00:00')),'valid':\
|
|
\ (Timestamp('2015-01-05 00:00:00'),Timestamp('2016-12-30 00:00:00'))}}}}name_path\
|
|
\ : stre.g.\"dataset.kwargs.segments.train.1\"Returns-------objectthe retrieved\
|
|
\ object"
|
|
function: __init__.get_item_from_obj
|
|
- docstring: " \nDetect placeholder in config and fill them with config_extend.The\
|
|
\ item of dict must be single item(int, str, etc), dict and list. Tuples are not\
|
|
\ supported.There are two type of variables:- user-defined variables :e.g. when\
|
|
\ config_extend is `{\"<MODEL>\": model, \"<DATASET>\": dataset}`, \"<MODEL>\"\
|
|
\ and \"<DATASET>\" in `config` will be replaced with `model` `dataset`- variables\
|
|
\ extracted from `config` :e.g. the variables like \"<dataset.kwargs.segments.train.0>\"\
|
|
\ will be replaced with the values from `config`Parameters----------config : dictthe\
|
|
\ parameter dict will be filledconfig_extend : dictthe value of all placeholdersReturns-------dictthe\
|
|
\ parameter dict"
|
|
function: __init__.fill_placeholder
|
|
- docstring: null
|
|
function: __init__.try_replace_placeholder
|
|
- docstring: " \nthis will work like a decoration functionThe decrated function\
|
|
\ will ignore and give warning when the parameter is not acceptableFor example,\
|
|
\ if you have a function `f` which may optionally consume the keywards `bar`.then\
|
|
\ you can call it by `auto_filter_kwargs(f)(bar=3)`, which will automatically\
|
|
\ filter out`bar` when f does not need barParameters----------func : CallableThe\
|
|
\ original functionReturns-------Callable:the new callable function"
|
|
function: __init__.auto_filter_kwargs
|
|
- docstring: null
|
|
function: __init__._func
|
|
- docstring: ' Wrapper class for anything that needs to set up during qlib.init
|
|
|
|
self._provider = None'
|
|
function: '__init__.Wrapper:'
|
|
- docstring: null
|
|
function: __init__.register
|
|
- docstring: ' register_wrapper
|
|
|
|
:param wrapper: A wrapper.:param cls_or_obj: A class or class name or object
|
|
instance.'
|
|
function: __init__.register_wrapper
|
|
- docstring: ' load dataset from multiple file formats
|
|
|
|
if isinstance(path_or_obj, pd.DataFrame):return path_or_objif not os.path.exists(path_or_obj):raise
|
|
ValueError(f"file {path_or_obj} doesn''t exist")_, extension = os.path.splitext(path_or_obj)if
|
|
extension == ".h5":return pd.read_hdf(path_or_obj)elif extension == ".pkl":return
|
|
pd.read_pickle(path_or_obj)elif extension == ".csv":return pd.read_csv(path_or_obj,
|
|
parse_dates=True, index_col=index_col)raise ValueError(f"unsupported file type
|
|
`{extension}`")'
|
|
function: __init__.load_dataset
|
|
- docstring: ' stock code to file name
|
|
|
|
Parameters----------code: str'
|
|
function: __init__.code_to_fname
|
|
- docstring: ' file name to stock code
|
|
|
|
Parameters----------fname: str'
|
|
function: __init__.fname_to_code
|
|
- docstring: ' Element-wise Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------Expressionfeature
|
|
operation output'
|
|
function: ops.ElemOperator
|
|
- docstring: null
|
|
function: ops.get_longest_back_rolling
|
|
- docstring: null
|
|
function: ops.get_extended_window_size
|
|
- docstring: ' Change Instrument Operator
|
|
|
|
In some case, one may want to change to another instrument when calculating, for
|
|
example, tocalculate beta of a stock with respect to a market index.This would
|
|
require changing the calculation of features from the stock (original instrument)
|
|
tothe index (reference instrument)Parameters----------instrument: new instrument
|
|
for which the downstream operations should be performed upon.i.e., SH000300 (CSI300
|
|
index), or ^GPSC (SP500 index).feature: the feature to be calculated for the new
|
|
instrument.Returns----------Expressionfeature operation output'
|
|
function: ops.ChangeInstrument
|
|
- docstring: null
|
|
function: ops.load
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Numpy Element-wise Operator
|
|
|
|
Parameters----------feature : Expressionfeature instancefunc : strnumpy feature
|
|
operation methodReturns----------Expressionfeature operation output'
|
|
function: ops.NpElemOperator
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Feature Absolute Value
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------Expressiona
|
|
feature instance with absolute output'
|
|
function: ops.Abs
|
|
- docstring: ' Feature Sign
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------Expressiona
|
|
feature instance with sign'
|
|
function: ops.Sign
|
|
- docstring: " \nTo avoid error raised by bool type input, we transform the\
|
|
\ data into float32."
|
|
function: ops._load_internal
|
|
- docstring: ' Feature Log
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------Expressiona
|
|
feature instance with log'
|
|
function: ops.Log
|
|
- docstring: ' Feature Mask
|
|
|
|
Parameters----------feature : Expressionfeature instanceinstrument : strinstrument
|
|
maskReturns----------Expressiona feature instance with masked instrument'
|
|
function: ops.Mask
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Not Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------Feature:feature
|
|
elementwise not output'
|
|
function: ops.Not
|
|
- docstring: ' Pair-wise operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instance or numeric valuefeature_right
|
|
: Expressionfeature instance or numeric valueReturns----------Feature:two features''
|
|
operation output'
|
|
function: ops.PairOperator
|
|
- docstring: null
|
|
function: ops.get_longest_back_rolling
|
|
- docstring: null
|
|
function: ops.get_extended_window_size
|
|
- docstring: ' Numpy Pair-wise operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instance or numeric valuefeature_right
|
|
: Expressionfeature instance or numeric valuefunc : stroperator functionReturns----------Feature:two
|
|
features'' operation output'
|
|
function: ops.NpPairOperator
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Power Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:The bases in feature_left raised to the exponents
|
|
in feature_right'
|
|
function: ops.Power
|
|
- docstring: ' Add Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:two features'' sum'
|
|
function: ops.Add
|
|
- docstring: ' Subtract Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:two features'' subtraction'
|
|
function: ops.Sub
|
|
- docstring: ' Multiply Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:two features'' product'
|
|
function: ops.Mul
|
|
- docstring: ' Division Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:two features'' division'
|
|
function: ops.Div
|
|
- docstring: ' Greater Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:greater elements taken from the input two features'
|
|
function: ops.Greater
|
|
- docstring: ' Less Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:smaller elements taken from the input two features'
|
|
function: ops.Less
|
|
- docstring: ' Greater Than Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:bool series indicate `left > right`'
|
|
function: ops.Gt
|
|
- docstring: ' Greater Equal Than Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:bool series indicate `left >= right`'
|
|
function: ops.Ge
|
|
- docstring: ' Less Than Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:bool series indicate `left < right`'
|
|
function: ops.Lt
|
|
- docstring: ' Less Equal Than Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:bool series indicate `left <= right`'
|
|
function: ops.Le
|
|
- docstring: ' Equal Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:bool series indicate `left == right`'
|
|
function: ops.Eq
|
|
- docstring: ' Not Equal Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:bool series indicate `left != right`'
|
|
function: ops.Ne
|
|
- docstring: ' And Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:two features'' row by row & output'
|
|
function: ops.And
|
|
- docstring: ' Or Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceReturns----------Feature:two features'' row by row | outputs'
|
|
function: ops.Or
|
|
- docstring: ' If Operator
|
|
|
|
Parameters----------condition : Expressionfeature instance with bool values as
|
|
conditionfeature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instance'
|
|
function: ops.If
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: null
|
|
function: ops.get_longest_back_rolling
|
|
- docstring: null
|
|
function: ops.get_extended_window_size
|
|
- docstring: ' Rolling Operator
|
|
|
|
The meaning of rolling and expanding is the same in pandas.When the window is
|
|
set to 0, the behaviour of the operator should follow `expanding`Otherwise, it
|
|
follows `rolling`Parameters----------feature : Expressionfeature instanceN : introlling
|
|
window sizefunc : strrolling methodReturns----------Expressionrolling outputs'
|
|
function: ops.Rolling
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: null
|
|
function: ops.get_longest_back_rolling
|
|
- docstring: null
|
|
function: ops.get_extended_window_size
|
|
- docstring: ' Feature Reference
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : intN = 0, retrieve
|
|
the first data; N > 0, retrieve data of N periods ago; N < 0, future dataReturns----------Expressiona
|
|
feature instance with target reference'
|
|
function: ops.Ref
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: null
|
|
function: ops.get_longest_back_rolling
|
|
- docstring: null
|
|
function: ops.get_extended_window_size
|
|
- docstring: ' Rolling Mean (MA)
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling average'
|
|
function: ops.Mean
|
|
- docstring: ' Rolling Sum
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling sum'
|
|
function: ops.Sum
|
|
- docstring: ' Rolling Std
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling std'
|
|
function: ops.Std
|
|
- docstring: ' Rolling Variance
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling variance'
|
|
function: ops.Var
|
|
- docstring: ' Rolling Skewness
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling skewness'
|
|
function: ops.Skew
|
|
- docstring: ' Rolling Kurtosis
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling kurtosis'
|
|
function: ops.Kurt
|
|
- docstring: ' Rolling Max
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling max'
|
|
function: ops.Max
|
|
- docstring: ' Rolling Max Index
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling max index'
|
|
function: ops.IdxMax
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Rolling Min
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling min'
|
|
function: ops.Min
|
|
- docstring: ' Rolling Min Index
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling min index'
|
|
function: ops.IdxMin
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Rolling Quantile
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling quantile'
|
|
function: ops.Quantile
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Rolling Median
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling median'
|
|
function: ops.Med
|
|
- docstring: ' Rolling Mean Absolute Deviation
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling mean absolute deviation'
|
|
function: ops.Mad
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: null
|
|
function: ops.mad
|
|
- docstring: ' Rolling Rank (Percentile)
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling rank'
|
|
function: ops.Rank
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: null
|
|
function: ops.rank
|
|
- docstring: ' Rolling Count
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with rolling count of number
|
|
of non-NaN elements'
|
|
function: ops.Count
|
|
- docstring: ' Rolling Delta
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with end minus start in rolling
|
|
window'
|
|
function: ops.Delta
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Rolling Slope
|
|
|
|
This operator calculate the slope between `idx` and `feature`.(e.g. [<feature_t1>,
|
|
<feature_t2>, <feature_t3>] and [1, 2, 3])Usage Example:- "Slope($close, %d)/$close"#
|
|
TODO:# Some users may want pair-wise rolling like `Slope(A, B, N)`Parameters----------feature
|
|
: Expressionfeature instanceN : introlling window sizeReturns----------Expressiona
|
|
feature instance with linear regression slope of given window'
|
|
function: ops.Slope
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Rolling R-value Square
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with linear regression r-value
|
|
square of given window'
|
|
function: ops.Rsquare
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Rolling Regression Residuals
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with regression residuals of
|
|
given window'
|
|
function: ops.Resi
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Rolling WMA
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : introlling window
|
|
sizeReturns----------Expressiona feature instance with weighted moving average
|
|
output'
|
|
function: ops.WMA
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: null
|
|
function: ops.weighted_mean
|
|
- docstring: ' Rolling Exponential Mean (EMA)
|
|
|
|
Parameters----------feature : Expressionfeature instanceN : int, floatrolling
|
|
window sizeReturns----------Expressiona feature instance with regression r-value
|
|
square of given window'
|
|
function: ops.EMA
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: null
|
|
function: ops.exp_weighted_mean
|
|
- docstring: ' Pair Rolling Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceN : introlling window sizeReturns----------Expressiona feature instance
|
|
with rolling output of two input features'
|
|
function: ops.PairRolling
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: null
|
|
function: ops.get_longest_back_rolling
|
|
- docstring: null
|
|
function: ops.get_extended_window_size
|
|
- docstring: ' Rolling Correlation
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceN : introlling window sizeReturns----------Expressiona feature instance
|
|
with rolling correlation of two input features'
|
|
function: ops.Corr
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Rolling Covariance
|
|
|
|
Parameters----------feature_left : Expressionfeature instancefeature_right : Expressionfeature
|
|
instanceN : introlling window sizeReturns----------Expressiona feature instance
|
|
with rolling max of two input features'
|
|
function: ops.Cov
|
|
- docstring: " \nResampling the data to target frequency.The resample function\
|
|
\ of pandas is used.- the timestamp will be at the start of the time span after\
|
|
\ resample.Parameters----------feature : ExpressionAn expression for calculating\
|
|
\ the featurefreq : strIt will be passed into the resample method for resampling\
|
|
\ basedn on given frequencyfunc : methodThe method to get the resampled valuesSome\
|
|
\ expression are high frequently used"
|
|
function: ops.TResample
|
|
- docstring: null
|
|
function: ops._load_internal
|
|
- docstring: ' Ops Wrapper
|
|
|
|
self._ops = {}'
|
|
function: 'ops.OpsWrapper:'
|
|
- docstring: null
|
|
function: ops.reset
|
|
- docstring: ' register operator
|
|
|
|
Parameters----------ops_list : List[Union[Type[ExpressionOps], dict]]- if type(ops_list)
|
|
is List[Type[ExpressionOps]], each element of ops_list represents the operator
|
|
class, which should be the subclass of `ExpressionOps`.- if type(ops_list) is
|
|
List[dict], each element of ops_list represents the config of operator, which
|
|
has the following format:.. code-block:: text{"class": class_name,"module_path":
|
|
path,}Note: `class` should be the class name of operator, `module_path` should
|
|
be a python module or path of file.'
|
|
function: ops.register
|
|
- docstring: ' A client class
|
|
|
|
Provide the connection tool functions for ClientProvider.'
|
|
function: 'client.Client:'
|
|
- docstring: ' Connect to server.
|
|
|
|
try:self.sio.connect("ws://" + self.server_host + ":" + str(self.server_port))except
|
|
socketio.exceptions.ConnectionError:self.logger.error("Cannot connect to server
|
|
- check your network or server status")'
|
|
function: client.connect_server
|
|
- docstring: ' Disconnect from server.
|
|
|
|
try:self.sio.eio.disconnect(True)except Exception as e:self.logger.error("Cannot
|
|
disconnect from server : %s" % e)'
|
|
function: client.disconnect
|
|
- docstring: ' Send a certain request to server.
|
|
|
|
Parameters----------request_type : strtype of proposed request, ''calendar''/''instrument''/''feature''.request_content
|
|
: dictrecords the information of the request.msg_proc_func : functhe function
|
|
to process the message when receiving response, should have arg `*args`.msg_queue:
|
|
QueueThe queue to pass the message after callback.'
|
|
function: client.send_request
|
|
- docstring: ' callback_wrapper
|
|
|
|
:param *args: args[0] is the response content'
|
|
function: client.request_callback
|
|
- docstring: " \nThis helper class tries to make the provider based on storage\
|
|
\ backend more convenientIt is not necessary to inherent this class if that provider\
|
|
\ don't rely on the backend storage"
|
|
function: 'data.ProviderBackendMixin:'
|
|
- docstring: null
|
|
function: data.get_default_backend
|
|
- docstring: null
|
|
function: data.backend_obj
|
|
- docstring: ' Calendar provider base class
|
|
|
|
Provide calendar data.'
|
|
function: data.CalendarProvider
|
|
- docstring: ' Get calendar of certain market in given time range.
|
|
|
|
Parameters----------start_time : strstart of the time range.end_time : strend
|
|
of the time range.freq : strtime frequency, available: year/quarter/month/week/day.future
|
|
: boolwhether including future trading day.Returns----------listcalendar list'
|
|
function: data.calendar
|
|
- docstring: ' Locate the start time index and end time index in a calendar
|
|
under certain frequency.
|
|
|
|
Parameters----------start_time : pd.Timestampstart of the time range.end_time
|
|
: pd.Timestampend of the time range.freq : strtime frequency, available: year/quarter/month/week/day.future
|
|
: boolwhether including future trading day.Returns-------pd.Timestampthe real
|
|
start time.pd.Timestampthe real end time.intthe index of start time.intthe index
|
|
of end time.'
|
|
function: data.locate_index
|
|
- docstring: ' Load calendar using memcache.
|
|
|
|
Parameters----------freq : strfrequency of read calendar file.future : boolwhether
|
|
including future trading day.Returns-------listlist of timestamps.dictdict composed
|
|
by timestamp as key and index as value for fast search.'
|
|
function: data._get_calendar
|
|
- docstring: ' Get the uri of calendar generation task.
|
|
|
|
return hash_args(start_time, end_time, freq, future)'
|
|
function: data._uri
|
|
- docstring: ' Load original calendar timestamp from file.
|
|
|
|
Parameters----------freq : strfrequency of read calendar file.future: boolReturns----------listlist
|
|
of timestamps'
|
|
function: data.load_calendar
|
|
- docstring: ' Instrument provider base class
|
|
|
|
Provide instrument data.'
|
|
function: data.InstrumentProvider
|
|
- docstring: ' Get the general config dictionary for a base market adding several
|
|
dynamic filters.
|
|
|
|
Parameters----------market : Union[List, str]str:market/industry/index shortname,
|
|
e.g. all/sse/szse/sse50/csi300/csi500.list:["ID1", "ID2"]. A list of stocksfilter_pipe
|
|
: listthe list of dynamic filters.Returns----------dict: if isinstance(market,
|
|
str)dict of stockpool config.{`market` => base market name, `filter_pipe` => list
|
|
of filters}example :.. code-block::{''market'': ''csi500'',''filter_pipe'': [{''filter_type'':
|
|
''ExpressionDFilter'',''rule_expression'': ''$open<40'',''filter_start_time'':
|
|
None,''filter_end_time'': None,''keep'': False},{''filter_type'': ''NameDFilter'',''name_rule_re'':
|
|
''SH[0-9]{4}55'',''filter_start_time'': None,''filter_end_time'': None}]}list:
|
|
if isinstance(market, list)just return the original list directly.NOTE: this will
|
|
make the instruments compatible with more cases. The user code will be simpler.'
|
|
function: data.instruments
|
|
- docstring: ' List the instruments based on a certain stockpool config.
|
|
|
|
Parameters----------instruments : dictstockpool config.start_time : strstart of
|
|
the time range.end_time : strend of the time range.as_list : boolreturn instruments
|
|
as list or dict.Returns-------dict or listinstruments list or dictionary with
|
|
time spans'
|
|
function: data.list_instruments
|
|
- docstring: null
|
|
function: data._uri
|
|
- docstring: null
|
|
function: data.get_inst_type
|
|
- docstring: ' Feature provider class
|
|
|
|
Provide feature data.'
|
|
function: data.FeatureProvider
|
|
- docstring: ' Get feature data.
|
|
|
|
Parameters----------instrument : stra certain instrument.field : stra certain
|
|
field of feature.start_time : strstart of the time range.end_time : strend of
|
|
the time range.freq : strtime frequency, available: year/quarter/month/week/day.Returns-------pd.Seriesdata
|
|
of a certain feature'
|
|
function: data.feature
|
|
- docstring: null
|
|
function: data.PITProvider
|
|
- docstring: " \nget the historical periods data series between `start_index`\
|
|
\ and `end_index`Parameters----------start_index: intstart_index is a relative\
|
|
\ index to the latest period to cur_timeend_index: intend_index is a relative\
|
|
\ index to the latest period to cur_timein most cases, the start_index and end_index\
|
|
\ will be a non-positive valuesFor example, start_index == -3 end_index == 0 and\
|
|
\ current period index is cur_idx,then the data between [start_index + cur_idx,\
|
|
\ end_index + cur_idx] will be retrieved.period: intThis is used for query specific\
|
|
\ period.The period is represented with int in Qlib. (e.g. 202001 may represent\
|
|
\ the first quarter in 2020)NOTE: `period` will override `start_index` and `end_index`Returns-------pd.SeriesThe\
|
|
\ index will be integers to indicate the periods of the dataAn typical examples\
|
|
\ will beTODORaises------FileNotFoundErrorThis exception will be raised if the\
|
|
\ queried data do not exist."
|
|
function: data.period_feature
|
|
- docstring: ' Expression provider class
|
|
|
|
Provide Expression data.'
|
|
function: data.ExpressionProvider
|
|
- docstring: null
|
|
function: data.get_expression_instance
|
|
- docstring: ' Get Expression data.
|
|
|
|
The responsibility of `expression`- parse the `field` and `load` the according
|
|
data.- When loading the data, it should handle the time dependency of the data.
|
|
`get_expression_instance` is commonly used in this methodParameters----------instrument
|
|
: stra certain instrument.field : stra certain field of feature.start_time : strstart
|
|
of the time range.end_time : strend of the time range.freq : strtime frequency,
|
|
available: year/quarter/month/week/day.Returns-------pd.Seriesdata of a certain
|
|
expressionThe data has two types of format1) expression with datetime index2)
|
|
expression with integer index- because the datetime is not as good as'
|
|
function: data.expression
|
|
- docstring: ' Dataset provider class
|
|
|
|
Provide Dataset data.'
|
|
function: data.DatasetProvider
|
|
- docstring: ' Get dataset data.
|
|
|
|
Parameters----------instruments : list or dictlist/dict of instruments or dict
|
|
of stockpool config.fields : listlist of feature instances.start_time : strstart
|
|
of the time range.end_time : strend of the time range.freq : strtime frequency.inst_processors: Iterable[Union[dict,
|
|
InstProcessor]]the operations performed on each instrumentReturns----------pd.DataFramea
|
|
pandas dataframe with <instrument, datetime> index.'
|
|
function: data.dataset
|
|
- docstring: ' Get task uri, used when generating rabbitmq task in qlib_server
|
|
|
|
Parameters----------instruments : list or dictlist/dict of instruments or dict
|
|
of stockpool config.fields : listlist of feature instances.start_time : strstart
|
|
of the time range.end_time : strend of the time range.freq : strtime frequency.disk_cache
|
|
: intwhether to skip(0)/use(1)/replace(2) disk_cache.'
|
|
function: data._uri
|
|
- docstring: " \nParse different types of input instruments to output instruments_dWrong\
|
|
\ format of input instruments will lead to exception."
|
|
function: data.get_instruments_d
|
|
- docstring: " \nGet column names from input fields"
|
|
function: data.get_column_names
|
|
- docstring: null
|
|
function: data.parse_fields
|
|
- docstring: " \nLoad and process the data, return the data set.- default using\
|
|
\ multi-kernel method."
|
|
function: data.dataset_processor
|
|
- docstring: " \nCalculate the expressions for **one** instrument, return a\
|
|
\ df result.If the expression has been calculated before, load from cache.return\
|
|
\ value: A data frame with index 'datetime' and other data columns."
|
|
function: data.inst_calculator
|
|
- docstring: ' Local calendar data provider class
|
|
|
|
Provide calendar data from local data source.'
|
|
function: data.LocalCalendarProvider
|
|
- docstring: ' Load original calendar timestamp from file.
|
|
|
|
Parameters----------freq : strfrequency of read calendar file.future: boolReturns----------listlist
|
|
of timestamps'
|
|
function: data.load_calendar
|
|
- docstring: ' Local instrument data provider class
|
|
|
|
Provide instrument data from local data source.'
|
|
function: data.LocalInstrumentProvider
|
|
- docstring: null
|
|
function: data._load_instruments
|
|
- docstring: null
|
|
function: data.list_instruments
|
|
- docstring: ' Local feature data provider class
|
|
|
|
Provide feature data from local data source.'
|
|
function: data.LocalFeatureProvider
|
|
- docstring: null
|
|
function: data.feature
|
|
- docstring: null
|
|
function: data.LocalPITProvider
|
|
- docstring: null
|
|
function: data.period_feature
|
|
- docstring: ' Local expression data provider class
|
|
|
|
Provide expression data from local data source.'
|
|
function: data.LocalExpressionProvider
|
|
- docstring: null
|
|
function: data.expression
|
|
- docstring: ' Local dataset data provider class
|
|
|
|
Provide dataset data from local data source.'
|
|
function: data.LocalDatasetProvider
|
|
- docstring: null
|
|
function: data.dataset
|
|
- docstring: " \nThis method is used to prepare the expression cache for the\
|
|
\ client.Then the client will load the data from expression cache by itself."
|
|
function: data.multi_cache_walker
|
|
- docstring: " \nIf the expressions of one instrument haven't been calculated\
|
|
\ before,calculate it and write it into expression cache."
|
|
function: data.cache_walker
|
|
- docstring: ' Client calendar data provider class
|
|
|
|
Provide calendar data by requesting data from server as a client.'
|
|
function: data.ClientCalendarProvider
|
|
- docstring: null
|
|
function: data.set_conn
|
|
- docstring: null
|
|
function: data.calendar
|
|
- docstring: ' Client instrument data provider class
|
|
|
|
Provide instrument data by requesting data from server as a client.'
|
|
function: data.ClientInstrumentProvider
|
|
- docstring: null
|
|
function: data.set_conn
|
|
- docstring: null
|
|
function: data.list_instruments
|
|
- docstring: null
|
|
function: data.inst_msg_proc_func
|
|
- docstring: ' Client dataset data provider class
|
|
|
|
Provide dataset data by requesting data from server as a client.'
|
|
function: data.ClientDatasetProvider
|
|
- docstring: null
|
|
function: data.set_conn
|
|
- docstring: " \nCall the server to generate the expression cache.Then\
|
|
\ load the data from the expression cache directly.- default using multi-kernel\
|
|
\ method."
|
|
function: data.dataset
|
|
- docstring: ' Local provider class
|
|
|
|
It is a set of interface that allow users to access data.Because PITD is not exposed
|
|
publicly to users, so it is not included in the interface.To keep compatible with
|
|
old qlib provider.'
|
|
function: 'data.BaseProvider:'
|
|
- docstring: null
|
|
function: data.calendar
|
|
- docstring: null
|
|
function: data.instruments
|
|
- docstring: null
|
|
function: data.list_instruments
|
|
- docstring: " \nParameters----------disk_cache : intwhether to skip(0)/use(1)/replace(2)\
|
|
\ disk_cacheThis function will try to use cache method which has a keyword `disk_cache`,and\
|
|
\ will use provider method if a type error is raised because the DatasetD instanceis\
|
|
\ a provider class."
|
|
function: data.features
|
|
- docstring: null
|
|
function: data.LocalProvider
|
|
- docstring: ' _uri
|
|
|
|
The server hope to get the uri of the request. The uri will be decidedby the dataprovider.
|
|
For ex, different cache layer has different uri.:param type: The type of resource
|
|
for the uri:param **kwargs:'
|
|
function: data._uri
|
|
- docstring: ' features_uri
|
|
|
|
Return the uri of the generated cache of features/dataset:param disk_cache::param
|
|
instruments::param fields::param start_time::param end_time::param freq:'
|
|
function: data.features_uri
|
|
- docstring: ' Client Provider
|
|
|
|
Requesting data from server as a client. Can propose requests:- Calendar : Directly
|
|
respond a list of calendars- Instruments (without filter): Directly respond a
|
|
list/dict of instruments- Instruments (with filters): Respond a list/dict of
|
|
instruments- Features : Respond a cache uriThe general workflow is described as
|
|
follows:When the user use client provider to propose a request, the client provider
|
|
will connect the server and send the request. The client will start to wait for
|
|
the response. The response will be made instantly indicating whether the cache
|
|
is available. The waiting procedure will terminate only when the client get the
|
|
response saying `feature_available` is true.`BUG` : Everytime we make request
|
|
for certain data we need to connect to the server, wait for the response and disconnect
|
|
from it. We can''t make a sequence of requests within one connection. You can
|
|
refer to https://python-socketio.readthedocs.io/en/latest/client.html for documentation
|
|
of python-socketIO client.'
|
|
function: data.ClientProvider
|
|
- docstring: null
|
|
function: data.is_instance_of_provider
|
|
- docstring: " \nExpression base classExpression is designed to handle the calculation\
|
|
\ of data with the format belowdata with two dimension for each instrument,- feature-\
|
|
\ time: it could be observation time or period time.- period time is designed\
|
|
\ for Point-in-time database. For example, the period time maybe 2014Q4, its\
|
|
\ value can observed for multiple times(different value may be observed at different\
|
|
\ time due to amendment)."
|
|
function: base.Expression
|
|
- docstring: ' load feature
|
|
|
|
This function is responsible for loading feature/expression based on the expression
|
|
engine.The concrete implementation will be separated into two parts:1) caching
|
|
data, handle errors.- This part is shared by all the expressions and implemented
|
|
in Expression2) processing and calculating data based on the specific expression.-
|
|
This part is different in each expression and implemented in each expressionExpression
|
|
Engine is shared by different data.Different data will have different extra information
|
|
for `args`.Parameters----------instrument : strinstrument code.start_index : strfeature
|
|
start index [in calendar].end_index : strfeature end index [in calendar].*args
|
|
may contain following information:1) if it is used in basic expression engine
|
|
data, it contains following argumentsfreq: strfeature frequency.2) if is used
|
|
in PIT data, it contains following argumentscur_pit:it is designed for the point-in-time
|
|
data.period: intThis is used for query specific period.The period is represented
|
|
with int in Qlib. (e.g. 202001 may represent the first quarter in 2020)Returns----------pd.Seriesfeature
|
|
series: The index of the series is the calendar index'
|
|
function: base.load
|
|
- docstring: null
|
|
function: base._load_internal
|
|
- docstring: ' Get the longest length of historical data the feature has accessed
|
|
|
|
This is designed for getting the needed range of the data to calculatethe features
|
|
in specific range at first. However, situations likeRef(Ref($close, -1), 1) can
|
|
not be handled rightly.So this will only used for detecting the length of historical
|
|
data needed.'
|
|
function: base.get_longest_back_rolling
|
|
- docstring: ' get_extend_window_size
|
|
|
|
For to calculate this Operator in range[start_index, end_index]We have to get
|
|
the *leaf feature* inrange[start_index - lft_etd, end_index + rght_etd].Returns----------(int,
|
|
int)lft_etd, rght_etd'
|
|
function: base.get_extended_window_size
|
|
- docstring: ' Static Expression
|
|
|
|
This kind of feature will load data from provider'
|
|
function: base.Feature
|
|
- docstring: null
|
|
function: base._load_internal
|
|
- docstring: null
|
|
function: base.get_longest_back_rolling
|
|
- docstring: null
|
|
function: base.get_extended_window_size
|
|
- docstring: null
|
|
function: base.PFeature
|
|
- docstring: null
|
|
function: base._load_internal
|
|
- docstring: ' Operator Expression
|
|
|
|
This kind of feature will use operator for featureconstruction on the fly.'
|
|
function: base.ExpressionOps
|
|
- docstring: " \nprocess the dataNOTE: **The processor could change the content\
|
|
\ of `df` inplace !!!!! **User should keep a copy of data outsideParameters----------df\
|
|
\ : pd.DataFrameThe raw_df of handler or result from previous processor."
|
|
function: 'inst_processor.InstProcessor:'
|
|
- docstring: ' Dynamic Instruments Filter Abstract class
|
|
|
|
Users can override this class to construct their own filterOverride __init__ to
|
|
input filter regulationsOverride filter_main to use the regulations to filter
|
|
instruments'
|
|
function: filter.BaseDFilter
|
|
- docstring: ' Construct an instance from config dict.
|
|
|
|
Parameters----------config : dictdict of config parameters.'
|
|
function: filter.from_config
|
|
- docstring: ' Construct an instance from config dict.
|
|
|
|
Returns----------dictreturn the dict of config parameters.'
|
|
function: filter.to_config
|
|
- docstring: ' Dynamic Instruments Filter Abstract class to filter a series of
|
|
certain features
|
|
|
|
Filters should provide parameters:- filter start time- filter end time- filter
|
|
ruleOverride __init__ to assign a certain rule to filter the series.Override _getFilterSeries
|
|
to use the rule to filter the series and get a dict of {inst => series}, or override
|
|
filter_main for more advanced series filter rule'
|
|
function: filter.SeriesDFilter
|
|
- docstring: ' Get time bound for all instruments.
|
|
|
|
Parameters----------instruments: dictthe dict of instruments in the form {instrument_name
|
|
=> list of timestamp tuple}.Returns----------pd.Timestamp, pd.Timestampthe lower
|
|
time bound and upper time bound of all the instruments.'
|
|
function: filter._getTimeBound
|
|
- docstring: ' Convert the target timestamp to a pandas series of bool value
|
|
within a time range.
|
|
|
|
Make the time inside the target_timestamp range TRUE, others FALSE.Parameters----------time_range
|
|
: D.calendarthe time range of the instruments.target_timestamp : listthe list
|
|
of tuple (timestamp, timestamp).Returns----------pd.Seriesthe series of bool value
|
|
for an instrument.'
|
|
function: filter._toSeries
|
|
- docstring: ' Filter the timestamp series with filter series by using element-wise
|
|
AND operation of the two series.
|
|
|
|
Parameters----------timestamp_series : pd.Seriesthe series of bool value indicating
|
|
existing time.filter_series : pd.Seriesthe series of bool value indicating filter
|
|
feature.Returns----------pd.Seriesthe series of bool value indicating whether
|
|
the date satisfies the filter condition and exists in target timestamp.'
|
|
function: filter._filterSeries
|
|
- docstring: ' Convert the timestamp series to a list of tuple (timestamp,
|
|
timestamp) indicating a continuous range of TRUE.
|
|
|
|
Parameters----------timestamp_series: pd.Seriesthe series of bool value after
|
|
being filtered.Returns----------listthe list of tuple (timestamp, timestamp).'
|
|
function: filter._toTimestamp
|
|
- docstring: ' Get filter series based on the rules assigned during the initialization
|
|
and the input time range.
|
|
|
|
Parameters----------instruments : dictthe dict of instruments to be filtered.fstart
|
|
: pd.Timestampstart time of filter.fend : pd.Timestampend time of filter... note::
|
|
fstart/fend indicates the intersection of instruments start/end time and filter
|
|
start/end time.Returns----------pd.Dataframea series of {pd.Timestamp => bool}.'
|
|
function: filter._getFilterSeries
|
|
- docstring: ' Implement this method to filter the instruments.
|
|
|
|
Parameters----------instruments: dictinput instruments to be filtered.start_time:
|
|
strstart of the time range.end_time: strend of the time range.Returns----------dictfiltered
|
|
instruments, same structure as input instruments.'
|
|
function: filter.filter_main
|
|
- docstring: ' Name dynamic instrument filter
|
|
|
|
Filter the instruments based on a regulated name format.A name rule regular expression
|
|
is required.'
|
|
function: filter.NameDFilter
|
|
- docstring: null
|
|
function: filter._getFilterSeries
|
|
- docstring: null
|
|
function: filter.from_config
|
|
- docstring: null
|
|
function: filter.to_config
|
|
- docstring: ' Expression dynamic instrument filter
|
|
|
|
Filter the instruments based on a certain expression.An expression rule indicating
|
|
a certain feature field is required.Examples----------- *basic features filter*
|
|
: rule_expression = ''$close/$open>5''- *cross-sectional features filter* : rule_expression
|
|
= ''$rank($close)<10''- *time-sequence features filter* : rule_expression = ''$Ref($close,
|
|
3)>100'''
|
|
function: filter.ExpressionDFilter
|
|
- docstring: null
|
|
function: filter._getFilterSeries
|
|
- docstring: null
|
|
function: filter.from_config
|
|
- docstring: null
|
|
function: pit.P
|
|
- docstring: null
|
|
function: pit._load_internal
|
|
- docstring: null
|
|
function: pit._load_feature
|
|
- docstring: null
|
|
function: pit.get_longest_back_rolling
|
|
- docstring: null
|
|
function: pit.get_extended_window_size
|
|
- docstring: null
|
|
function: pit.PRef
|
|
- docstring: null
|
|
function: cache.QlibCacheException
|
|
- docstring: ' Memory Cache Unit.
|
|
|
|
self.size_limit = kwargs.pop("size_limit", 0)self._size = 0self.od = OrderedDict()#
|
|
TODO: thread safe?__setitem__ failure might cause inconsistent size?# precalculate
|
|
the size after od.__setitem__self._adjust_size(key, value)self.od.__setitem__(key,
|
|
value)# move the key to end,make it latestself.od.move_to_end(key)if self.limited:#
|
|
pop the oldest items beyond size limitwhile self._size > self.size_limit:self.popitem(last=False)v
|
|
= self.od.__getitem__(key)self.od.move_to_end(key)return vreturn key in self.odreturn
|
|
self.od.__len__()return f"{self.__class__.__name__}<size_limit:{self.size_limit
|
|
if self.limited else ''no limit''} total_size:{self._size}>\n{self.od.__repr__()}"'
|
|
function: cache.MemCacheUnit
|
|
- docstring: null
|
|
function: cache.set_limit_size
|
|
- docstring: ' whether memory cache is limited
|
|
|
|
return self.size_limit > 0@property'
|
|
function: cache.limited
|
|
- docstring: null
|
|
function: cache.total_size
|
|
- docstring: null
|
|
function: cache.clear
|
|
- docstring: null
|
|
function: cache.popitem
|
|
- docstring: null
|
|
function: cache.pop
|
|
- docstring: null
|
|
function: cache._adjust_size
|
|
- docstring: null
|
|
function: cache._get_value_size
|
|
- docstring: null
|
|
function: cache.MemCacheLengthUnit
|
|
- docstring: null
|
|
function: cache._get_value_size
|
|
- docstring: null
|
|
function: cache.MemCacheSizeofUnit
|
|
- docstring: null
|
|
function: cache._get_value_size
|
|
- docstring: ' Memory cache.
|
|
|
|
'
|
|
function: 'cache.MemCache:'
|
|
- docstring: null
|
|
function: cache.clear
|
|
- docstring: null
|
|
function: 'cache.MemCacheExpire:'
|
|
- docstring: ' set cache
|
|
|
|
:param mem_cache: MemCache attribute(''c''/''i''/''f'').:param key: cache key.:param
|
|
value: cache value.'
|
|
function: cache.set_cache
|
|
- docstring: ' get mem cache
|
|
|
|
:param mem_cache: MemCache attribute(''c''/''i''/''f'').:param key: cache key.:return:
|
|
cache value; if cache not exist, return None.'
|
|
function: cache.get_cache
|
|
- docstring: null
|
|
function: 'cache.CacheUtils:'
|
|
- docstring: null
|
|
function: cache.organize_meta_file
|
|
- docstring: null
|
|
function: cache.reset_lock
|
|
- docstring: null
|
|
function: cache.visit
|
|
- docstring: " \n) from lock_acquired@staticmethod@contextlib.contextmanager"
|
|
function: cache.acquire
|
|
- docstring: null
|
|
function: cache.reader_lock
|
|
- docstring: null
|
|
function: cache.writer_lock
|
|
- docstring: ' Provider cache base class
|
|
|
|
self.provider = providerself.logger = get_module_logger(self.__class__.__name__)return
|
|
getattr(self.provider, attr)@staticmethod'
|
|
function: 'cache.BaseProviderCache:'
|
|
- docstring: null
|
|
function: cache.check_cache_exists
|
|
- docstring: null
|
|
function: cache.clear_cache
|
|
- docstring: null
|
|
function: cache.get_cache_dir
|
|
- docstring: ' Expression cache mechanism base class.
|
|
|
|
This class is used to wrap expression provider with self-defined expression cache
|
|
mechanism... note:: Override the `_uri` and `_expression` method to create your
|
|
own expression cache mechanism.'
|
|
function: cache.ExpressionCache
|
|
- docstring: ' Get expression data.
|
|
|
|
.. note:: Same interface as `expression` method in expression provider'
|
|
function: cache.expression
|
|
- docstring: ' Get expression cache file uri.
|
|
|
|
Override this method to define how to get expression cache file uri corresponding
|
|
to users'' own cache mechanism.'
|
|
function: cache._uri
|
|
- docstring: ' Get expression data using cache.
|
|
|
|
Override this method to define how to get expression data corresponding to users''
|
|
own cache mechanism.'
|
|
function: cache._expression
|
|
- docstring: ' Update expression cache to latest calendar.
|
|
|
|
Override this method to define how to update expression cache corresponding to
|
|
users'' own cache mechanism.Parameters----------cache_uri : str or Paththe complete
|
|
uri of expression cache file (include dir path).freq : strReturns-------int0(successful
|
|
update)/ 1(no need to update)/ 2(update failure).'
|
|
function: cache.update
|
|
- docstring: ' Dataset cache mechanism base class.
|
|
|
|
This class is used to wrap dataset provider with self-defined dataset cache mechanism...
|
|
note:: Override the `_uri` and `_dataset` method to create your own dataset cache
|
|
mechanism.'
|
|
function: cache.DatasetCache
|
|
- docstring: ' Get feature dataset.
|
|
|
|
.. note:: Same interface as `dataset` method in dataset provider.. note:: The
|
|
server use redis_lock to make sureread-write conflicts will not be triggeredbut
|
|
client readers are not considered.'
|
|
function: cache.dataset
|
|
- docstring: ' Get dataset cache file uri.
|
|
|
|
Override this method to define how to get dataset cache file uri corresponding
|
|
to users'' own cache mechanism.'
|
|
function: cache._uri
|
|
- docstring: ' Get feature dataset using cache.
|
|
|
|
Override this method to define how to get feature dataset corresponding to users''
|
|
own cache mechanism.'
|
|
function: cache._dataset
|
|
- docstring: ' Get a uri of feature dataset using cache.
|
|
|
|
specially:disk_cache=1 means using data set cache and return the uri of cache
|
|
file.disk_cache=0 means client knows the path of expression cache,server checks
|
|
if the cache exists(if not, generate it), and client loads data by itself.Override
|
|
this method to define how to get feature dataset uri corresponding to users''
|
|
own cache mechanism.'
|
|
function: cache._dataset_uri
|
|
- docstring: ' Update dataset cache to latest calendar.
|
|
|
|
Override this method to define how to update dataset cache corresponding to users''
|
|
own cache mechanism.Parameters----------cache_uri : str or Paththe complete uri
|
|
of dataset cache file (include dir path).freq : strReturns-------int0(successful
|
|
update)/ 1(no need to update)/ 2(update failure)'
|
|
function: cache.update
|
|
- docstring: ' cache data to origin data
|
|
|
|
:param data: pd.DataFrame, cache data.:param fields: feature fields.:return: pd.DataFrame.'
|
|
function: cache.cache_to_origin_data
|
|
- docstring: ' normalize uri args
|
|
|
|
instruments = normalize_cache_instruments(instruments)fields = normalize_cache_fields(fields)freq
|
|
= freq.lower()return instruments, fields, freq'
|
|
function: cache.normalize_uri_args
|
|
- docstring: ' Prepared cache mechanism for server.
|
|
|
|
super(DiskExpressionCache, self).__init__(provider)self.r = get_redis_connection()#
|
|
remote==True means client is using this module, writing behaviour will not be
|
|
allowed.self.remote = kwargs.get("remote", False)'
|
|
function: cache.DiskExpressionCache
|
|
- docstring: null
|
|
function: cache.get_cache_dir
|
|
- docstring: null
|
|
function: cache._uri
|
|
- docstring: " \nIn most cases, we do not need reader_lock.Because updating\
|
|
\ data is a small probability event compare to reading data."
|
|
function: cache._expression
|
|
- docstring: ' use bin file to save like feature-data.
|
|
|
|
# Make sure the cache runs right when the directory is deleted# while runningmeta
|
|
= {"info": {"instrument": instrument, "field": field, "freq": freq, "last_update":
|
|
last_update},"meta": {"last_visit": time.time(), "visits": 1},}self.logger.debug(f"generating
|
|
expression cache: {meta}")self.clear_cache(cache_path)meta_path = cache_path.with_suffix(".meta")with
|
|
meta_path.open("wb") as f:pickle.dump(meta, f, protocol=C.dump_protocol_version)meta_path.chmod(stat.S_IRWXU
|
|
| stat.S_IRGRP | stat.S_IROTH)df = expression_data.to_frame()r = np.hstack([df.index[0],
|
|
expression_data]).astype("<f")r.tofile(str(cache_path))'
|
|
function: cache.gen_expression_cache
|
|
- docstring: null
|
|
function: cache.update
|
|
- docstring: ' Prepared cache mechanism for server.
|
|
|
|
super(DiskDatasetCache, self).__init__(provider)self.r = get_redis_connection()self.remote
|
|
= kwargs.get("remote", False)@staticmethod'
|
|
function: cache.DiskDatasetCache
|
|
- docstring: null
|
|
function: cache._uri
|
|
- docstring: null
|
|
function: cache.get_cache_dir
|
|
- docstring: ' read_cache_from
|
|
|
|
This function can read data from the disk cache dataset:param cache_path::param
|
|
start_time::param end_time::param fields: The fields order of the dataset cache
|
|
is sorted. So rearrange the columns to make it consistent.:return:'
|
|
function: cache.read_data_from_cache
|
|
- docstring: null
|
|
function: cache._dataset
|
|
- docstring: null
|
|
function: cache._dataset_uri
|
|
- docstring: " \nThe lock is not considered in the class. Please consider the\
|
|
\ lock outside the code.This class is the proxy of the disk data."
|
|
function: 'cache.IndexManager:'
|
|
- docstring: null
|
|
function: cache.get_index
|
|
- docstring: null
|
|
function: cache.sync_to_disk
|
|
- docstring: null
|
|
function: cache.sync_from_disk
|
|
- docstring: null
|
|
function: cache.update
|
|
- docstring: null
|
|
function: cache.append_index
|
|
- docstring: null
|
|
function: cache.build_index_from_data
|
|
- docstring: ' gen_dataset_cache
|
|
|
|
.. note:: This function does not consider the cache read write lock. Pleaseacquire
|
|
the lock outside this functionThe format the cache contains 3 parts(followed by
|
|
typical filename).- index : cache/d41366901e25de3ec47297f12e2ba11d.index- The
|
|
content of the file may be in following format(pandas.Series).. code-block:: pythonstart
|
|
end1999-11-10 00:00:00 0 11999-11-11 00:00:00 1 21999-11-12 00:00:00 2 3.....
|
|
note:: The start is closed. The end is open!!!!!- Each line contains two element
|
|
<start_index, end_index> with a timestamp as its index.- It indicates the `start_index`
|
|
(included) and `end_index` (excluded) of the data for `timestamp`- meta data:
|
|
cache/d41366901e25de3ec47297f12e2ba11d.meta- data : cache/d41366901e25de3ec47297f12e2ba11d-
|
|
This is a hdf file sorted by datetime:param cache_path: The path to store the
|
|
cache.:param instruments: The instruments to store the cache.:param fields: The
|
|
fields to store the cache.:param freq: The freq to store the cache.:param inst_processors: Instrument
|
|
processors.:return type pd.DataFrame; The fields of the returned DataFrame are
|
|
consistent with the parameters of the function.'
|
|
function: cache.gen_dataset_cache
|
|
- docstring: null
|
|
function: cache.update
|
|
- docstring: ' Simple dataset cache that can be used locally or on client.
|
|
|
|
super(SimpleDatasetCache, self).__init__(provider)try:self.local_cache_path: Path
|
|
= Path(C["local_cache_path"]).expanduser().resolve()except (KeyError, TypeError):self.logger.error("Assign
|
|
a local_cache_path in config if you want to use this cache mechanism")raiseself.logger.info(f"DatasetCache
|
|
directory: {self.local_cache_path}, "f"modify the cache directory via the local_cache_path
|
|
in the config")'
|
|
function: cache.SimpleDatasetCache
|
|
- docstring: null
|
|
function: cache._uri
|
|
- docstring: null
|
|
function: cache._dataset
|
|
- docstring: ' Prepared cache mechanism for server.
|
|
|
|
'
|
|
function: cache.DatasetURICache
|
|
- docstring: null
|
|
function: cache._uri
|
|
- docstring: null
|
|
function: cache.dataset
|
|
- docstring: null
|
|
function: cache.CalendarCache
|
|
- docstring: null
|
|
function: cache.MemoryCalendarCache
|
|
- docstring: ' FileStorageMixin, applicable to FileXXXStorage
|
|
|
|
Subclasses need to have provider_uri, freq, storage_name, file_name attributes'
|
|
function: 'file_storage.FileStorageMixin:'
|
|
- docstring: null
|
|
function: file_storage.provider_uri
|
|
- docstring: null
|
|
function: file_storage.dpm
|
|
- docstring: null
|
|
function: file_storage.support_freq
|
|
- docstring: null
|
|
function: file_storage.uri
|
|
- docstring: ' check self.uri
|
|
|
|
Raises-------ValueError'
|
|
function: file_storage.check
|
|
- docstring: null
|
|
function: file_storage.FileCalendarStorage
|
|
- docstring: null
|
|
function: file_storage.file_name
|
|
- docstring: ' the freq to read from file
|
|
|
|
if not hasattr(self, "_freq_file_cache"):freq = Freq(self.freq)if freq not in
|
|
self.support_freq:# NOTE: uri# 1. If `uri` does not exist# - Get the `min_uri`
|
|
of the closest `freq` under the same "directory" as the `uri`# - Read data
|
|
from `min_uri` and resample to `freq`freq = Freq.get_recent_freq(freq, self.support_freq)if
|
|
freq is None:raise ValueError(f"can''t find a freq from {self.support_freq} that
|
|
can resample to {self.freq}!")self._freq_file_cache = freqreturn self._freq_file_cache'
|
|
function: file_storage._freq_file
|
|
- docstring: null
|
|
function: file_storage._read_calendar
|
|
- docstring: null
|
|
function: file_storage._write_calendar
|
|
- docstring: null
|
|
function: file_storage.uri
|
|
- docstring: null
|
|
function: file_storage.data
|
|
- docstring: null
|
|
function: file_storage._get_storage_freq
|
|
- docstring: null
|
|
function: file_storage.extend
|
|
- docstring: null
|
|
function: file_storage.clear
|
|
- docstring: null
|
|
function: file_storage.index
|
|
- docstring: null
|
|
function: file_storage.insert
|
|
- docstring: null
|
|
function: file_storage.remove
|
|
- docstring: null
|
|
function: file_storage.FileInstrumentStorage
|
|
- docstring: null
|
|
function: file_storage._read_instrument
|
|
- docstring: null
|
|
function: file_storage._write_instrument
|
|
- docstring: null
|
|
function: file_storage.clear
|
|
- docstring: null
|
|
function: file_storage.data
|
|
- docstring: null
|
|
function: file_storage.update
|
|
- docstring: null
|
|
function: file_storage.FileFeatureStorage
|
|
- docstring: null
|
|
function: file_storage.clear
|
|
- docstring: null
|
|
function: file_storage.data
|
|
- docstring: null
|
|
function: file_storage.write
|
|
- docstring: null
|
|
function: file_storage.start_index
|
|
- docstring: null
|
|
function: storage.UserCalendarStorage
|
|
- docstring: null
|
|
function: storage.data
|
|
- docstring: null
|
|
function: storage.UserInstrumentStorage
|
|
- docstring: null
|
|
function: storage.data
|
|
- docstring: '
|
|
|
|
'
|
|
function: storage.UserFeatureStorage
|
|
- docstring: null
|
|
function: 'storage.BaseStorage:'
|
|
- docstring: null
|
|
function: storage.storage_name
|
|
- docstring: " \nThe behavior of CalendarStorage's methods and List's methods of\
|
|
\ the same name remain consistent"
|
|
function: storage.CalendarStorage
|
|
- docstring: ' get all data
|
|
|
|
Raises------ValueErrorIf the data(storage) does not exist, raise ValueError'
|
|
function: storage.data
|
|
- docstring: null
|
|
function: storage.clear
|
|
- docstring: null
|
|
function: storage.extend
|
|
- docstring: " \nRaises------ValueErrorIf the data(storage) does not exist,\
|
|
\ raise ValueError"
|
|
function: storage.index
|
|
- docstring: null
|
|
function: storage.insert
|
|
- docstring: ' x.__setitem__(i, o) <==> (x[i] = o)
|
|
|
|
@overloadx.__setitem__(s, o) <==> (x[s] = o)'
|
|
function: storage.remove
|
|
- docstring: null
|
|
function: storage.InstrumentStorage
|
|
- docstring: ' get all data
|
|
|
|
Raises------ValueErrorIf the data(storage) does not exist, raise ValueError'
|
|
function: storage.data
|
|
- docstring: null
|
|
function: storage.clear
|
|
- docstring: ' D.update([E, ]**F) -> None. Update D from mapping/iterable
|
|
E and F.
|
|
|
|
Notes------If E present and has a .keys() method, does: for k in E: D[k] =
|
|
E[k]If E present and lacks .keys() method, does: for (k, v) in E: D[k] = vIn
|
|
either case, this is followed by: for k, v in F.items(): D[k] = v'
|
|
function: storage.update
|
|
- docstring: null
|
|
function: storage.FeatureStorage
|
|
- docstring: ' get all data
|
|
|
|
Notes------if data(storage) does not exist, return empty pd.Series: `return pd.Series(dtype=np.float32)`'
|
|
function: storage.data
|
|
- docstring: ' get FeatureStorage start index
|
|
|
|
Notes-----If the data(storage) does not exist, return None'
|
|
function: storage.start_index
|
|
- docstring: ' get FeatureStorage end index
|
|
|
|
Notes-----The right index of the data range (both sides are closed)The next data
|
|
appending point will be `end_index + 1`If the data(storage) does not exist, return
|
|
None'
|
|
function: storage.end_index
|
|
- docstring: null
|
|
function: storage.clear
|
|
- docstring: ' Write data_array to FeatureStorage starting from index.
|
|
|
|
Notes------If index is None, append data_array to feature.If len(data_array) ==
|
|
0; returnIf (index - self.end_index) >= 1, self[end_index+1: index] will be filled
|
|
with np.nanExamples---------.. code-block::feature:3 44 55 6>>> self.write([6,
|
|
7], index=6)feature:3 44 55 66 67 7>>> self.write([8], index=9)feature:3 44 55 66 67 78 np.nan9 8>>>
|
|
self.write([1, np.nan], index=3)feature:3 14 np.nan5 66 67 78 np.nan9 8'
|
|
function: storage.write
|
|
- docstring: ' Rebase the start_index and end_index of the FeatureStorage.
|
|
|
|
start_index and end_index are closed intervals: [start_index, end_index]Examples---------..
|
|
code-block::feature:3 44 55 6>>> self.rebase(start_index=4)feature:4 55 6>>>
|
|
self.rebase(start_index=3)feature:3 np.nan4 55 6>>> self.write([3], index=3)feature:3 34 55 6>>>
|
|
self.rebase(end_index=4)feature:3 34 5>>> self.write([6, 7, 8], index=4)feature:3 34 65 76 8>>>
|
|
self.rebase(start_index=4, end_index=5)feature:4 65 7'
|
|
function: storage.rebase
|
|
- docstring: ' overwrite all data in FeatureStorage with data
|
|
|
|
Parameters----------data: Union[List, np.ndarray, Tuple]dataindex: intdata start
|
|
index'
|
|
function: storage.rewrite
|
|
- docstring: " \nDataLoader is designed for loading raw data from original data\
|
|
\ source."
|
|
function: loader.DataLoader
|
|
- docstring: " \nload the data as pd.DataFrame.Example of the data (The multi-index\
|
|
\ of the columns is optional.):.. code-block:: textfeature \
|
|
\ label$close $volume Ref($close,\
|
|
\ 1) Mean($close, 3) $high-$low LABEL0datetime instrument2010-01-04 SH600000\
|
|
\ 81.807068 17145150.0 83.737389 83.016739 2.741058 0.0032SH600004\
|
|
\ 13.313329 11800983.0 13.313329 13.317701 0.183632 0.0042SH600005\
|
|
\ 37.796539 12231662.0 38.258602 37.919757 0.970325 0.0289Parameters----------instruments\
|
|
\ : str or dictit can either be the market name or the config file of instruments\
|
|
\ generated by InstrumentProvider.start_time : strstart of the time range.end_time\
|
|
\ : strend of the time range.Returns-------pd.DataFrame:data load from the under\
|
|
\ layer source"
|
|
function: loader.load
|
|
- docstring: " \n(D)ata(L)oader (W)ith (P)arser for features and namesExtracting\
|
|
\ this class so that QlibDataLoader and other dataloaders(such as QdbDataLoader)\
|
|
\ can share the fields."
|
|
function: loader.DLWParser
|
|
- docstring: null
|
|
function: loader._parse_fields_info
|
|
- docstring: " \nload the dataframe for specific groupParameters----------instruments\
|
|
\ :the instruments.exprs : listthe expressions to describe the content of the\
|
|
\ data.names : listthe name of the data.Returns-------pd.DataFrame:the queried\
|
|
\ dataframe."
|
|
function: loader.load_group_df
|
|
- docstring: null
|
|
function: loader.load
|
|
- docstring: ' Same as QlibDataLoader. The fields can be define by config
|
|
|
|
self,config: Tuple[list, tuple, dict],filter_pipe: List = None,swap_level: bool
|
|
= True,freq: Union[str, dict] = "day",inst_processors: Union[dict, list] = None,):'
|
|
function: loader.QlibDataLoader
|
|
- docstring: null
|
|
function: loader.load_group_df
|
|
- docstring: " \nDataLoader that supports loading data from file or as provided."
|
|
function: loader.StaticDataLoader
|
|
- docstring: null
|
|
function: loader.load
|
|
- docstring: null
|
|
function: loader._maybe_load_raw_data
|
|
- docstring: ' DataLoaderDH
|
|
|
|
DataLoader based on (D)ata (H)andlerIt is designed to load multiple data from
|
|
data handler- If you just want to load data from single datahandler, you can write
|
|
them in single data handlerTODO: What make this module not that easy to use.-
|
|
For online scenario- The underlayer data handler should be configured. But data
|
|
loader doesn''t provide such interface & hook.'
|
|
function: loader.DataLoaderDH
|
|
- docstring: " \nThe steps to using a handler1. initialized data handler (call\
|
|
\ by `init`).2. use the data.The data handler try to maintain a handler with 2\
|
|
\ level.`datetime` & `instruments`.Any order of the index level can be supported\
|
|
\ (The order will be implied in the data).The order <`datetime`, `instruments`>\
|
|
\ will be used when the dataframe index name is missed.Example of the data:The\
|
|
\ multi-index of the columns is optional... code-block:: textfeature \
|
|
\ label$close $volume Ref($close,\
|
|
\ 1) Mean($close, 3) $high-$low LABEL0datetime instrument2010-01-04 SH600000\
|
|
\ 81.807068 17145150.0 83.737389 83.016739 2.741058 0.0032SH600004\
|
|
\ 13.313329 11800983.0 13.313329 13.317701 0.183632 0.0042SH600005\
|
|
\ 37.796539 12231662.0 38.258602 37.919757 0.970325 0.0289Tips\
|
|
\ for improving the performance of datahandler- Fetching data with `col_set=CS_RAW`\
|
|
\ will return the raw data and may avoid pandas from copying the data when calling\
|
|
\ `loc`"
|
|
function: handler.DataHandler
|
|
- docstring: " \nconfiguration of data.# what data to be loaded from data sourceThis\
|
|
\ method will be used when loading pickled handler from dataset.The data will\
|
|
\ be initialized with different time range."
|
|
function: handler.config
|
|
- docstring: " \nSet Up the data in case of running initialization for multiple\
|
|
\ timeIt is responsible for maintaining following variable1) self._dataParameters----------enable_cache\
|
|
\ : booldefault value is false:- if `enable_cache` == True:the processed data\
|
|
\ will be saved on disk, and handler will load the cached data from the disk directlywhen\
|
|
\ we call `init` next time"
|
|
function: handler.setup_data
|
|
- docstring: " \nfetch data from underlying data sourceDesign motivation:-\
|
|
\ providing a unified interface for underlying data.- Potential to make the interface\
|
|
\ more friendly.- User can improve performance when fetching data in this extra\
|
|
\ layerParameters----------selector : Union[pd.Timestamp, slice, str]describe\
|
|
\ how to select data by indexIt can be categories as following- fetch single index-\
|
|
\ fetch a range of index- a slice range- pd.Index for specific indexesFollowing\
|
|
\ conflicts may occur- Does [\"20200101\", \"20210101\"] mean selecting this slice\
|
|
\ or these two days?- slice have higher prioritieslevel : Union[str, int]which\
|
|
\ index level to select the datacol_set : Union[str, List[str]]- if isinstance(col_set,\
|
|
\ str):select a set of meaningful, pd.Index columns.(e.g. features, columns)-\
|
|
\ if col_set == CS_RAW:the raw dataset will be returned.- if isinstance(col_set,\
|
|
\ List[str]):select several sets of meaningful columns, the returned data has\
|
|
\ multiple levelsproc_func: Callable- Give a hook for processing data before fetching-\
|
|
\ An example to explain the necessity of the hook:- A Dataset learned some processors\
|
|
\ to process data which is related to data segmentation- It will apply them every\
|
|
\ time when preparing data.- The learned processor require the dataframe remains\
|
|
\ the same format when fitting and applying- However the data format will change\
|
|
\ according to the parameters.- So the processors should be applied to the underlayer\
|
|
\ data.squeeze : boolwhether squeeze columns and indexReturns-------pd.DataFrame."
|
|
function: handler.fetch
|
|
- docstring: null
|
|
function: handler._fetch_data
|
|
- docstring: " \nget the column namesParameters----------col_set : strselect\
|
|
\ a set of meaningful columns.(e.g. features, columns)Returns-------list:list\
|
|
\ of column names"
|
|
function: handler.get_cols
|
|
- docstring: " \nget range selector by number of periodsArgs:cur_date (pd.Timestamp\
|
|
\ or str): current dateperiods (int): number of periods"
|
|
function: handler.get_range_selector
|
|
- docstring: " \nget an iterator of sliced data with given periodsArgs:periods\
|
|
\ (int): number of periods.min_periods (int): minimum periods for sliced dataframe.kwargs\
|
|
\ (dict): will be passed to `self.fetch`."
|
|
function: handler.get_range_iterator
|
|
- docstring: " \nDataHandler with **(L)earnable (P)rocessor**This handler will\
|
|
\ produce three pieces of data in pd.DataFrame format.- DK_R / self._data: the\
|
|
\ raw data loaded from the loader- DK_I / self._infer: the data processed for\
|
|
\ inference- DK_L / self._learn: the data processed for learning model.The motivation\
|
|
\ of using different processor workflows for learning and inferenceHere are some\
|
|
\ examples.- The instrument universe for learning and inference may be different.-\
|
|
\ The processing of some samples may rely on label (for example, some samples\
|
|
\ hit the limit may need extra processing or be dropped).- These processors only\
|
|
\ apply to the learning phase.Tips for data handler- To reduce the memory cost-\
|
|
\ `drop_raw=True`: this will modify the data inplace on raw data;- Please note\
|
|
\ processed data like `self._infer` or `self._learn` are concepts different from\
|
|
\ `segments` in Qlib's `Dataset` like \"train\" and \"test\"- Processed data like\
|
|
\ `self._infer` or `self._learn` are underlying data processed with different\
|
|
\ processors- `segments` in Qlib's `Dataset` like \"train\" and \"test\" are simply\
|
|
\ the time segmentations when querying data(\"train\" are often before \"test\"\
|
|
\ in time-series).- For example, you can query `data._infer` processed by `infer_processors`\
|
|
\ in the \"train\" time segmentation."
|
|
function: handler.DataHandlerLP
|
|
- docstring: null
|
|
function: handler.get_all_processors
|
|
- docstring: " \nfit data without processing the data"
|
|
function: handler.fit
|
|
- docstring: " \nfit and process dataThe input of the `fit` will be the output\
|
|
\ of the previous processor"
|
|
function: handler.fit_process_data
|
|
- docstring: null
|
|
function: handler._run_proc_l
|
|
- docstring: " \nNOTE: it will return True if `len(proc_l) == 0`"
|
|
function: handler._is_proc_readonly
|
|
- docstring: " \nprocess_data data. Fun `processor.fit` if necessaryNotation:\
|
|
\ (data) [processor]# data processing flow of self.process_type == DataHandlerLP.PTYPE_I..\
|
|
\ code-block:: text(self._data)-[shared_processors]-(_shared_df)-[learn_processors]-(_learn_df)\\\
|
|
\\-[infer_processors]-(_infer_df)# data processing flow of self.process_type ==\
|
|
\ DataHandlerLP.PTYPE_A.. code-block:: text(self._data)-[shared_processors]-(_shared_df)-[infer_processors]-(_infer_df)-[learn_processors]-(_learn_df)Parameters----------with_fit\
|
|
\ : boolThe input of the `fit` will be the output of the previous processor"
|
|
function: handler.process_data
|
|
- docstring: " \nconfiguration of data.# what data to be loaded from data sourceThis\
|
|
\ method will be used when loading pickled handler from dataset.The data will\
|
|
\ be initialized with different time range."
|
|
function: handler.config
|
|
- docstring: " \nSet up the data in case of running initialization for multiple\
|
|
\ timeParameters----------init_type : strThe type `IT_*` listed above.enable_cache\
|
|
\ : booldefault value is false:- if `enable_cache` == True:the processed data\
|
|
\ will be saved on disk, and handler will load the cached data from the disk directlywhen\
|
|
\ we call `init` next time"
|
|
function: handler.setup_data
|
|
- docstring: null
|
|
function: handler._get_df_by_key
|
|
- docstring: " \nfetch data from underlying data sourceParameters----------selector\
|
|
\ : Union[pd.Timestamp, slice, str]describe how to select data by index.level\
|
|
\ : Union[str, int]which index level to select the data.col_set : strselect a\
|
|
\ set of meaningful columns.(e.g. features, columns).data_key : strthe data to\
|
|
\ fetch: DK_*.proc_func: Callableplease refer to the doc of DataHandler.fetchReturns-------pd.DataFrame:"
|
|
function: handler.fetch
|
|
- docstring: " \nget the column namesParameters----------col_set : strselect\
|
|
\ a set of meaningful columns.(e.g. features, columns).data_key : DATA_KEY_TYPEthe\
|
|
\ data to fetch: DK_*.Returns-------list:list of column names"
|
|
function: handler.get_cols
|
|
- docstring: " \nMotivation- A user creates a datahandler in his customized\
|
|
\ package. Then he wants to share the processed handler toother users without\
|
|
\ introduce the package dependency and complicated data processing logic.- This\
|
|
\ class make it possible by casting the class to DataHandlerLP and only keep the\
|
|
\ processed dataParameters----------handler : DataHandlerLPA subclass of DataHandlerLPReturns-------DataHandlerLP:the\
|
|
\ converted processed data"
|
|
function: handler.cast
|
|
- docstring: " \nMotivation:- When user want to get a quick data handler.The\
|
|
\ created data handler will have only one shared Dataframe without processors.After\
|
|
\ creating the handler, user may often want to dump the handler for reuseHere\
|
|
\ is a typical use case.. code-block:: pythonfrom qlib.data.dataset import DataHandlerLPdh\
|
|
\ = DataHandlerLP.from_df(df)dh.to_pickle(fname, dump_all=True)TODO:- The StaticDataLoader\
|
|
\ is quite slow. It don't have to copy the data again..."
|
|
function: handler.from_df
|
|
- docstring: " \nget the level index of `df` given `level`Parameters----------df\
|
|
\ : pd.DataFramedatalevel : Union[str, int]index levelReturns-------int:The level\
|
|
\ index in the multiple index"
|
|
function: utils.get_level_index
|
|
- docstring: " \nfetch data from `data` with `selector` and `level`selector are\
|
|
\ assumed to be well processed.`fetch_df_by_index` is only responsible for get\
|
|
\ the right levelParameters----------selector : Union[pd.Timestamp, slice, str,\
|
|
\ list]selectorlevel : Union[int, str]the level to use the selectorReturns-------Data\
|
|
\ of the given index."
|
|
function: utils.fetch_df_by_index
|
|
- docstring: null
|
|
function: utils.fetch_df_by_col
|
|
- docstring: " \nConvert the format of df.MultiIndex according to the following\
|
|
\ rules:- If `level` is the first level of df.MultiIndex, do nothing- If `level`\
|
|
\ is the second level of df.MultiIndex, swap the level of index.NOTE:the number\
|
|
\ of levels of df.MultiIndex should be 2Parameters----------df : Union[pd.DataFrame,\
|
|
\ pd.Series]raw DataFrame/Serieslevel : str, optionalthe level that will be converted\
|
|
\ to the first one, by default \"datetime\"Returns-------Union[pd.DataFrame, pd.Series]converted\
|
|
\ DataFrame/Series"
|
|
function: utils.convert_index_format
|
|
- docstring: " \ninitialize the handler part of the task **inplace**Parameters----------task\
|
|
\ : dictthe task to be handledReturns-------Union[DataHandler, None]:returns"
|
|
function: utils.init_task_handler
|
|
- docstring: " \nget a group of columns from multi-index columns DataFrameParameters----------df\
|
|
\ : pd.DataFramewith multi of columns.group : strthe name of the feature group,\
|
|
\ i.e. the first level value of the group index."
|
|
function: processor.get_group_columns
|
|
- docstring: null
|
|
function: processor.Processor
|
|
- docstring: " \nlearn data processing parametersParameters----------df : pd.DataFrameWhen\
|
|
\ we fit and process data with processor one by one. The fit function reiles on\
|
|
\ the output of previousprocessor, i.e. `df`."
|
|
function: processor.fit
|
|
- docstring: " \nIs this processor usable for inferenceSome processors are\
|
|
\ not usable for inference.Returns-------bool:if it is usable for infenrece."
|
|
function: processor.is_for_infer
|
|
- docstring: " \nDoes the processor treat the input data readonly (i.e. does\
|
|
\ not write the input data) when processingKnowning the readonly information is\
|
|
\ helpful to the Handler to avoid uncessary copy"
|
|
function: processor.readonly
|
|
- docstring: null
|
|
function: processor.config
|
|
- docstring: null
|
|
function: processor.DropnaProcessor
|
|
- docstring: null
|
|
function: processor.readonly
|
|
- docstring: null
|
|
function: processor.DropnaLabel
|
|
- docstring: ' The samples are dropped according to label. So it is not usable
|
|
for inference
|
|
|
|
return False'
|
|
function: processor.is_for_infer
|
|
- docstring: null
|
|
function: processor.DropCol
|
|
- docstring: null
|
|
function: processor.readonly
|
|
- docstring: null
|
|
function: processor.FilterCol
|
|
- docstring: null
|
|
function: processor.readonly
|
|
- docstring: ' Use tanh to process noise data
|
|
|
|
'
|
|
function: processor.TanhProcess
|
|
- docstring: null
|
|
function: processor.tanh_denoise
|
|
- docstring: ' Process infinity
|
|
|
|
'
|
|
function: processor.ProcessInf
|
|
- docstring: null
|
|
function: processor.replace_inf
|
|
- docstring: null
|
|
function: processor.process_inf
|
|
- docstring: ' Process NaN
|
|
|
|
self.fields_group = fields_groupself.fill_value = fill_valueif self.fields_group
|
|
is None:df.fillna(self.fill_value, inplace=True)else:cols = get_group_columns(df,
|
|
self.fields_group)# this implementation is extremely slow# df.fillna({col: self.fill_value
|
|
for col in cols}, inplace=True)# So we use numpy to accelerate filling valuesnan_select
|
|
= np.isnan(df.values)nan_select[:, ~df.columns.isin(cols)] = Falsedf.values[nan_select]
|
|
= self.fill_valuereturn df'
|
|
function: processor.Fillna
|
|
- docstring: null
|
|
function: processor.MinMaxNorm
|
|
- docstring: null
|
|
function: processor.fit
|
|
- docstring: null
|
|
function: processor.normalize
|
|
- docstring: ' ZScore Normalization
|
|
|
|
# NOTE: correctly set the `fit_start_time` and `fit_end_time` is very important
|
|
!!!# `fit_end_time` **must not** include any information from the test data!!!self.fit_start_time
|
|
= fit_start_timeself.fit_end_time = fit_end_timeself.fields_group = fields_group'
|
|
function: processor.ZScoreNorm
|
|
- docstring: null
|
|
function: processor.fit
|
|
- docstring: null
|
|
function: processor.normalize
|
|
- docstring: ' Robust ZScore Normalization
|
|
|
|
Use robust statistics for Z-Score normalization:mean(x) = median(x)std(x) = MAD(x)
|
|
* 1.4826Reference:https://en.wikipedia.org/wiki/Median_absolute_deviation.'
|
|
function: processor.RobustZScoreNorm
|
|
- docstring: null
|
|
function: processor.fit
|
|
- docstring: ' Cross Sectional ZScore Normalization
|
|
|
|
self.fields_group = fields_groupif method == "zscore":self.zscore_func = zscoreelif
|
|
method == "robust":self.zscore_func = robust_zscoreelse:raise NotImplementedError(f"This
|
|
type of input is not supported")# try not modify original dataframeif not isinstance(self.fields_group,
|
|
list):self.fields_group = [self.fields_group]for g in self.fields_group:cols =
|
|
get_group_columns(df, g)df[cols] = df[cols].groupby("datetime", group_keys=False).apply(self.zscore_func)return
|
|
df'
|
|
function: processor.CSZScoreNorm
|
|
- docstring: " \nCross Sectional Rank Normalization.\"Cross Sectional\" is often\
|
|
\ used to describe data operations.The operations across different stocks are\
|
|
\ often called Cross Sectional Operation.For example, CSRankNorm is an operation\
|
|
\ that grouping the data by each day and rank `across` all the stocks in each\
|
|
\ day.Explanation about 3.46 & 0.5.. code-block:: pythonimport numpy as npimport\
|
|
\ pandas as pdx = np.random.random(10000) # for any variablex_rank = pd.Series(x).rank(pct=True)\
|
|
\ # if it is converted to rank, it will be a uniform distributedx_rank_norm =\
|
|
\ (x_rank - x_rank.mean()) / x_rank.std() # Normally, we will normalize it to\
|
|
\ make it like normal distributionx_rank.mean() # accounts for 0.51 / x_rank.std()\
|
|
\ # accounts for 3.46"
|
|
function: processor.CSRankNorm
|
|
- docstring: ' Cross Sectional Fill Nan
|
|
|
|
self.fields_group = fields_groupcols = get_group_columns(df, self.fields_group)df[cols]
|
|
= df[cols].groupby("datetime", group_keys=False).apply(lambda x: x.fillna(x.mean()))return
|
|
df'
|
|
function: processor.CSZFillna
|
|
- docstring: ' Process the storage of from df into hasing stock format
|
|
|
|
from .storage import HashingStockStorage # pylint: disable=C0415return HashingStockStorage.from_df(df)'
|
|
function: processor.HashStockFormat
|
|
- docstring: " \nThis is a filter to filter stock.Only keep the data that exist\
|
|
\ from start_time to end_time (the existence in the middle is not checked.)WARNING:\
|
|
\ It may induce leakage!!!"
|
|
function: processor.TimeRangeFlt
|
|
- docstring: " \nPreparing data for model training and inferencing."
|
|
function: __init__.Dataset
|
|
- docstring: " \nconfig is designed to configure and parameters that cannot\
|
|
\ be learned from the data"
|
|
function: __init__.config
|
|
- docstring: " \nSetup the data.We split the setup_data function for following\
|
|
\ situation:- User have a Dataset object with learned status on disk.- User load\
|
|
\ the Dataset object from the disk.- User call `setup_data` to load new data.-\
|
|
\ User prepare data for model based on previous status."
|
|
function: __init__.setup_data
|
|
- docstring: " \nThe type of dataset depends on the model. (It could be pd.DataFrame,\
|
|
\ pytorch.DataLoader, etc.)The parameters should specify the scope for the prepared\
|
|
\ dataThe method should:- process the data- return the processed dataReturns-------object:return\
|
|
\ the object"
|
|
function: __init__.prepare
|
|
- docstring: " \nDataset with Data(H)andlerUser should try to put the data preprocessing\
|
|
\ functions into handler.Only following data processing functions should be placed\
|
|
\ in Dataset:- The processing is related to specific model.- The processing is\
|
|
\ related to data split."
|
|
function: __init__.DatasetH
|
|
- docstring: " \nInitialize the DatasetHParameters----------handler_kwargs\
|
|
\ : dictConfig of DataHandler, which could include the following arguments:- arguments\
|
|
\ of DataHandler.conf_data, such as 'instruments', 'start_time' and 'end_time'.kwargs\
|
|
\ : dictConfig of DatasetH, such as- segments : dictConfig of segments which is\
|
|
\ same as 'segments' in self.__init__"
|
|
function: __init__.config
|
|
- docstring: " \nSetup the DataParameters----------handler_kwargs : dictinit\
|
|
\ arguments of DataHandler, which could include the following arguments:- init_type\
|
|
\ : Init Type of Handler- enable_cache : whether to enable cache"
|
|
function: __init__.setup_data
|
|
- docstring: " \nGive a query, retrieve the according dataParameters----------slc\
|
|
\ : please refer to the docs of `prepare`NOTE: it may not be an instance of slice.\
|
|
\ It may be a segment of `segments` from `def prepare`"
|
|
function: __init__._prepare_seg
|
|
- docstring: " \nPrepare the data for learning and inference.Parameters----------segments\
|
|
\ : Union[List[Text], Tuple[Text], Text, slice]Describe the scope of the data\
|
|
\ to be preparedHere are some examples:- 'train'- ['train', 'valid']col_set :\
|
|
\ strThe col_set will be passed to self.handler when fetching data.TODO: make\
|
|
\ it automatic:- select DK_I for test data- select DK_L for training data.data_key\
|
|
\ : strThe data to fetch: DK_*Default is DK_I, which indicate fetching data for\
|
|
\ **inference**.kwargs :The parameters that kwargs may contain:flt_col : strIt\
|
|
\ only exists in TSDatasetH, can be used to add a column of data(True or False)\
|
|
\ to filter data.This parameter is only supported when it is an instance of TSDatasetH.Returns-------Union[List[pd.DataFrame],\
|
|
\ pd.DataFrame]:Raises------NotImplementedError:"
|
|
function: __init__.prepare
|
|
- docstring: null
|
|
function: __init__.get_min_time
|
|
- docstring: null
|
|
function: __init__.get_max_time
|
|
- docstring: ' it will act like sort and return the max value or None
|
|
|
|
candidate = Nonefor k, seg in segments.items():point = seg[idx]if point is None:#
|
|
None indicates unbounded, return directlyreturn Noneelif candidate is None or
|
|
cmp(key_func(candidate), key_func(point)):candidate = pointreturn candidate'
|
|
function: __init__._get_extrema
|
|
- docstring: " \n(T)ime-(S)eries DataSamplerThis is the result of TSDatasetHIt\
|
|
\ works like `torch.data.utils.Dataset`, it provides a very convenient interface\
|
|
\ for constructing time-seriesdataset based on tabular data.- On time step dimension,\
|
|
\ the smaller index indicates the historical data and the larger index indicates\
|
|
\ the futuredata.If user have further requirements for processing data, user could\
|
|
\ process them based on `TSDataSampler` or createmore powerful subclasses.Known\
|
|
\ Issues:- For performance issues, this Sampler will convert dataframe into arrays\
|
|
\ for better performance. This could resultin a different data typeIndices design:TSDataSampler\
|
|
\ has a index mechanism to help users query time-series data efficiently.The definition\
|
|
\ of related variables:data_arr: np.ndarrayThe original data. it will contains\
|
|
\ all the original data.The querying are often for time-series of a specific stock.By\
|
|
\ leveraging this data charactoristics to speed up querying, the multi-index of\
|
|
\ data_arr is rearranged in (instrument, datetime) orderdata_index: pd.MultiIndex\
|
|
\ with index order <instrument, datetime>it has the same shape with `idx_map`.\
|
|
\ Each elements of them are expected to be aligned.idx_map: np.ndarrayIt is the\
|
|
\ indexable data. It originates from data_arr, and then filtered by 1) `start`\
|
|
\ and `end` 2) `flt_data`The extra data in data_arr is useful in following cases1)\
|
|
\ creating meaningful time series data before `start` instead of padding them\
|
|
\ with zeros2) some data are excluded by `flt_data` (e.g. no <X, y> sample pair\
|
|
\ for that index). but they are still used in time-series in XFinnally, it will\
|
|
\ look like.array([[ 0, 0],[ 1, 0],[ 2, 0],...,[241, 348],[242, 348],[243,\
|
|
\ 348]], dtype=int32)It list all indexable data(some data only used in historical\
|
|
\ time series data may not be indexabla), the values are the corresponding row\
|
|
\ and col in idx_dfidx_df: pd.DataFrameIt aims to map the <datetime, instrument>\
|
|
\ key to the original position in data_arrFor example, it may look like (NOTE:\
|
|
\ the index for a instrument time-series is continoues in memory)instrument SH600000\
|
|
\ SH600008 SH600009 SH600010 SH600011 SH600015 ...datetime2017-01-03 0\
|
|
\ 242 473 717 NaN 974 ...2017-01-04 1 243\
|
|
\ 474 718 NaN 975 ...2017-01-05 2 244 475\
|
|
\ 719 NaN 976 ...2017-01-06 3 245 476 720\
|
|
\ NaN 977 ...With these two indices(idx_map, idx_df) and original data(data_arr),\
|
|
\ we can make the following queries fast (implemented in __getitem__)(1) Get the\
|
|
\ i-th indexable sample(time-series): (indexable sample index) -> [idx_map]\
|
|
\ -> (row col) -> [idx_df] -> (index in data_arr)(2) Get the specific sample by\
|
|
\ <datetime, instrument>: (<datetime, instrument>, i.e. <row, col>) -> [idx_df]\
|
|
\ -> (index in data_arr)(3) Get the index of a time-series data: (get the <row,\
|
|
\ col>, refer to (1), (2)) -> [idx_df] -> (all indices in data_arr for time-series)"
|
|
function: '__init__.TSDataSampler:'
|
|
- docstring: null
|
|
function: __init__.slice_idx_map_and_data_index
|
|
- docstring: null
|
|
function: __init__.idx_map2arr
|
|
- docstring: null
|
|
function: __init__.flt_idx_map
|
|
- docstring: " \nGet the pandas index of the data, it will be useful in following\
|
|
\ scenarios- Special sampler will be used (e.g. user want to sample day by day)"
|
|
function: __init__.get_index
|
|
- docstring: null
|
|
function: __init__.config
|
|
- docstring: " \nThe relation of the dataParameters----------data : pd.DataFrameA\
|
|
\ DataFrame with index in order <instrument, datetime>RSQR5 RESI5 WVMA5\
|
|
\ LABEL0instrument datetimeSH600000 2017-01-03 0.016389 0.461632 -1.154788\
|
|
\ -0.0480562017-01-04 0.884545 -0.110597 -1.059332 -0.0301392017-01-05 0.507540\
|
|
\ -0.535493 -1.099665 -0.6449832017-01-06 -1.267771 -0.669685 -1.636733 0.2953662017-01-09\
|
|
\ 0.339346 0.074317 -0.984989 0.765540Returns-------Tuple[pd.DataFrame, dict]:1)\
|
|
\ the first element: reshape the original index into a <datetime(row), instrument(column)>\
|
|
\ 2D dataframeinstrument SH600000 SH600008 SH600009 SH600010 SH600011 SH600015\
|
|
\ ...datetime2017-01-03 0 242 473 717 NaN 974\
|
|
\ ...2017-01-04 1 243 474 718 NaN 975 ...2017-01-05\
|
|
\ 2 244 475 719 NaN 976 ...2017-01-06 \
|
|
\ 3 245 476 720 NaN 977 ...2) the second element: {<original\
|
|
\ index>: <row, col>}"
|
|
function: __init__.build_index
|
|
- docstring: null
|
|
function: __init__.empty
|
|
- docstring: " \nget series indices of self.data_arr from the row, col indices\
|
|
\ of self.idx_dfParameters----------row : intthe row in self.idx_dfcol : intthe\
|
|
\ col in self.idx_dfReturns-------np.array:The indices of data of the data"
|
|
function: __init__._get_indices
|
|
- docstring: " \nget the col index and row index of a given sample index in\
|
|
\ self.idx_dfParameters----------idx :the input of `__getitem__`Returns-------Tuple[int]:the\
|
|
\ row and col index"
|
|
function: __init__._get_row_col
|
|
- docstring: " \n(T)ime-(S)eries Dataset (H)andlerConvert the tabular data to Time-Series\
|
|
\ dataRequirements analysisThe typical workflow of a user to get time-series data\
|
|
\ for an sample- process features- slice proper data from data handler: dimension\
|
|
\ of sample <feature, >- Build relation of samples by <time, instrument> index-\
|
|
\ Be able to sample times series of data <timestep, feature>- It will be better\
|
|
\ if the interface is like \"torch.utils.data.Dataset\"- User could build customized\
|
|
\ batch based on the data- The dimension of a batch of data <batch_idx, feature,\
|
|
\ timestep>"
|
|
function: __init__.TSDatasetH
|
|
- docstring: null
|
|
function: __init__.config
|
|
- docstring: null
|
|
function: __init__.setup_data
|
|
- docstring: null
|
|
function: __init__._extend_slice
|
|
- docstring: " \nsplit the _prepare_raw_seg is to leave a hook for data preprocessing\
|
|
\ before creating processing dataNOTE: TSDatasetH only support slc segment on\
|
|
\ datetime !!!"
|
|
function: __init__._prepare_seg
|
|
- docstring: " \nTo initialize the Reweighter, users should provide specific\
|
|
\ methods to let reweighter do the reweighting (such as sample-wise, rule-based)."
|
|
function: 'weight.Reweighter:'
|
|
- docstring: " \nGet weights for dataParameters----------data : objectThe input\
|
|
\ data.The first dimension is the index of samplesReturns-------object:the weights\
|
|
\ info for the data"
|
|
function: weight.reweight
|
|
- docstring: " \nBase data storage for datahandler- pd.DataFrame is the default\
|
|
\ data storage format in Qlib datahandler- If users want to use custom data storage,\
|
|
\ they should define subclass inherited BaseHandlerStorage, and implement the\
|
|
\ following method"
|
|
function: 'storage.BaseHandlerStorage:'
|
|
- docstring: ' fetch data from the data storage
|
|
|
|
Parameters----------selector : Union[pd.Timestamp, slice, str]describe how to
|
|
select data by indexlevel : Union[str, int]which index level to select the data-
|
|
if level is None, apply selector to df directlycol_set : Union[str, List[str]]-
|
|
if isinstance(col_set, str):select a set of meaningful columns.(e.g. features,
|
|
columns)if col_set == DataHandler.CS_RAW:the raw dataset will be returned.- if
|
|
isinstance(col_set, List[str]):select several sets of meaningful columns, the
|
|
returned data has multiple levelfetch_orig : boolReturn the original data instead
|
|
of copy if possible.proc_func: Callableplease refer to the doc of DataHandler.fetchReturns-------pd.DataFramethe
|
|
dataframe fetched'
|
|
function: storage.fetch
|
|
- docstring: null
|
|
function: storage.from_df
|
|
- docstring: ' whether the arg `proc_func` in `fetch` method is supported.
|
|
|
|
raise NotImplementedError("is_proc_func_supported method is not implemented!")'
|
|
function: storage.is_proc_func_supported
|
|
- docstring: ' Hashing data storage for datahanlder
|
|
|
|
- The default data storage pandas.DataFrame is too slow when randomly accessing
|
|
one stock''s data- HashingStockStorage hashes the multiple stocks'' data(pandas.DataFrame)
|
|
by the key `stock_id`.- HashingStockStorage hashes the pandas.DataFrame into a
|
|
dict, whose key is the stock_id(str) and value this stock data(panda.DataFrame),
|
|
it has the following format:{stock1_id: stock1_data,stock2_id: stock2_data,...stockn_id:
|
|
stockn_data,}- By the `fetch` method, users can access any stock data with much
|
|
lower time cost than default data storage'
|
|
function: storage.HashingStockStorage
|
|
- docstring: null
|
|
function: storage.from_df
|
|
- docstring: ' fetch the data with stock selector
|
|
|
|
Parameters----------selector : Union[pd.Timestamp, slice, str]describe how to
|
|
select data by indexlevel : Union[str, int]which index level to select the data-
|
|
if level is None, apply selector to df directly- the `_fetch_hash_df_by_stock`
|
|
will parse the stock selector in arg `selector`Returns-------DictThe dict whose
|
|
key is stock_id, value is the stock''s data'
|
|
function: storage._fetch_hash_df_by_stock
|
|
- docstring: null
|
|
function: storage.fetch
|
|
- docstring: ' Risk Analysis
|
|
|
|
NOTE:The calculation of annulaized return is different from the definition of
|
|
annualized return.It is implemented by design.Qlib tries to cumulated returns
|
|
by summation instead of production to avoid the cumulated curve being skewed exponentially.All
|
|
the calculation of annualized returns follows this principle in Qlib.TODO: add
|
|
a parameter to enable calculating metrics with production accumulation of return.Parameters----------r
|
|
: pandas.Seriesdaily return series.N: intscaler for annualizing information_ratio
|
|
(day: 252, week: 50, month: 12), at least one of `N` and `freq` should existfreq:
|
|
stranalysis frequency used for calculating the scaler, at least one of `N` and
|
|
`freq` should exist'
|
|
function: evaluate.risk_analysis
|
|
- docstring: null
|
|
function: evaluate.cal_risk_analysis_scaler
|
|
- docstring: ' analyze statistical time-series indicators of trading
|
|
|
|
Parameters----------df : pandas.DataFramecolumns: like [''pa'', ''pos'', ''ffr'',
|
|
''deal_amount'', ''value''].Necessary fields:- ''pa'' is the price advantage in
|
|
trade indicators- ''pos'' is the positive rate in trade indicators- ''ffr'' is
|
|
the fulfill rate in trade indicatorsOptional fields:- ''deal_amount'' is the total
|
|
deal deal_amount, only necessary when method is ''amount_weighted''- ''value''
|
|
is the total trade value, only necessary when method is ''value_weighted''index:
|
|
Index(datetime)method : str, optionalstatistics method of pa/ffr, by default "mean"-
|
|
if method is ''mean'', count the mean statistical value of each trade indicator-
|
|
if method is ''amount_weighted'', count the deal_amount weighted mean statistical
|
|
value of each trade indicator- if method is ''value_weighted'', count the value
|
|
weighted mean statistical value of each trade indicatorNote: statistics method
|
|
of pos is always "mean"Returns-------pd.DataFramestatistical value of each trade
|
|
indicators'
|
|
function: evaluate.indicator_analysis
|
|
- docstring: ' initialize the strategy and executor, then executor the backtest
|
|
of daily frequency
|
|
|
|
Parameters----------start_time : Union[str, pd.Timestamp]closed start time for
|
|
backtest**NOTE**: This will be applied to the outmost executor''s calendar.end_time
|
|
: Union[str, pd.Timestamp]closed end time for backtest**NOTE**: This will be applied
|
|
to the outmost executor''s calendar.E.g. Executor[day](Executor[1min]), setting
|
|
`end_time == 20XX0301` will include all the minutes on 20XX0301strategy : Union[str,
|
|
dict, BaseStrategy]for initializing outermost portfolio strategy. Please refer
|
|
to the docs of init_instance_by_config for more information.E.g... code-block::
|
|
python# dictstrategy = {"class": "TopkDropoutStrategy","module_path": "qlib.contrib.strategy.signal_strategy","kwargs":
|
|
{"signal": (model, dataset),"topk": 50,"n_drop": 5,},}# BaseStrategypred_score
|
|
= pd.read_pickle("score.pkl")["score"]STRATEGY_CONFIG = {"topk": 50,"n_drop":
|
|
5,"signal": pred_score,}strategy = TopkDropoutStrategy(**STRATEGY_CONFIG)# str
|
|
example.# 1) specify a pickle object# - path like ''file:///<path to pickle
|
|
file>/obj.pkl''# 2) specify a class name# - "ClassName": getattr(module,
|
|
"ClassName")() will be used.# 3) specify module path with class name# - "a.b.c.ClassName"
|
|
getattr(<a.b.c.module>, "ClassName")() will be used.executor : Union[str, dict,
|
|
BaseExecutor]for initializing the outermost executor.benchmark: strthe benchmark
|
|
for reporting.account : Union[float, int, Position]information for describing
|
|
how to creating the accountFor `float` or `int`:Using Account with only initial
|
|
cashFor `Position`:Using Account with a Positionexchange_kwargs : dictthe kwargs
|
|
for initializing ExchangeE.g... code-block:: pythonexchange_kwargs = {"freq":
|
|
freq,"limit_threshold": None, # limit_threshold is None, using C.limit_threshold"deal_price":
|
|
None, # deal_price is None, using C.deal_price"open_cost": 0.0005,"close_cost":
|
|
0.0015,"min_cost": 5,}pos_type : strthe type of Position.Returns-------report_normal:
|
|
pd.DataFramebacktest reportpositions_normal: pd.DataFramebacktest positions'
|
|
function: evaluate.backtest_daily
|
|
- docstring: " \nA backtest for long-short strategy:param pred: The trading\
|
|
\ signal produced on day `T`.:param topk: The short topk securities and\
|
|
\ long topk securities.:param deal_price: The price to deal the trading.:param\
|
|
\ shift: Whether to shift prediction by one day. The trading day will be\
|
|
\ T+1 if shift==1.:param open_cost: open transaction cost.:param close_cost:\
|
|
\ close transaction cost.:param trade_unit: 100 for China A.:param limit_threshold:\
|
|
\ limit move 0.1 (10%) for example, long and short with same limit.:param min_cost:\
|
|
\ min transaction cost.:param subscribe_fields: subscribe fields.:param extract_codes:\
|
|
\ bool.will we pass the codes extracted from the pred to the exchange.NOTE: This\
|
|
\ will be faster with offline qlib.:return: The result of backtest,\
|
|
\ it is represented by a dict.{ \"long\": long_returns(excess),\"short\": short_returns(excess),\"\
|
|
long_short\": long_short_returns}"
|
|
function: evaluate.long_short_backtest
|
|
- docstring: null
|
|
function: 'analyzer.AnalyzerTemp:'
|
|
- docstring: " \nIt behaves the same as self.recorder.load_object.But it is\
|
|
\ an easier interface because users don't have to care about `get_path` and `artifact_path`Parameters----------name\
|
|
\ : strthe name for the file to be load.Return------The stored records."
|
|
function: analyzer.load
|
|
- docstring: " \nAnalyse data index, distribution .etcParameters----------Return------The\
|
|
\ handled data."
|
|
function: analyzer.analyse
|
|
- docstring: " \nThis is the Signal Analysis class that generates the analysis\
|
|
\ results such as IC and IR.default output image filename is \"HFAnalyzerTable.jpeg\""
|
|
function: analyzer.HFAnalyzer
|
|
- docstring: null
|
|
function: analyzer.analyse
|
|
- docstring: " \nThis is the Signal Analysis class that generates the analysis\
|
|
\ results such as IC and IR.default output image filename is \"signalAnalysis.jpeg\""
|
|
function: analyzer.SignalAnalyzer
|
|
- docstring: ' Get position value by existed close data df
|
|
|
|
close_data_df:pd.DataFramemulti-indexclose_data_df[''$close''][stock_id][evaluate_date]:
|
|
close price for (stock_id, evaluate_date)position:same in get_position_value()'
|
|
function: evaluate_portfolio._get_position_value_from_df
|
|
- docstring: ' sum of close*amount
|
|
|
|
get value of positionuse close pricepositions:{Timestamp(''2016-01-05 00:00:00''):{''SH600022'':{''amount'':100.00,''price'':12.00},''cash'':100000.0}}It
|
|
means Hold 100.0 ''SH600022'' and 100000.0 RMB in ''2016-01-05'''
|
|
function: evaluate_portfolio.get_position_value
|
|
- docstring: null
|
|
function: evaluate_portfolio.get_position_list_value
|
|
- docstring: ' Parameters
|
|
|
|
generate daily return series from position viewpositions: positions generated
|
|
by strategyinit_asset_value : init asset valuereturn: pd.Series of daily return
|
|
, return_series[date] = daily return rate'
|
|
function: evaluate_portfolio.get_daily_return_series_from_positions
|
|
- docstring: ' Annualized Returns
|
|
|
|
p_r = (p_end / p_start)^{(250/n)} - 1p_r annual returnp_end final valuep_start
|
|
init valuen days of backtest'
|
|
function: evaluate_portfolio.get_annual_return_from_positions
|
|
- docstring: ' Risk Analysis from daily return series
|
|
|
|
Parameters----------r : pandas.Seriesdaily return seriesmethod : strinterest calculation
|
|
method, ci(compound interest)/si(simple interest)'
|
|
function: evaluate_portfolio.get_annaul_return_from_return_series
|
|
- docstring: ' Risk Analysis
|
|
|
|
Parameters----------r : pandas.Seriesdaily return seriesmethod : strinterest calculation
|
|
method, ci(compound interest)/si(simple interest)risk_free_rate : floatrisk_free_rate,
|
|
default as 0.00, can set as 0.03 etc'
|
|
function: evaluate_portfolio.get_sharpe_ratio_from_return_series
|
|
- docstring: ' Risk Analysis from asset value
|
|
|
|
cumprod wayParameters----------r : pandas.Seriesdaily return series'
|
|
function: evaluate_portfolio.get_max_drawdown_from_series
|
|
- docstring: null
|
|
function: evaluate_portfolio.get_turnover_rate
|
|
- docstring: ' Risk Analysis beta
|
|
|
|
Parameters----------r : pandas.Seriesdaily return series of strategyb : pandas.Seriesdaily
|
|
return series of baseline'
|
|
function: evaluate_portfolio.get_beta
|
|
- docstring: null
|
|
function: evaluate_portfolio.get_alpha
|
|
- docstring: null
|
|
function: evaluate_portfolio.get_volatility_from_series
|
|
- docstring: ' Rank IC
|
|
|
|
Parameters----------r : pandas.Seriesdaily score series of featureb : pandas.Seriesdaily
|
|
return series'
|
|
function: evaluate_portfolio.get_rank_ic
|
|
- docstring: " \nThis model will load a score file, and return score at date exists\
|
|
\ in score file."
|
|
function: online_model.ScoreFileModel
|
|
- docstring: null
|
|
function: online_model.get_data_with_date
|
|
- docstring: null
|
|
function: online_model.predict
|
|
- docstring: null
|
|
function: online_model.score
|
|
- docstring: null
|
|
function: online_model.fit
|
|
- docstring: " \nload a pickle fileParameterfile_path : string / pathlib.Path()path\
|
|
\ of file to be loaded:returnAn instance loaded from file"
|
|
function: utils.load_instance
|
|
- docstring: " \nsave(dump) an instance to a pickle fileParameterinstance :data\
|
|
\ to be dumpedfile_path : string / pathlib.Path()path of file to be dumped"
|
|
function: utils.save_instance
|
|
- docstring: null
|
|
function: utils.create_user_folder
|
|
- docstring: " \n1. Get the dates that need to do trading till today for user {user_id}dates[0]\
|
|
\ indicate the latest trading date of User{user_id},if User{user_id} haven't do\
|
|
\ trading before, than dates[0] presents the init date of User{user_id}.2. Set\
|
|
\ the exchange with exchange_config fileParameterum : UserManager()today : pd.Timestamp()user_id\
|
|
\ : str:returndates : list of pd.Timestamptrade_exchange : Exchange()"
|
|
function: utils.prepare
|
|
- docstring: " \nThis module is designed to manager the users in online systemall\
|
|
\ users' data were assumed to be saved in user_data_pathParameteruser_data_path\
|
|
\ : stringdata path that all users' data were saved invariables:data_path : stringdata\
|
|
\ path that all users' data were saved inusers_file : stringA path of the file\
|
|
\ record the add_date of userssave_report : boolwhether to save report after each\
|
|
\ trading processusers : dict{}[user_id]->User()the python dict save instances\
|
|
\ of User() for each user_iduser_record : pd.Dataframeuser_id(string), add_date(string)indicate\
|
|
\ the add_date for each users"
|
|
function: 'manager.UserManager:'
|
|
- docstring: " \nload all users' data into manager"
|
|
function: manager.load_users
|
|
- docstring: " \nreturn a instance of User() represents a user to be processedParameteruser_id\
|
|
\ : string:returnuser : User()"
|
|
function: manager.load_user
|
|
- docstring: " \nsave a instance of User() to user data pathParameteruser_id\
|
|
\ : string"
|
|
function: manager.save_user_data
|
|
- docstring: " \nadd the new user {user_id} into user datawill create a new\
|
|
\ folder named \"{user_id}\" in user data pathParameteruser_id : stringinit_cash\
|
|
\ : intconfig_file : str/pathlib.Path()path of config file"
|
|
function: manager.add_user
|
|
- docstring: " \nremove user {user_id} in current user datasetwill delete the\
|
|
\ folder \"{user_id}\" in user data path:paramuser_id : string"
|
|
function: manager.remove_user
|
|
- docstring: " \nParameters----------client: strThe qlib client config file(.yaml)"
|
|
function: 'operator.Operator:'
|
|
- docstring: ' Initial UserManager(), get predict date and trade date
|
|
|
|
Parameters----------client: strThe qlib client config file(.yaml)path : strPath
|
|
to save user account.date : str (YYYY-MM-DD)Trade date, when the generated order
|
|
list will be traded.Return----------um: UserManager()pred_date: pd.Timestamptrade_date:
|
|
pd.Timestamp'
|
|
function: operator.init
|
|
- docstring: ' Add a new user into the a folder to run ''online'' module.
|
|
|
|
Parameters----------id : strUser id, should be unique.config : strThe file path
|
|
(yaml) of user configpath : strPath to save user account.date : str (YYYY-MM-DD)The
|
|
date that user account was added.'
|
|
function: operator.add_user
|
|
- docstring: ' Remove user from folder used in ''online'' module.
|
|
|
|
Parameters----------id : strUser id, should be unique.path : strPath to save user
|
|
account.'
|
|
function: operator.remove_user
|
|
- docstring: ' Generate order list that will be traded at ''date''.
|
|
|
|
Parameters----------date : str (YYYY-MM-DD)Trade date, when the generated order
|
|
list will be traded.path : strPath to save user account.'
|
|
function: operator.generate
|
|
- docstring: ' Execute the orderlist at ''date''.
|
|
|
|
Parameters----------date : str (YYYY-MM-DD)Trade date, that the generated order
|
|
list will be traded.exchange_config: strThe file path (yaml) of exchange configpath
|
|
: strPath to save user account.'
|
|
function: operator.execute
|
|
- docstring: ' Update account at ''date''.
|
|
|
|
Parameters----------date : str (YYYY-MM-DD)Trade date, that the generated order
|
|
list will be traded.path : strPath to save user account.type : strwhich executor
|
|
was been used to execute the order list''SIM'': SimulatorExecutor()'
|
|
function: operator.update
|
|
- docstring: ' Run the ( generate_trade_decision -> execute_order_list -> update_account)
|
|
process everyday
|
|
|
|
from start date to end date.Parameters----------id : struser id, need to be uniqueconfig
|
|
: strThe file path (yaml) of user configexchange_config: strThe file path (yaml)
|
|
of exchange configstart : str "YYYY-MM-DD"The start date to run the online simulateend
|
|
: str "YYYY-MM-DD"The end date to run the online simulatepath : strPath to save
|
|
user account.bench : strThe benchmark that our result compared with.''SH000905''
|
|
for csi500, ''SH000300'' for csi300'
|
|
function: operator.simulate
|
|
- docstring: ' show the newly report (mean, std, information_ratio, annualized_return)
|
|
|
|
Parameters----------id : struser id, need to be uniquepath : strPath to save user
|
|
account.bench : strThe benchmark that our result compared with.''SH000905'' for
|
|
csi500, ''SH000300'' for csi300'
|
|
function: operator.show
|
|
- docstring: " \nWill be called in online moduleneed to return the data that\
|
|
\ used to predict the label (score) of stocks at date.:paramdate: pd.Timestamppredict\
|
|
\ date:return:data: the input data that used to predict the label (score) of stocks\
|
|
\ at predict date."
|
|
function: __init__.get_data_with_date
|
|
- docstring: " \nA user in online system, which contains account, strategy\
|
|
\ and model three module.Parameteraccount : Account()strategy :a strategy instancemodel\
|
|
\ :a model instancereport_save_path : stringthe path to save report. Will not\
|
|
\ save report if Noneverbose : boolWhether to print the info during the process"
|
|
function: 'user.User:'
|
|
- docstring: " \ninit state when each trading date beginParameterdate : pd.Timestamp"
|
|
function: user.init_state
|
|
- docstring: " \nreturn the latest trading date for user {user_id}Parameteruser_id\
|
|
\ : string:returndate : string (e.g '2018-10-08')"
|
|
function: user.get_latest_trading_date
|
|
- docstring: " \nshow the newly report (mean, std, information_ratio, annualized_return)Parameterbenchmark\
|
|
\ : stringbench that to be compared, 'SH000905' for csi500"
|
|
function: user.showReport
|
|
- docstring: null
|
|
function: 'order_generator.OrderGenerator:'
|
|
- docstring: ' generate_order_list_from_target_weight_position
|
|
|
|
:param current: The current position:type current: Position:param trade_exchange::type
|
|
trade_exchange: Exchange:param target_weight_position: {stock_id : weight}:type
|
|
target_weight_position: dict:param risk_degree::type risk_degree: float:param
|
|
pred_start_time::type pred_start_time: pd.Timestamp:param pred_end_time::type
|
|
pred_end_time: pd.Timestamp:param trade_start_time::type trade_start_time: pd.Timestamp:param
|
|
trade_end_time::type trade_end_time: pd.Timestamp:rtype: list'
|
|
function: order_generator.generate_order_list_from_target_weight_position
|
|
- docstring: ' Order Generator With Interact
|
|
|
|
'
|
|
function: order_generator.OrderGenWInteract
|
|
- docstring: ' generate_order_list_from_target_weight_position
|
|
|
|
No adjustment for for the nontradable share.All the tadable value is assigned
|
|
to the tadable stock according to the weight.if interact == True, will use the
|
|
price at trade date to generate order listelse, will only use the price before
|
|
the trade date to generate order list:param current::type current: Position:param
|
|
trade_exchange::type trade_exchange: Exchange:param target_weight_position::type
|
|
target_weight_position: dict:param risk_degree::type risk_degree: float:param
|
|
pred_start_time::type pred_start_time: pd.Timestamp:param pred_end_time::type
|
|
pred_end_time: pd.Timestamp:param trade_start_time::type trade_start_time: pd.Timestamp:param
|
|
trade_end_time::type trade_end_time: pd.Timestamp:rtype: list'
|
|
function: order_generator.generate_order_list_from_target_weight_position
|
|
- docstring: ' Order Generator Without Interact
|
|
|
|
'
|
|
function: order_generator.OrderGenWOInteract
|
|
- docstring: ' generate_order_list_from_target_weight_position
|
|
|
|
generate order list directly not using the information (e.g. whether can be traded,
|
|
the accurate trade price)at trade date.In target weight position, generating order
|
|
list need to know the price of objective stock in trade date,but we cannot get
|
|
thatvalue when do not interact with exchange, so we check the %close price at
|
|
pred_date or price recordedin current position.:param current::type current: Position:param
|
|
trade_exchange::type trade_exchange: Exchange:param target_weight_position::type
|
|
target_weight_position: dict:param risk_degree::type risk_degree: float:param
|
|
pred_start_time::type pred_start_time: pd.Timestamp:param pred_end_time::type
|
|
pred_end_time: pd.Timestamp:param trade_start_time::type trade_start_time: pd.Timestamp:param
|
|
trade_end_time::type trade_end_time: pd.Timestamp:rtype: list of generated orders'
|
|
function: order_generator.generate_order_list_from_target_weight_position
|
|
- docstring: " \nParameters-----------signal :the information to describe a\
|
|
\ signal. Please refer to the docs of `qlib.backtest.signal.create_signal_from`the\
|
|
\ decision of the strategy will base on the given signalrisk_degree : floatposition\
|
|
\ percentage of total value.trade_exchange : Exchangeexchange that provides market\
|
|
\ info, used to deal order and generate report- If `trade_exchange` is None, self.trade_exchange\
|
|
\ will be set with common_infra- It allowes different trade_exchanges is used\
|
|
\ in different executions.- For example:- In daily execution, both daily exchange\
|
|
\ and minutely are usable, but the daily exchange is recommended because it runs\
|
|
\ faster.- In minutely execution, the daily exchange is not usable, only the minutely\
|
|
\ exchange is recommended."
|
|
function: signal_strategy.BaseSignalStrategy
|
|
- docstring: ' get_risk_degree
|
|
|
|
Return the proportion of your total value you will use in investment.Dynamically
|
|
risk_degree will result in Market timing.'
|
|
function: signal_strategy.get_risk_degree
|
|
- docstring: " \nParameters-----------topk : intthe number of stocks in the\
|
|
\ portfolio.n_drop : intnumber of stocks to be replaced in each trading date.method_sell\
|
|
\ : strdropout method_sell, random/bottom.method_buy : strdropout method_buy,\
|
|
\ random/top.hold_thresh : intminimum holding daysbefore sell stock , will check\
|
|
\ current.get_stock_count(order.stock_id) >= self.hold_thresh.only_tradable :\
|
|
\ boolwill the strategy only consider the tradable stock when buying and selling.if\
|
|
\ only_tradable:strategy will make decision with the tradable state of the stock\
|
|
\ info and avoid buy and sell them.else:strategy will make buy sell decision without\
|
|
\ checking the tradable state of the stock.forbid_all_trade_at_limit : boolif\
|
|
\ forbid all trades when limit_up or limit_down reached.if forbid_all_trade_at_limit:strategy\
|
|
\ will not do any trade when price reaches limit up/down, even not sell at limit\
|
|
\ up nor buy atlimit down, though allowed in reality.else:strategy will sell at\
|
|
\ limit up and buy ad limit down."
|
|
function: signal_strategy.TopkDropoutStrategy
|
|
- docstring: null
|
|
function: signal_strategy.generate_trade_decision
|
|
- docstring: null
|
|
function: signal_strategy.get_first_n
|
|
- docstring: null
|
|
function: signal_strategy.get_last_n
|
|
- docstring: null
|
|
function: signal_strategy.filter_stock
|
|
- docstring: null
|
|
function: signal_strategy.get_first_n
|
|
- docstring: null
|
|
function: signal_strategy.get_last_n
|
|
- docstring: null
|
|
function: signal_strategy.filter_stock
|
|
- docstring: " \nsignal :the information to describe a signal. Please refer\
|
|
\ to the docs of `qlib.backtest.signal.create_signal_from`the decision of the\
|
|
\ strategy will base on the given signaltrade_exchange : Exchangeexchange that\
|
|
\ provides market info, used to deal order and generate report- If `trade_exchange`\
|
|
\ is None, self.trade_exchange will be set with common_infra- It allowes different\
|
|
\ trade_exchanges is used in different executions.- For example:- In daily execution,\
|
|
\ both daily exchange and minutely are usable, but the daily exchange is recommended\
|
|
\ because it runs faster.- In minutely execution, the daily exchange is not usable,\
|
|
\ only the minutely exchange is recommended."
|
|
function: signal_strategy.WeightStrategyBase
|
|
- docstring: " \nGenerate target position from score for this date and the\
|
|
\ current position.The cash is not considered in the positionParameters-----------score\
|
|
\ : pd.Seriespred score for this trade date, index is stock_id, contain 'score'\
|
|
\ column.current : Position()current position.trade_start_time: pd.Timestamptrade_end_time:\
|
|
\ pd.Timestamp"
|
|
function: signal_strategy.generate_target_weight_position
|
|
- docstring: null
|
|
function: signal_strategy.generate_trade_decision
|
|
- docstring: " Enhanced Indexing Strategy\nEnhanced indexing combines the arts\
|
|
\ of active management and passive management,with the aim of outperforming a\
|
|
\ benchmark index (e.g., S&P 500) in terms ofportfolio return while controlling\
|
|
\ the risk exposure (a.k.a. tracking error).Users need to prepare their risk model\
|
|
\ data like below:.. code-block:: text\u251C\u2500\u2500 /path/to/riskmodel\u251C\
|
|
\u2500\u2500\u2500\u2500 20210101\u251C\u2500\u2500\u2500\u2500\u2500\u2500 factor_exp.{csv|pkl|h5}\u251C\
|
|
\u2500\u2500\u2500\u2500\u2500\u2500 factor_cov.{csv|pkl|h5}\u251C\u2500\u2500\
|
|
\u2500\u2500\u2500\u2500 specific_risk.{csv|pkl|h5}\u251C\u2500\u2500\u2500\u2500\
|
|
\u2500\u2500 blacklist.{csv|pkl|h5} # optionalThe risk model data can be obtained\
|
|
\ from risk data provider. You can also use`qlib.model.riskmodel.structured.StructuredCovEstimator`\
|
|
\ to prepare these data.Args:riskmodel_path (str): risk model pathname_mapping\
|
|
\ (dict): alternative file names"
|
|
function: signal_strategy.EnhancedIndexingStrategy
|
|
- docstring: null
|
|
function: signal_strategy.get_risk_data
|
|
- docstring: ' TWAP Strategy for trading
|
|
|
|
NOTE:- This TWAP strategy will celling round when trading. This will make the
|
|
TWAP trading strategy produce the orderearlier when the total trade unit of amount
|
|
is less than the trading step'
|
|
function: rule_strategy.TWAPStrategy
|
|
- docstring: " \nParameters----------outer_trade_decision : BaseTradeDecision,\
|
|
\ optional"
|
|
function: rule_strategy.reset
|
|
- docstring: null
|
|
function: rule_strategy.generate_trade_decision
|
|
- docstring: " \n(S)elect the (B)etter one among every two adjacent trading (B)ars\
|
|
\ to sell or buy."
|
|
function: rule_strategy.SBBStrategyBase
|
|
- docstring: " \nParameters----------outer_trade_decision : BaseTradeDecision,\
|
|
\ optional"
|
|
function: rule_strategy.reset
|
|
- docstring: null
|
|
function: rule_strategy._pred_price_trend
|
|
- docstring: null
|
|
function: rule_strategy.generate_trade_decision
|
|
- docstring: " \n(S)elect the (B)etter one among every two adjacent trading (B)ars\
|
|
\ to sell or buy with (EMA) signal."
|
|
function: rule_strategy.SBBStrategyEMA
|
|
- docstring: null
|
|
function: rule_strategy._reset_signal
|
|
- docstring: " \nreset level-shared infra- After reset the trade calendar,\
|
|
\ the signal will be changed"
|
|
function: rule_strategy.reset_level_infra
|
|
- docstring: null
|
|
function: rule_strategy._pred_price_trend
|
|
- docstring: " \nParameters----------instruments : Union[List, str], optionalinstruments\
|
|
\ of Volatility, by default \"csi300\"freq : str, optionalfreq of Volatility,\
|
|
\ by default \"day\"Note: `freq` may be different from `time_per_step`"
|
|
function: rule_strategy.ACStrategy
|
|
- docstring: null
|
|
function: rule_strategy._reset_signal
|
|
- docstring: " \nreset level-shared infra- After reset the trade calendar,\
|
|
\ the signal will be changed"
|
|
function: rule_strategy.reset_level_infra
|
|
- docstring: " \nParameters----------outer_trade_decision : BaseTradeDecision,\
|
|
\ optional"
|
|
function: rule_strategy.reset
|
|
- docstring: null
|
|
function: rule_strategy.generate_trade_decision
|
|
- docstring: " \nParameters----------trade_range : Tupleplease refer to the\
|
|
\ `trade_range` parameter of BaseStrategysample_ratio : floatthe ratio of all\
|
|
\ orders are sampledvolume_ratio : floatthe volume of the total dayraito of the\
|
|
\ total volume of a specific daymarket : strstock pool for sampling"
|
|
function: rule_strategy.RandomOrderStrategy
|
|
- docstring: null
|
|
function: rule_strategy.generate_trade_decision
|
|
- docstring: " \nMotivation:- This class provides an interface for user to read\
|
|
\ orders from csv files."
|
|
function: rule_strategy.FileOrderStrategy
|
|
- docstring: " \nParameters----------execute_result :execute_result will be\
|
|
\ ignored in FileOrderStrategy"
|
|
function: rule_strategy.generate_trade_decision
|
|
- docstring: " \nParameters----------topk : inttop-N stocks to buyrisk_degree\
|
|
\ : floatposition percentage of total value buy_method:rank_fill: assign the weight\
|
|
\ stocks that rank high first(1/topk max)average_fill: assign the weight to the\
|
|
\ stocks rank high averagely."
|
|
function: cost_control.SoftTopkStrategy
|
|
- docstring: ' get_risk_degree
|
|
|
|
Return the proportion of your total value you will used in investment.Dynamically
|
|
risk_degree will result in Market timing'
|
|
function: cost_control.get_risk_degree
|
|
- docstring: " \nParameters----------score:pred score for this trade date,\
|
|
\ pd.Series, index is stock_id, contain 'score' columncurrent:current position,\
|
|
\ use Position() classtrade_date:trade dategenerate target position from score\
|
|
\ for this date and the current positionThe cache is not considered in the position"
|
|
function: cost_control.generate_target_weight_position
|
|
- docstring: ' Portfolio Optimizer
|
|
|
|
The following optimization algorithms are supported:- `gmv`: Global Minimum Variance
|
|
Portfolio- `mvo`: Mean Variance Optimized Portfolio- `rp`: Risk Parity- `inv`:
|
|
Inverse VolatilityNote:This optimizer always assumes full investment and no-shorting.'
|
|
function: optimizer.PortfolioOptimizer
|
|
- docstring: null
|
|
function: optimizer._optimize
|
|
- docstring: ' Inverse volatility
|
|
|
|
vola = np.diag(S) ** 0.5w = 1 / volaw /= w.sum()return w'
|
|
function: optimizer._optimize_inv
|
|
- docstring: ' optimize global minimum variance portfolio
|
|
|
|
This method solves the following optimization problemmin_w w'' S ws.t. w >= 0,
|
|
sum(w) == 1where `S` is the covariance matrix.'
|
|
function: optimizer._optimize_gmv
|
|
- docstring: ' optimize mean-variance portfolio
|
|
|
|
This method solves the following optimization problemmin_w - w'' r + lamb *
|
|
w'' S ws.t. w >= 0, sum(w) == 1where `S` is the covariance matrix, `u` is the
|
|
expected returns,and `lamb` is the risk aversion parameter.'
|
|
function: optimizer._optimize_mvo
|
|
- docstring: ' optimize risk parity portfolio
|
|
|
|
This method solves the following optimization problemmin_w sum_i [w_i - (w'' S
|
|
w) / ((S w)_i * N)]**2s.t. w >= 0, sum(w) == 1where `S` is the covariance matrix
|
|
and `N` is the number of stocks.'
|
|
function: optimizer._optimize_rp
|
|
- docstring: ' global minimum variance optimization objective
|
|
|
|
Optimization objectivemin_w w'' S w'
|
|
function: optimizer._get_objective_gmv
|
|
- docstring: null
|
|
function: optimizer.func
|
|
- docstring: ' mean-variance optimization objective
|
|
|
|
Optimization objectivemin_w - w'' r + lamb * w'' S w'
|
|
function: optimizer._get_objective_mvo
|
|
- docstring: null
|
|
function: optimizer.func
|
|
- docstring: ' risk-parity optimization objective
|
|
|
|
Optimization objectivemin_w sum_i [w_i - (w'' S w) / ((S w)_i * N)]**2'
|
|
function: optimizer._get_objective_rp
|
|
- docstring: null
|
|
function: optimizer.func
|
|
- docstring: ' optimization constraints
|
|
|
|
Defines the following constraints:- no shorting and leverage: 0 <= w <= 1- full
|
|
investment: sum(w) == 1- turnover constraint: |w - w0| <= delta'
|
|
function: optimizer._get_constrains
|
|
- docstring: ' solve optimization
|
|
|
|
Args:n (int): number of parametersobj (callable): optimization objectivebounds
|
|
(Bounds): bounds of parameterscons (list): optimization constraints'
|
|
function: optimizer._solve
|
|
- docstring: ' Construct portfolio with a optimization related method
|
|
|
|
@abc.abstractmethodGenerate a optimized portfolio allocation'
|
|
function: base.BaseOptimizer
|
|
- docstring: " \nPortfolio Optimizer for Enhanced IndexingNotations:w0: current\
|
|
\ holding weightswb: benchmark weightr: expected returnF: factor exposurecov_b:\
|
|
\ factor covariancevar_u: residual variance (diagonal)lamb: risk aversion parameterdelta:\
|
|
\ total turnover limitb_dev: benchmark deviation limitf_dev: factor deviation\
|
|
\ limitAlso denote:d = w - wb: benchmark deviationv = d @ F: factor deviationThe\
|
|
\ optimization problem for enhanced indexing:max_w d @ r - lamb * (v @ cov_b\
|
|
\ @ v + var_u @ d**2)s.t. w >= 0sum(w) == 1sum(|w - w0|) <= deltad >= -b_devd\
|
|
\ <= b_devv >= -f_devv <= f_dev"
|
|
function: enhanced_indexing.EnhancedIndexingOptimizer
|
|
- docstring: ' sub_fig_generator.
|
|
|
|
it will return a generator, each row contains <col_n> sub graphFIXME: Known limitation:-
|
|
The last row will not be plotted automatically, please plot it outside the functionParameters----------sub_fs
|
|
:the figure size of each subgraph in <col_n> * <row_n> subgraphscol_n :the number
|
|
of subgraph in each row; It will generating a new graph after generating <col_n>
|
|
of subgraphs.row_n :the number of subgraph in each columnwspace :the width of
|
|
the space for subgraphs in each rowhspace :the height of blank space for subgraphs
|
|
in each columnYou can try 0.3 if you feel it is too crowdedReturns-------It will
|
|
return graphs with the shape of <col_n> each iter (it is squeezed).'
|
|
function: utils.sub_fig_generator
|
|
- docstring: " \nThis function `guesses` the rangebreaks required to remove gaps\
|
|
\ in datetime index.It basically calculates the difference between a `continuous`\
|
|
\ datetime index and index given.For more details on `rangebreaks` params in plotly,\
|
|
\ seehttps://plotly.com/python/reference/layout/xaxis/#layout-xaxis-rangebreaksParameters----------dt_index:\
|
|
\ pd.DatetimeIndexThe datetimes of the data.Returns-------the `rangebreaks` to\
|
|
\ be passed into plotly axis."
|
|
function: utils.guess_plotly_rangebreaks
|
|
- docstring: " \n:param df::param layout::param graph_kwargs::param name_dict::param\
|
|
\ kwargs:layout: dictgo.Layout parametersgraph_kwargs: dictGraph parameters, eg:\
|
|
\ go.Bar(**graph_kwargs)"
|
|
function: 'graph.BaseGraph:'
|
|
- docstring: " \n:return:"
|
|
function: graph._init_data
|
|
- docstring: " \n:param kwargs"
|
|
function: graph._init_parameters
|
|
- docstring: " \n:param graph_type::param kwargs::return:"
|
|
function: graph.get_instance_with_graph_parameters
|
|
- docstring: " \n:param figure_list::return:"
|
|
function: graph.show_graph_in_notebook
|
|
- docstring: " \n:return:"
|
|
function: graph._get_layout
|
|
- docstring: " \n:return:"
|
|
function: graph._get_data
|
|
- docstring: " \n:return:"
|
|
function: graph.figure
|
|
- docstring: null
|
|
function: graph.ScatterGraph
|
|
- docstring: null
|
|
function: graph.BarGraph
|
|
- docstring: null
|
|
function: graph.DistplotGraph
|
|
- docstring: " \n:return:"
|
|
function: graph._get_data
|
|
- docstring: null
|
|
function: graph.HeatmapGraph
|
|
- docstring: " \n:return:"
|
|
function: graph._get_data
|
|
- docstring: null
|
|
function: graph.HistogramGraph
|
|
- docstring: " \n:return:"
|
|
function: graph._get_data
|
|
- docstring: ' Create subplots same as df.plot(subplots=True)
|
|
|
|
Simple package for `plotly.tools.subplots`'
|
|
function: 'graph.SubplotsGraph:'
|
|
- docstring: " \n:return:"
|
|
function: graph._init_sub_graph_data
|
|
- docstring: " \n:return:"
|
|
function: graph._init_subplots_kwargs
|
|
- docstring: " \n:return:"
|
|
function: graph._init_figure
|
|
- docstring: null
|
|
function: 'base.FeaAnalyser:'
|
|
- docstring: null
|
|
function: base.calc_stat_values
|
|
- docstring: null
|
|
function: base.plot_single
|
|
- docstring: null
|
|
function: base.skip
|
|
- docstring: " \nCombine the sub feature analysers and plot then in a single graph"
|
|
function: ana.CombFeaAna
|
|
- docstring: null
|
|
function: ana.skip
|
|
- docstring: ' The statistics of features are finished in the underlying analysers
|
|
|
|
'
|
|
function: ana.calc_stat_values
|
|
- docstring: null
|
|
function: ana.plot_all
|
|
- docstring: null
|
|
function: ana.NumFeaAnalyser
|
|
- docstring: null
|
|
function: ana.skip
|
|
- docstring: null
|
|
function: ana.ValueCNT
|
|
- docstring: null
|
|
function: ana.calc_stat_values
|
|
- docstring: null
|
|
function: ana.plot_single
|
|
- docstring: null
|
|
function: ana.FeaDistAna
|
|
- docstring: null
|
|
function: ana.plot_single
|
|
- docstring: null
|
|
function: ana.FeaInfAna
|
|
- docstring: null
|
|
function: ana.calc_stat_values
|
|
- docstring: null
|
|
function: ana.skip
|
|
- docstring: null
|
|
function: ana.plot_single
|
|
- docstring: null
|
|
function: ana.FeaNanAna
|
|
- docstring: null
|
|
function: ana.calc_stat_values
|
|
- docstring: null
|
|
function: ana.skip
|
|
- docstring: null
|
|
function: ana.plot_single
|
|
- docstring: null
|
|
function: ana.FeaNanAnaRatio
|
|
- docstring: null
|
|
function: ana.calc_stat_values
|
|
- docstring: null
|
|
function: ana.skip
|
|
- docstring: null
|
|
function: ana.plot_single
|
|
- docstring: ' Analysis the auto-correlation of features
|
|
|
|
'
|
|
function: ana.FeaACAna
|
|
- docstring: null
|
|
function: ana.calc_stat_values
|
|
- docstring: null
|
|
function: ana.plot_single
|
|
- docstring: null
|
|
function: ana.FeaSkewTurt
|
|
- docstring: null
|
|
function: ana.calc_stat_values
|
|
- docstring: null
|
|
function: ana.plot_single
|
|
- docstring: null
|
|
function: ana.FeaMeanStd
|
|
- docstring: null
|
|
function: ana.calc_stat_values
|
|
- docstring: null
|
|
function: ana.plot_single
|
|
- docstring: " \nMotivation:- display the values without further analysis"
|
|
function: ana.RawFeaAna
|
|
- docstring: null
|
|
function: ana.calc_stat_values
|
|
- docstring: " \n:param pred_label::param reverse::param N::return:"
|
|
function: analysis_model_performance._group_return
|
|
- docstring: " \n:param data::param dist::return:"
|
|
function: analysis_model_performance._plot_qq
|
|
- docstring: " \n:param pred_label: pd.DataFramemust contain one column of realized\
|
|
\ return with name `label` and one column of predicted score names `score`.:param\
|
|
\ methods: Sequence[Literal[\"IC\", \"Rank IC\"]]IC series to plot.IC is sectional\
|
|
\ pearson correlation between label and scoreRank IC is the spearman correlation\
|
|
\ between label and scoreFor the Monthly IC, IC histogram, IC Q-Q plot. Only\
|
|
\ the first type of IC will be plotted.:return:"
|
|
function: analysis_model_performance._pred_ic
|
|
- docstring: null
|
|
function: analysis_model_performance._corr_series
|
|
- docstring: null
|
|
function: analysis_model_performance._pred_autocorr
|
|
- docstring: null
|
|
function: analysis_model_performance._pred_turnover
|
|
- docstring: " \nif show_nature_day:date_index = pd.date_range(ic_df.index.min(),\
|
|
\ ic_df.index.max())ic_df = ic_df.reindex(date_index)ic_bar_figure = BarGraph(ic_df,layout=dict(title=\"\
|
|
Information Coefficient (IC)\",xaxis=dict(tickangle=45, rangebreaks=kwargs.get(\"\
|
|
rangebreaks\", guess_plotly_rangebreaks(ic_df.index))),),).figurereturn ic_bar_figure"
|
|
function: analysis_model_performance.ic_figure
|
|
- docstring: ' Parse position dict to position DataFrame
|
|
|
|
:param position: position data:return: position DataFrame;.. code-block:: pythonposition_df
|
|
= parse_position(positions)print(position_df.head())# status: 0-hold, -1-sell,
|
|
1-buyamount cash count price status weightinstrument datetimeSZ000547 2017-01-04 44.154290 211405.285654 1 205.189575 1 0.031255SZ300202 2017-01-04 60.638845 211405.285654 1 154.356506 1 0.032290SH600158 2017-01-04 46.531681 211405.285654 1 153.895142 1 0.024704SH600545 2017-01-04 197.173093 211405.285654 1 48.607037 1 0.033063SZ000930 2017-01-04 103.938300 211405.285654 1 80.759453 1 0.028958'
|
|
function: parse_position.parse_position
|
|
- docstring: ' Concat position with custom label
|
|
|
|
:param position_df: position DataFrame:param label_data::return: concat result'
|
|
function: parse_position._add_label_to_position
|
|
- docstring: ' Concat position with bench
|
|
|
|
:param position_df: position DataFrame:param bench: report normal data:return:
|
|
concat result'
|
|
function: parse_position._add_bench_to_position
|
|
- docstring: ' calculate label rank
|
|
|
|
:param df::return:'
|
|
function: parse_position._calculate_label_rank
|
|
- docstring: null
|
|
function: parse_position._calculate_day_value
|
|
- docstring: ' Concat position data with pred/report_normal
|
|
|
|
:param position: position data:param report_normal: report normal, must be container
|
|
''bench'' column:param label_data::param calculate_label_rank::param start_date:
|
|
start date:param end_date: end date:return: concat result,columns: [''amount'',
|
|
''cash'', ''count'', ''price'', ''status'', ''weight'', ''label'',''rank_ratio'',
|
|
''rank_label_mean'', ''excess_return'', ''score'', ''bench'']index: [''instrument'',
|
|
''date'']'
|
|
function: parse_position.get_position_data
|
|
- docstring: " \n:param position::param report_normal::param label_data::param\
|
|
\ start_date::param end_date::return:"
|
|
function: cumulative_return._get_cum_return_data_with_position
|
|
- docstring: ' Get average analysis figures
|
|
|
|
:param position: position:param report_normal::param label_data::param start_date::param
|
|
end_date::return:'
|
|
function: cumulative_return._get_figure_with_position
|
|
- docstring: ' Backtest buy, sell, and holding cumulative return graph
|
|
|
|
Example:.. code-block:: pythonfrom qlib.data import Dfrom qlib.contrib.evaluate
|
|
import risk_analysis, backtest, long_short_backtestfrom qlib.contrib.strategy
|
|
import TopkDropoutStrategy# backtest parametersbparas = {}bparas[''limit_threshold'']
|
|
= 0.095bparas[''account''] = 1000000000sparas = {}sparas[''topk''] = 50sparas[''n_drop'']
|
|
= 5strategy = TopkDropoutStrategy(**sparas)report_normal_df, positions = backtest(pred_df,
|
|
strategy, **bparas)pred_df_dates = pred_df.index.get_level_values(level=''datetime'')features_df
|
|
= D.features(D.instruments(''csi500''), [''Ref($close, -1)/$close - 1''], pred_df_dates.min(),
|
|
pred_df_dates.max())features_df.columns = [''label'']qcr.analysis_position.cumulative_return_graph(positions,
|
|
report_normal_df, features_df)Graph desc:- Axis X: Trading day.- Axis Y:- Above
|
|
axis Y: `(((Ref($close, -1)/$close - 1) * weight).sum() / weight.sum()).cumsum()`.-
|
|
Below axis Y: Daily weight sum.- In the **sell** graph, `y < 0` stands for profit;
|
|
in other cases, `y > 0` stands for profit.- In the **buy_minus_sell** graph, the
|
|
**y** value of the **weight** graph at the bottom is `buy_weight + sell_weight`.-
|
|
In each graph, the **red line** in the histogram on the right represents the average.:param
|
|
position: position data:param report_normal:.. code-block:: pythonreturn cost bench turnoverdate2017-01-04 0.003421 0.000864 0.011693 0.5763252017-01-05 0.000508 0.000447 0.000721 0.2278822017-01-06 -0.003321 0.000212 -0.004322 0.1027652017-01-09 0.006753 0.000212 0.006874 0.1058642017-01-10 -0.000416 0.000440 -0.003350 0.208396:param
|
|
label_data: `D.features` result; index is `pd.MultiIndex`, index name is [`instrument`,
|
|
`datetime`]; columns names is [`label`].**The label T is the change from T to
|
|
T+1**, it is recommended to use ``close``, example: `D.features(D.instruments(''csi500''),
|
|
[''Ref($close, -1)/$close-1''])`.. code-block:: pythonlabelinstrument datetimeSH600004 2017-12-11 -0.0135022017-12-12 -0.0723672017-12-13 -0.0686052017-12-14 0.0124402017-12-15 -0.102778:param
|
|
show_notebook: True or False. If True, show graph in notebook, else return figures:param
|
|
start_date: start date:param end_date: end date:return:'
|
|
function: cumulative_return.cumulative_return_graph
|
|
- docstring: ' Get average analysis figures
|
|
|
|
:param position: position:param label_data::param start_date::param end_date::return:'
|
|
function: rank_label._get_figure_with_position
|
|
- docstring: ' Ranking percentage of stocks buy, sell, and holding on the trading
|
|
day.
|
|
|
|
Average rank-ratio(similar to **sell_df[''label''].rank(ascending=False) / len(sell_df)**)
|
|
of daily tradingExample:.. code-block:: pythonfrom qlib.data import Dfrom qlib.contrib.evaluate
|
|
import backtestfrom qlib.contrib.strategy import TopkDropoutStrategy# backtest
|
|
parametersbparas = {}bparas[''limit_threshold''] = 0.095bparas[''account''] =
|
|
1000000000sparas = {}sparas[''topk''] = 50sparas[''n_drop''] = 230strategy = TopkDropoutStrategy(**sparas)_,
|
|
positions = backtest(pred_df, strategy, **bparas)pred_df_dates = pred_df.index.get_level_values(level=''datetime'')features_df
|
|
= D.features(D.instruments(''csi500''), [''Ref($close, -1)/$close-1''], pred_df_dates.min(),
|
|
pred_df_dates.max())features_df.columns = [''label'']qcr.analysis_position.rank_label_graph(positions,
|
|
features_df, pred_df_dates.min(), pred_df_dates.max()):param position: position
|
|
data; **qlib.backtest.backtest** result.:param label_data: **D.features** result;
|
|
index is **pd.MultiIndex**, index name is **[instrument, datetime]**; columns
|
|
names is **[label]**.**The label T is the change from T to T+1**, it is recommended
|
|
to use ``close``, example: `D.features(D.instruments(''csi500''), [''Ref($close,
|
|
-1)/$close-1''])`... code-block:: pythonlabelinstrument datetimeSH600004 2017-12-11 -0.0135022017-12-12 -0.0723672017-12-13 -0.0686052017-12-14 0.0124402017-12-15 -0.102778:param
|
|
start_date: start date:param end_date: end_date:param show_notebook: **True**
|
|
or **False**. If True, show graph in notebook, else return figures.:return:'
|
|
function: rank_label.rank_label_graph
|
|
- docstring: " \n:param df::param is_ex::return:"
|
|
function: report._calculate_maximum
|
|
- docstring: " \nCalculate mdd:param series::return:"
|
|
function: report._calculate_mdd
|
|
- docstring: " \n:param df::return:"
|
|
function: report._calculate_report_data
|
|
- docstring: " \n:param df::return:"
|
|
function: report._report_figure
|
|
- docstring: ' display backtest report
|
|
|
|
Example:.. code-block:: pythonimport qlibimport pandas as pdfrom qlib.utils.time
|
|
import Freqfrom qlib.utils import flatten_dictfrom qlib.backtest import backtest,
|
|
executorfrom qlib.contrib.evaluate import risk_analysisfrom qlib.contrib.strategy
|
|
import TopkDropoutStrategy# init qlibqlib.init(provider_uri=<qlib data dir>)CSI300_BENCH
|
|
= "SH000300"FREQ = "day"STRATEGY_CONFIG = {"topk": 50,"n_drop": 5,# pred_score,
|
|
pd.Series"signal": pred_score,}EXECUTOR_CONFIG = {"time_per_step": "day","generate_portfolio_metrics":
|
|
True,}backtest_config = {"start_time": "2017-01-01","end_time": "2020-08-01","account":
|
|
100000000,"benchmark": CSI300_BENCH,"exchange_kwargs": {"freq": FREQ,"limit_threshold":
|
|
0.095,"deal_price": "close","open_cost": 0.0005,"close_cost": 0.0015,"min_cost":
|
|
5,},}# strategy objectstrategy_obj = TopkDropoutStrategy(**STRATEGY_CONFIG)# executor
|
|
objectexecutor_obj = executor.SimulatorExecutor(**EXECUTOR_CONFIG)# backtestportfolio_metric_dict,
|
|
indicator_dict = backtest(executor=executor_obj, strategy=strategy_obj, **backtest_config)analysis_freq
|
|
= "{0}{1}".format(*Freq.parse(FREQ))# backtest inforeport_normal_df, positions_normal
|
|
= portfolio_metric_dict.get(analysis_freq)qcr.analysis_position.report_graph(report_normal_df):param
|
|
report_df: **df.index.name** must be **date**, **df.columns** must contain **return**,
|
|
**turnover**, **cost**, **bench**... code-block:: pythonreturn cost bench turnoverdate2017-01-04 0.003421 0.000864 0.011693 0.5763252017-01-05 0.000508 0.000447 0.000721 0.2278822017-01-06 -0.003321 0.000212 -0.004322 0.1027652017-01-09 0.006753 0.000212 0.006874 0.1058642017-01-10 -0.000416 0.000440 -0.003350 0.208396:param
|
|
show_notebook: whether to display graphics in notebook, the default is **True**.:return:
|
|
if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure**
|
|
list.'
|
|
function: report.report_graph
|
|
- docstring: ' Get risk analysis data with report
|
|
|
|
:param report_normal_df: report data:param report_long_short_df: report data:param
|
|
date: date string:return:'
|
|
function: risk_analysis._get_risk_analysis_data_with_report
|
|
- docstring: ' risk_df to standard
|
|
|
|
:param risk_df: risk data:return:'
|
|
function: risk_analysis._get_all_risk_analysis
|
|
- docstring: ' Get monthly analysis data
|
|
|
|
:param report_normal_df:# :param report_long_short_df::return:'
|
|
function: risk_analysis._get_monthly_risk_analysis_with_report
|
|
- docstring: " \n:param monthly_df::param feature::return:"
|
|
function: risk_analysis._get_monthly_analysis_with_feature
|
|
- docstring: ' Get analysis graph figure
|
|
|
|
:param analysis_df::return:'
|
|
function: risk_analysis._get_risk_analysis_figure
|
|
- docstring: ' Get analysis monthly graph figure
|
|
|
|
:param report_normal_df::param report_long_short_df::return:'
|
|
function: risk_analysis._get_monthly_risk_analysis_figure
|
|
- docstring: ' Generate analysis graph and monthly analysis
|
|
|
|
Example:.. code-block:: pythonimport qlibimport pandas as pdfrom qlib.utils.time
|
|
import Freqfrom qlib.utils import flatten_dictfrom qlib.backtest import backtest,
|
|
executorfrom qlib.contrib.evaluate import risk_analysisfrom qlib.contrib.strategy
|
|
import TopkDropoutStrategy# init qlibqlib.init(provider_uri=<qlib data dir>)CSI300_BENCH
|
|
= "SH000300"FREQ = "day"STRATEGY_CONFIG = {"topk": 50,"n_drop": 5,# pred_score,
|
|
pd.Series"signal": pred_score,}EXECUTOR_CONFIG = {"time_per_step": "day","generate_portfolio_metrics":
|
|
True,}backtest_config = {"start_time": "2017-01-01","end_time": "2020-08-01","account":
|
|
100000000,"benchmark": CSI300_BENCH,"exchange_kwargs": {"freq": FREQ,"limit_threshold":
|
|
0.095,"deal_price": "close","open_cost": 0.0005,"close_cost": 0.0015,"min_cost":
|
|
5,},}# strategy objectstrategy_obj = TopkDropoutStrategy(**STRATEGY_CONFIG)# executor
|
|
objectexecutor_obj = executor.SimulatorExecutor(**EXECUTOR_CONFIG)# backtestportfolio_metric_dict,
|
|
indicator_dict = backtest(executor=executor_obj, strategy=strategy_obj, **backtest_config)analysis_freq
|
|
= "{0}{1}".format(*Freq.parse(FREQ))# backtest inforeport_normal_df, positions_normal
|
|
= portfolio_metric_dict.get(analysis_freq)analysis = dict()analysis["excess_return_without_cost"]
|
|
= risk_analysis(report_normal_df["return"] - report_normal_df["bench"], freq=analysis_freq)analysis["excess_return_with_cost"]
|
|
= risk_analysis(report_normal_df["return"] - report_normal_df["bench"] - report_normal_df["cost"],
|
|
freq=analysis_freq)analysis_df = pd.concat(analysis) # type: pd.DataFrameanalysis_position.risk_analysis_graph(analysis_df,
|
|
report_normal_df):param analysis_df: analysis data, index is **pd.MultiIndex**;
|
|
columns names is **[risk]**... code-block:: pythonriskexcess_return_without_cost
|
|
mean 0.000692std 0.005374annualized_return 0.174495information_ratio 2.045576max_drawdown -0.079103excess_return_with_cost mean 0.000499std 0.005372annualized_return 0.125625information_ratio 1.473152max_drawdown -0.088263:param
|
|
report_normal_df: **df.index.name** must be **date**, df.columns must contain
|
|
**return**, **turnover**, **cost**, **bench**... code-block:: pythonreturn cost bench turnoverdate2017-01-04 0.003421 0.000864 0.011693 0.5763252017-01-05 0.000508 0.000447 0.000721 0.2278822017-01-06 -0.003321 0.000212 -0.004322 0.1027652017-01-09 0.006753 0.000212 0.006874 0.1058642017-01-10 -0.000416 0.000440 -0.003350 0.208396:param
|
|
report_long_short_df: **df.index.name** must be **date**, df.columns contain **long**,
|
|
**short**, **long_short**... code-block:: pythonlong short long_shortdate2017-01-04 -0.001360 0.001394 0.0000342017-01-05 0.002456 0.000058 0.0025142017-01-06 0.000120 0.002739 0.0028592017-01-09 0.001436 0.001838 0.0032732017-01-10 0.000824 -0.001944 -0.001120:param
|
|
show_notebook: Whether to display graphics in a notebook, default **True**.If
|
|
True, show graph in notebookIf False, return graph figure:return:'
|
|
function: risk_analysis.risk_analysis_graph
|
|
- docstring: " \n:param pred_label::return:"
|
|
function: score_ic._get_score_ic
|
|
- docstring: ' score IC
|
|
|
|
Example:.. code-block:: pythonfrom qlib.data import Dfrom qlib.contrib.report
|
|
import analysis_positionpred_df_dates = pred_df.index.get_level_values(level=''datetime'')features_df
|
|
= D.features(D.instruments(''csi500''), [''Ref($close, -2)/Ref($close, -1)-1''],
|
|
pred_df_dates.min(), pred_df_dates.max())features_df.columns = [''label'']pred_label
|
|
= pd.concat([features_df, pred], axis=1, sort=True).reindex(features_df.index)analysis_position.score_ic_graph(pred_label):param
|
|
pred_label: index is **pd.MultiIndex**, index name is **[instrument, datetime]**;
|
|
columns names is **[score, label]**... code-block:: pythoninstrument datetime score labelSH600004 2017-12-11 -0.013502 -0.0135022017-12-12 -0.072367 -0.0723672017-12-13 -0.068605 -0.0686052017-12-14 0.012440 0.0124402017-12-15 -0.102778 -0.102778:param
|
|
show_notebook: whether to display graphics in notebook, the default is **True**.:return:
|
|
if show_notebook is True, display in notebook; else return **plotly.graph_objs.Figure**
|
|
list.'
|
|
function: score_ic.score_ic_graph
|
|
- docstring: " \ncalculate the precision for long and short operation:param pred/label:\
|
|
\ index is **pd.MultiIndex**, index name is **[datetime, instruments]**; columns\
|
|
\ names is **[score]**... code-block:: pythonscoredatetime instrument2020-12-01\
|
|
\ 09:30:00 SH600068 0.553634SH600195 0.550017SH600276 0.540321SH600584\
|
|
\ 0.517297SH600715 0.544674label :labeldate_col :date_colReturns-------(pd.Series,\
|
|
\ pd.Series)long precision and short precision in time level"
|
|
function: alpha.calc_long_short_prec
|
|
- docstring: null
|
|
function: alpha.N
|
|
- docstring: " \ncalculate long-short returnNote:`label` must be raw stock returns.Parameters----------pred\
|
|
\ : pd.Seriesstock predictionslabel : pd.Seriesstock returnsdate_col : strdatetime\
|
|
\ index namequantile : floatlong-short quantileReturns----------long_short_r :\
|
|
\ pd.Seriesdaily long-short returnslong_avg_r : pd.Seriesdaily long-average returns"
|
|
function: alpha.calc_long_short_return
|
|
- docstring: null
|
|
function: alpha.N
|
|
- docstring: ' pred_autocorr.
|
|
|
|
Limitation:- If the datetime is not sequential densely, the correlation will be
|
|
calulated based on adjacent dates. (some users may expected NaN):param pred: pd.Series
|
|
with following formatinstrument datetimeSH600000 2016-01-04 -0.0004032016-01-05 -0.0007532016-01-06 -0.0218012016-01-07 -0.0652302016-01-08 -0.062465:type
|
|
pred: pd.Series:param lag:'
|
|
function: alpha.pred_autocorr
|
|
- docstring: " \ncalculate auto correlation for pred_dictParameters----------pred_dict\
|
|
\ : dictA dict like {<method_name>: <prediction>}kwargs :all these arguments\
|
|
\ will be passed into pred_autocorr"
|
|
function: alpha.pred_autocorr_all
|
|
- docstring: ' calc_ic.
|
|
|
|
Parameters----------pred :predlabel :labeldate_col :date_colReturns-------(pd.Series,
|
|
pd.Series)ic and rank ic'
|
|
function: alpha.calc_ic
|
|
- docstring: ' calc_all_ic.
|
|
|
|
Parameters----------pred_dict_all :A dict like {<method_name>: <prediction>}label:A
|
|
pd.Series of label valuesReturns-------{''Q2+IND_z'': {''ic'': <ic series like>2016-01-04 -0.057407...2020-05-28 0.1834702020-05-29 0.171393''ric'':
|
|
<rank ic series like>2016-01-04 -0.040888...2020-05-28 0.2366652020-05-29 0.183886}...}'
|
|
function: alpha.calc_all_ic
|
|
- docstring: null
|
|
function: handler.check_transform_proc
|
|
- docstring: null
|
|
function: handler.Alpha360
|
|
- docstring: null
|
|
function: handler.get_label_config
|
|
- docstring: null
|
|
function: handler.get_feature_config
|
|
- docstring: null
|
|
function: handler.Alpha360vwap
|
|
- docstring: null
|
|
function: handler.get_label_config
|
|
- docstring: null
|
|
function: handler.Alpha158
|
|
- docstring: null
|
|
function: handler.get_feature_config
|
|
- docstring: null
|
|
function: handler.get_label_config
|
|
- docstring: ' create factors from config
|
|
|
|
config = {''kbar'': {}, # whether to use some hard-code kbar features''price'':
|
|
{ # whether to use raw price features''windows'': [0, 1, 2, 3, 4], # use price
|
|
at n days ago''feature'': [''OPEN'', ''HIGH'', ''LOW''] # which price field to
|
|
use},''volume'': { # whether to use raw volume features''windows'': [0, 1, 2,
|
|
3, 4], # use volume at n days ago},''rolling'': { # whether to use rolling operator
|
|
based features''windows'': [5, 10, 20, 30, 60], # rolling windows size''include'':
|
|
[''ROC'', ''MA'', ''STD''], # rolling operator to use#if include is None we will
|
|
use default operators''exclude'': [''RANK''], # rolling operator not to use}}'
|
|
function: handler.parse_config_to_fields
|
|
- docstring: null
|
|
function: handler.use
|
|
- docstring: null
|
|
function: handler.Alpha158vwap
|
|
- docstring: null
|
|
function: data.ArcticFeatureProvider
|
|
- docstring: null
|
|
function: dataset._to_tensor
|
|
- docstring: " \ncreate time series slices from pandas indexArgs:index (pd.MultiIndex):\
|
|
\ pandas multiindex with <instrument, datetime> orderseq_len (int): sequence length"
|
|
function: dataset._create_ts_slices
|
|
- docstring: ' get date parse function
|
|
|
|
This method is used to parse date arguments as target type.Example:get_date_parse_fn(''20120101'')(''2017-01-01'')
|
|
=> ''20170101''get_date_parse_fn(20120101)(''2017-01-01'') => 20170101'
|
|
function: dataset._get_date_parse_fn
|
|
- docstring: null
|
|
function: dataset._fn
|
|
- docstring: null
|
|
function: dataset._fn
|
|
- docstring: null
|
|
function: dataset._fn
|
|
- docstring: ' padding 2d <time * feature> data with zeros
|
|
|
|
Args:x (np.ndarray): 2d data with shape <time * feature>seq_len (int): target
|
|
sequence lengthzeros (np.ndarray): zeros with shape <seq_len * feature>'
|
|
function: dataset._maybe_padding
|
|
- docstring: ' Memory Augmented Time Series Dataset
|
|
|
|
Args:handler (DataHandler): data handlersegments (dict): data split segmentsseq_len
|
|
(int): time series sequence lengthhorizon (int): label horizonnum_states (int):
|
|
how many memory states to be addedmemory_mode (str): memory mode (daily or sample)batch_size
|
|
(int): batch size (<0 will use daily sampling)n_samples (int): number of samples
|
|
in the same dayshuffle (bool): whether shuffle datadrop_last (bool): whether drop
|
|
last batch < batch_sizeinput_size (int): reshape flatten rows as this input_size
|
|
(backward compatibility)'
|
|
function: dataset.MTSDatasetH
|
|
- docstring: null
|
|
function: dataset.setup_data
|
|
- docstring: null
|
|
function: dataset._prepare_seg
|
|
- docstring: null
|
|
function: dataset.restore_index
|
|
- docstring: null
|
|
function: dataset.restore_daily_index
|
|
- docstring: null
|
|
function: dataset.assign_data
|
|
- docstring: null
|
|
function: dataset.clear_memory
|
|
- docstring: ' enable traning mode
|
|
|
|
self.batch_size, self.n_samples, self.drop_last, self.shuffle = self.params'
|
|
function: dataset.train
|
|
- docstring: ' enable evaluation mode
|
|
|
|
self.batch_size = -1self.n_samples = Noneself.drop_last = Falseself.shuffle =
|
|
False'
|
|
function: dataset.eval
|
|
- docstring: null
|
|
function: 'highfreq_provider.HighFreqProvider:'
|
|
- docstring: ' Generate the training, validation and test datasets for prediction
|
|
|
|
Returns:Tuple[BaseDataset, BaseDataset, BaseDataset]: The training and test datasets'
|
|
function: highfreq_provider.get_pre_datasets
|
|
- docstring: null
|
|
function: highfreq_provider.get_backtest
|
|
- docstring: ' initialize qlib
|
|
|
|
qlib.init(region=REG_CN,auto_mount=False,custom_ops=[DayLast, FFillNan, BFillNan,
|
|
Date, Select, IsNull, IsInf, Cut],expression_cache=None,**qlib_conf,)'
|
|
function: highfreq_provider._init_qlib
|
|
- docstring: ' preload the calendar for cache
|
|
|
|
# This code used the copy-on-write feature of Linux# to avoid calculating the
|
|
calendar multiple times in the subprocess.# This code may accelerate, but may
|
|
be not useful on Windows and Mac OsCal.calendar(freq=self.freq)get_calendar_day(freq=self.freq)'
|
|
function: highfreq_provider._prepare_calender_cache
|
|
- docstring: null
|
|
function: highfreq_provider._gen_dataframe
|
|
- docstring: null
|
|
function: highfreq_provider._gen_data
|
|
- docstring: null
|
|
function: highfreq_provider._gen_dataset
|
|
- docstring: null
|
|
function: highfreq_provider._gen_day_dataset
|
|
- docstring: null
|
|
function: highfreq_provider.generate_dataset
|
|
- docstring: null
|
|
function: highfreq_provider._gen_stock_dataset
|
|
- docstring: " \nThis processor is designed for Alpha158. And will be replaced\
|
|
\ by simple processors in the future"
|
|
function: processor.ConfigSectionProcessor
|
|
- docstring: null
|
|
function: processor._transform
|
|
- docstring: null
|
|
function: processor._label_norm
|
|
- docstring: null
|
|
function: highfreq_handler.HighFreqHandler
|
|
- docstring: null
|
|
function: highfreq_handler.get_feature_config
|
|
- docstring: null
|
|
function: highfreq_handler.get_normalized_price_feature
|
|
- docstring: null
|
|
function: highfreq_handler.HighFreqGeneralHandler
|
|
- docstring: null
|
|
function: highfreq_handler.get_feature_config
|
|
- docstring: null
|
|
function: highfreq_handler.get_normalized_price_feature
|
|
- docstring: null
|
|
function: highfreq_handler.HighFreqBacktestHandler
|
|
- docstring: null
|
|
function: highfreq_handler.get_feature_config
|
|
- docstring: null
|
|
function: highfreq_handler.HighFreqGeneralBacktestHandler
|
|
- docstring: null
|
|
function: highfreq_handler.get_feature_config
|
|
- docstring: null
|
|
function: highfreq_handler.HighFreqOrderHandler
|
|
- docstring: null
|
|
function: highfreq_handler.get_feature_config
|
|
- docstring: null
|
|
function: highfreq_handler.get_normalized_price_feature
|
|
- docstring: null
|
|
function: highfreq_handler.get_normalized_vwap_price_feature
|
|
- docstring: null
|
|
function: highfreq_handler.get_volume_feature
|
|
- docstring: null
|
|
function: highfreq_handler.HighFreqBacktestOrderHandler
|
|
- docstring: null
|
|
function: highfreq_processor.HighFreqTrans
|
|
- docstring: null
|
|
function: highfreq_processor.fit
|
|
- docstring: null
|
|
function: highfreq_processor.HighFreqNorm
|
|
- docstring: null
|
|
function: sepdf.align_index
|
|
- docstring: " \n(Sep)erate DataFrameWe usually concat multiple dataframe to be\
|
|
\ processed together(Such as feature, label, weight, filter).However, they are\
|
|
\ usually be used separately at last.This will result in extra cost for concatenating\
|
|
\ and splitting data(reshaping and copying data in the memory is very expensive)SepDataFrame\
|
|
\ tries to act like a DataFrame whose column with multiindex"
|
|
function: 'sepdf.SepDataFrame:'
|
|
- docstring: null
|
|
function: sepdf.loc
|
|
- docstring: null
|
|
function: sepdf.index
|
|
- docstring: " \nAssumptions:- inplace methods will return None"
|
|
function: sepdf.apply_each
|
|
- docstring: null
|
|
function: sepdf.sort_index
|
|
- docstring: null
|
|
function: sepdf.copy
|
|
- docstring: null
|
|
function: sepdf._update_join
|
|
- docstring: null
|
|
function: sepdf.droplevel
|
|
- docstring: null
|
|
function: sepdf.columns
|
|
- docstring: null
|
|
function: sepdf.merge
|
|
- docstring: ' Mock Class
|
|
|
|
self._sdf = sdfself.axis = Noneself.join = joinself.axis = axisreturn selfif self.axis
|
|
== 1:if isinstance(args, str):return self._sdf[args]elif isinstance(args, (tuple,
|
|
list)):new_df_dict = {k: self._sdf[k] for k in args}return SepDataFrame(new_df_dict,
|
|
join=self.join if self.join in args else args[0], skip_align=True)else:raise NotImplementedError(f"This
|
|
type of input is not supported")elif self.axis == 0:return SepDataFrame({k: df.loc(axis=0)[args]
|
|
for k, df in self._sdf._df_dict.items()}, join=self.join, skip_align=True)else:df
|
|
= self._sdfif isinstance(args, tuple):ax0, *ax1 = argsif len(ax1) == 0:ax1 = Noneif
|
|
ax1 is not None:df = df.loc(axis=1)[ax1]if ax0 is not None:df = df.loc(axis=0)[ax0]return
|
|
dfelse:return df.loc(axis=0)[args]# Patch pandas DataFrame# Tricking isinstance
|
|
to accept SepDataFrame as its subclassimport builtins'
|
|
function: 'sepdf.SDFLoc:'
|
|
- docstring: null
|
|
function: 'tuner.Tuner:'
|
|
- docstring: null
|
|
function: tuner.tune
|
|
- docstring: " \nImplement this method to give an optimization factor using\
|
|
\ parameters in space.:return: {'loss': a factor for optimization, float type,'status':\
|
|
\ the status of this evaluation step, STATUS_OK or STATUS_FAIL}."
|
|
function: tuner.objective
|
|
- docstring: " \nImplement this method to setup the searching space of tuner.:return:\
|
|
\ searching space, dict type."
|
|
function: tuner.setup_space
|
|
- docstring: " \nImplement this method to save the best parameters of this\
|
|
\ tuner."
|
|
function: tuner.save_local_best_params
|
|
- docstring: null
|
|
function: tuner.QLibTuner
|
|
- docstring: null
|
|
function: tuner.objective
|
|
- docstring: null
|
|
function: tuner.fetch_result
|
|
- docstring: null
|
|
function: tuner.setup_estimator_config
|
|
- docstring: null
|
|
function: tuner.setup_space
|
|
- docstring: null
|
|
function: 'pipeline.Pipeline:'
|
|
- docstring: null
|
|
function: pipeline.run
|
|
- docstring: " \nImplement this method to build the tuner by configreturn:\
|
|
\ tuner"
|
|
function: pipeline.init_tuner
|
|
- docstring: null
|
|
function: 'config.TunerConfigManager:'
|
|
- docstring: " \n:param config: The config dict for tuner experiment:param\
|
|
\ TUNER_CONFIG_MANAGER: The tuner config manager"
|
|
function: 'config.PipelineExperimentConfig:'
|
|
- docstring: null
|
|
function: utils.ICLoss
|
|
- docstring: ' forward.
|
|
|
|
FIXME:- Some times it will be a slightly different from the result from `pandas.corr()`-
|
|
It may be caused by the precision problem of model;:param pred::param y::param
|
|
idx: Assume the level of the idx is (date, inst), and it is sorted'
|
|
function: utils.forward
|
|
- docstring: " \nClip the weights.Parameters----------clip_weight: floatThe clip\
|
|
\ threshold.clip_method: strThe clip method. Current available: \"clamp\", \"\
|
|
tanh\", and \"sigmoid\"."
|
|
function: utils.preds_to_weight_with_clamp
|
|
- docstring: null
|
|
function: utils.SingleMetaBase
|
|
- docstring: null
|
|
function: 'dataset.InternalData:'
|
|
- docstring: " \nafter running this function `self.data_ic_df` will become\
|
|
\ set.Each col represents a data.Each row represents the Timestamp of performance\
|
|
\ of that data.For example,.. code-block:: python2021-06-21 2021-06-04 2021-05-21\
|
|
\ 2021-05-07 2021-04-20 2021-04-06 2021-03-22 2021-03-08 ...2021-07-02 2021-06-18\
|
|
\ 2021-06-03 2021-05-20 2021-05-06 2021-04-19 2021-04-02 2021-03-19 ...datetime\
|
|
\ \
|
|
\ ...2018-01-02 0.079782 0.115975 0.070866 0.028849 -0.081170\
|
|
\ 0.140380 0.063864 0.110987 ...2018-01-03 0.123386 0.107789 0.071037\
|
|
\ 0.045278 -0.060782 0.167446 0.089779 0.124476 ...2018-01-04 0.140775\
|
|
\ 0.097206 0.063702 0.042415 -0.078164 0.173218 0.098914 0.114389\
|
|
\ ...2018-01-05 0.030320 -0.037209 -0.044536 -0.047267 -0.081888 0.045648\
|
|
\ 0.059947 0.047652 ...2018-01-08 0.107201 0.009219 -0.015995 -0.036594\
|
|
\ -0.086633 0.108965 0.122164 0.108508 ...... ... \
|
|
\ ... ... ... ... ... ... ... ..."
|
|
function: dataset.setup
|
|
- docstring: null
|
|
function: dataset._calc_perf
|
|
- docstring: ' update the data for online trading
|
|
|
|
# TODO:# when new data are totally(including label) available# - update the prediction#
|
|
- update the data similarity map(if applied)'
|
|
function: dataset.update
|
|
- docstring: ' Meta Task for Data Selection
|
|
|
|
'
|
|
function: dataset.MetaTaskDS
|
|
- docstring: null
|
|
function: dataset._get_processed_meta_info
|
|
- docstring: null
|
|
function: dataset.get_meta_input
|
|
- docstring: " \nA dataset for meta model.Parameters----------task_tpl : Union[dict,\
|
|
\ list]Decide what tasks are used.- dict : the task template, the prepared task\
|
|
\ is generated with `step`, `trunc_days` and `RollingGen`- list : when list, use\
|
|
\ the list of tasks directlythe list is supposed to be sorted according timelinestep\
|
|
\ : intthe rolling steptrunc_days: intdays to be truncated based on the test startrolling_ext_days:\
|
|
\ intsometimes users want to train meta models for a longer test period but with\
|
|
\ smaller rolling steps for more task samples.the total length of test periods\
|
|
\ will be `step + rolling_ext_days`exp_name : Union[str, InternalData]Decide what\
|
|
\ meta_info are used for prediction.- str: the name of the experiment to store\
|
|
\ the performance of data- InternalData: a prepared internal datasegments: Union[Dict[Text,\
|
|
\ Tuple], float]the segments to divide databoth left and rightif segments is a\
|
|
\ float:the float represents the percentage of data for traininghist_step_n: intlength\
|
|
\ of historical steps for the meta infomationtask_mode : strPlease refer to the\
|
|
\ docs of MetaTask"
|
|
function: dataset.MetaDatasetDS
|
|
- docstring: " \nPlease refer to `self.internal_data.setup` for detailed information\
|
|
\ about `self.internal_data.data_ic_df`Indices with format below can be successfully\
|
|
\ sliced by `ic_df.loc[:end, pd.IndexSlice[:, :end]]`2021-06-21 2021-06-04 ..\
|
|
\ 2021-03-22 2021-03-082021-07-02 2021-06-18 .. 2021-04-02 NoneReturns-------a\
|
|
\ pd.DataFrame with similar content below.- each column corresponds to a trained\
|
|
\ model named by the training data range- each row corresponds to a day of data\
|
|
\ tested by the models of the columns- The rows cells that overlaps with the data\
|
|
\ used by columns are masked2009-01-05 2009-02-09 ... 2011-04-27 2011-05-262009-02-06\
|
|
\ 2009-03-06 ... 2011-05-25 2011-06-23datetime ...2009-01-13\
|
|
\ NaN 0.310639 ... -0.169057 0.1377922009-01-14 NaN 0.261086\
|
|
\ ... -0.143567 0.082581... ... ... ... ... \
|
|
\ ...2011-06-30 -0.054907 -0.020219 ... -0.023226 NaN2011-07-01\
|
|
\ -0.075762 -0.026626 ... -0.003167 NaN"
|
|
function: dataset._prepare_meta_ipt
|
|
- docstring: " \nmask overlap informationdata after self.name[end] with\
|
|
\ self.trunc_days that contains future info are also considered as overlap infoApproximately\
|
|
\ the diagnal + horizon length of data are masked."
|
|
function: dataset.mask_overlap
|
|
- docstring: null
|
|
function: model.TimeReweighter
|
|
- docstring: null
|
|
function: model.reweight
|
|
- docstring: " \nThe meta-model for meta-learning-based data selection."
|
|
function: model.MetaModelDS
|
|
- docstring: null
|
|
function: model.run_epoch
|
|
- docstring: " \nThe meta-learning-based data selection interacts directly\
|
|
\ with meta-dataset due to the close-form proxy measurement.Parameters----------meta_dataset\
|
|
\ : MetaDatasetDSThe meta-model takes the meta-dataset for its training process."
|
|
function: model.fit
|
|
- docstring: null
|
|
function: model._prepare_task
|
|
- docstring: null
|
|
function: net.TimeWeightMeta
|
|
- docstring: null
|
|
function: net.forward
|
|
- docstring: " \nParameters----------alpha : floatthe regularization for sub\
|
|
\ model (useful when align meta model with linear submodel)"
|
|
function: net.PredNet
|
|
- docstring: null
|
|
function: net.get_sample_weights
|
|
- docstring: ' Please refer to the docs of MetaTaskDS for the description of
|
|
the variables
|
|
|
|
weights = self.get_sample_weights(X, time_perf, time_belong, ignore_weight=ignore_weight)X_w
|
|
= X.T * weights.view(1, -1)theta = torch.inverse(X_w @ X + self.alpha * torch.eye(X_w.shape[0]))
|
|
@ X_w @ yreturn X_test @ theta, weights'
|
|
function: net.forward
|
|
- docstring: " \nThis is the multiple segments signal record class that generates\
|
|
\ the signal prediction.This class inherits the ``RecordTemp`` class."
|
|
function: record_temp.MultiSegRecord
|
|
- docstring: null
|
|
function: record_temp.generate
|
|
- docstring: " \nThis is the Signal MSE Record class that computes the mean squared\
|
|
\ error (MSE).This class inherits the ``SignalMseRecord`` class."
|
|
function: record_temp.SignalMseRecord
|
|
- docstring: null
|
|
function: record_temp.generate
|
|
- docstring: " \nLoad High-Freq Calendar Date Using Memcache.!!!NOTE: Loading the\
|
|
\ calendar is quite slow. So loading calendar before start multiprocessing will\
|
|
\ make it faster.Parameters----------freq : strfrequency of read calendar file.future\
|
|
\ : boolwhether including future trading day.Returns-------_calendar:array of\
|
|
\ date."
|
|
function: high_freq.get_calendar_day
|
|
- docstring: ' Load High-Freq Calendar Minute Using Memcache
|
|
|
|
flag = f"{freq}_future_{future}_day"if flag in H["c"]:_calendar = H["c"][flag]else:_calendar
|
|
= np.array(list(map(lambda x: x.minute // 30, Cal.load_calendar(freq, future))))H["c"][flag]
|
|
= _calendarreturn _calendar'
|
|
function: high_freq.get_calendar_minute
|
|
- docstring: ' DayCumsum Operator during start time and end time.
|
|
|
|
Parameters----------feature : Expressionfeature instancestart : strthe start time
|
|
of backtest in one day.!!!NOTE: "9:30" means the time period of (9:30, 9:31) is
|
|
in transaction.end : strthe end time of backtest in one day.!!!NOTE: "14:59" means
|
|
the time period of (14:59, 15:00) is in transaction,but (15:00, 15:01) is not.So
|
|
start="9:30" and end="14:59" means trading all day.Returns----------feature:a
|
|
series of that each value equals the cumsum value during start time and end time.Otherwise,
|
|
the value is zero.'
|
|
function: high_freq.DayCumsum
|
|
- docstring: null
|
|
function: high_freq.period_cusum
|
|
- docstring: null
|
|
function: high_freq._load_internal
|
|
- docstring: ' DayLast Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:a
|
|
series of that each value equals the last value of its day'
|
|
function: high_freq.DayLast
|
|
- docstring: null
|
|
function: high_freq._load_internal
|
|
- docstring: ' FFillNan Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:a
|
|
forward fill nan feature'
|
|
function: high_freq.FFillNan
|
|
- docstring: null
|
|
function: high_freq._load_internal
|
|
- docstring: ' BFillNan Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:a
|
|
backfoward fill nan feature'
|
|
function: high_freq.BFillNan
|
|
- docstring: null
|
|
function: high_freq._load_internal
|
|
- docstring: ' Date Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:a
|
|
series of that each value is the date corresponding to feature.index'
|
|
function: high_freq.Date
|
|
- docstring: null
|
|
function: high_freq._load_internal
|
|
- docstring: ' Select Operator
|
|
|
|
Parameters----------feature_left : Expressionfeature instance, select conditionfeature_right
|
|
: Expressionfeature instance, select valueReturns----------feature:value(feature_right)
|
|
that meets the condition(feature_left)'
|
|
function: high_freq.Select
|
|
- docstring: null
|
|
function: high_freq._load_internal
|
|
- docstring: ' IsNull Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:A
|
|
series indicating whether the feature is nan'
|
|
function: high_freq.IsNull
|
|
- docstring: null
|
|
function: high_freq._load_internal
|
|
- docstring: ' IsInf Operator
|
|
|
|
Parameters----------feature : Expressionfeature instanceReturns----------feature:A
|
|
series indicating whether the feature is inf'
|
|
function: high_freq.IsInf
|
|
- docstring: null
|
|
function: high_freq._load_internal
|
|
- docstring: ' Cut Operator
|
|
|
|
Parameters----------feature : Expressionfeature instancel : intl > 0, delete the
|
|
first l elements of feature (default is None, which means 0)r : intr < 0, delete
|
|
the last -r elements of feature (default is None, which means 0)Returns----------feature:A
|
|
series with the first l and last -r elements deleted from the feature.Note: It
|
|
is deleted from the raw data, not the sliced data'
|
|
function: high_freq.Cut
|
|
- docstring: null
|
|
function: high_freq._load_internal
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.LocalformerModel
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.use_gpu
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.mse
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.loss_fn
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.metric_fn
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.train_epoch
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.test_epoch
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.fit
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.predict
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.PositionalEncoding
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.forward
|
|
- docstring: null
|
|
function: pytorch_localformer_ts._get_clones
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.LocalformerEncoder
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.forward
|
|
- docstring: null
|
|
function: pytorch_localformer_ts.Transformer
|
|
- docstring: null
|
|
function: pytorch_localformer.LocalformerModel
|
|
- docstring: null
|
|
function: pytorch_localformer.use_gpu
|
|
- docstring: null
|
|
function: pytorch_localformer.mse
|
|
- docstring: null
|
|
function: pytorch_localformer.loss_fn
|
|
- docstring: null
|
|
function: pytorch_localformer.metric_fn
|
|
- docstring: null
|
|
function: pytorch_localformer.train_epoch
|
|
- docstring: null
|
|
function: pytorch_localformer.test_epoch
|
|
- docstring: null
|
|
function: pytorch_localformer.fit
|
|
- docstring: null
|
|
function: pytorch_localformer.predict
|
|
- docstring: null
|
|
function: pytorch_localformer.PositionalEncoding
|
|
- docstring: null
|
|
function: pytorch_localformer.forward
|
|
- docstring: null
|
|
function: pytorch_localformer._get_clones
|
|
- docstring: null
|
|
function: pytorch_localformer.LocalformerEncoder
|
|
- docstring: null
|
|
function: pytorch_localformer.forward
|
|
- docstring: null
|
|
function: pytorch_localformer.Transformer
|
|
- docstring: ' DNN Model
|
|
|
|
Parameters----------input_dim : intinput dimensionoutput_dim : intoutput dimensionlayers
|
|
: tuplelayer sizeslr : floatlearning rateoptimizer : stroptimizer nameGPU : intthe
|
|
GPU ID used for training'
|
|
function: pytorch_nn.DNNModelPytorch
|
|
- docstring: null
|
|
function: pytorch_nn.use_gpu
|
|
- docstring: null
|
|
function: pytorch_nn.fit
|
|
- docstring: null
|
|
function: pytorch_nn.get_lr
|
|
- docstring: null
|
|
function: pytorch_nn.get_loss
|
|
- docstring: null
|
|
function: pytorch_nn.get_metric
|
|
- docstring: ' Reusing predicting NN.
|
|
|
|
Scenarios1) test inference (data may come from CPU and expect the output data
|
|
is on CPU)2) evaluation on training (data may come from GPU)'
|
|
function: pytorch_nn._nn_predict
|
|
- docstring: null
|
|
function: pytorch_nn.predict
|
|
- docstring: null
|
|
function: pytorch_nn.save
|
|
- docstring: null
|
|
function: pytorch_nn.load
|
|
- docstring: ' Computes and stores the average and current value
|
|
|
|
self.reset()'
|
|
function: 'pytorch_nn.AverageMeter:'
|
|
- docstring: null
|
|
function: pytorch_nn.reset
|
|
- docstring: null
|
|
function: pytorch_nn.update
|
|
- docstring: null
|
|
function: pytorch_nn.Net
|
|
- docstring: null
|
|
function: pytorch_nn._weight_init
|
|
- docstring: null
|
|
function: pytorch_gats_ts.DailyBatchSampler
|
|
- docstring: ' GATs Model
|
|
|
|
Parameters----------lr : floatlearning rated_feat : intinput dimensions for each
|
|
time stepmetric : strthe evaluation metric used in early stopoptimizer : stroptimizer
|
|
nameGPU : intthe GPU ID used for training'
|
|
function: pytorch_gats_ts.GATs
|
|
- docstring: null
|
|
function: pytorch_gats_ts.use_gpu
|
|
- docstring: null
|
|
function: pytorch_gats_ts.mse
|
|
- docstring: null
|
|
function: pytorch_gats_ts.loss_fn
|
|
- docstring: null
|
|
function: pytorch_gats_ts.metric_fn
|
|
- docstring: null
|
|
function: pytorch_gats_ts.get_daily_inter
|
|
- docstring: null
|
|
function: pytorch_gats_ts.train_epoch
|
|
- docstring: null
|
|
function: pytorch_gats_ts.test_epoch
|
|
- docstring: null
|
|
function: pytorch_gats_ts.fit
|
|
- docstring: null
|
|
function: pytorch_gats_ts.predict
|
|
- docstring: null
|
|
function: pytorch_gats_ts.GATModel
|
|
- docstring: null
|
|
function: pytorch_gats_ts.cal_attention
|
|
- docstring: ' Build a basic CNN encoder
|
|
|
|
Parameters----------input_dim : intThe input dimensionoutput_dim : intThe output
|
|
dimensionkernel_size : intThe size of convolutional kernels'
|
|
function: pytorch_krnn.CNNEncoderBase
|
|
- docstring: " \nParameters----------x : torch.Tensorinput dataReturns-------torch.TensorUpdated\
|
|
\ representations"
|
|
function: pytorch_krnn.forward
|
|
- docstring: ' Build K parallel RNNs
|
|
|
|
Parameters----------input_dim : intThe input dimensionoutput_dim : intThe output
|
|
dimensiondup_num : intThe number of parallel RNNsrnn_layers: intThe number of
|
|
RNN layers'
|
|
function: pytorch_krnn.KRNNEncoderBase
|
|
- docstring: " \nParameters----------x : torch.TensorInput datan_id : torch.TensorNode\
|
|
\ indicesReturns-------torch.TensorUpdated representations"
|
|
function: pytorch_krnn.forward
|
|
- docstring: ' Build an encoder composed of CNN and KRNN
|
|
|
|
Parameters----------cnn_input_dim : intThe input dimension of CNNcnn_output_dim
|
|
: intThe output dimension of CNNcnn_kernel_size : intThe size of convolutional
|
|
kernelsrnn_output_dim : intThe output dimension of KRNNrnn_dup_num : intThe number
|
|
of parallel duplicates for KRNNrnn_layers : intThe number of RNN layers'
|
|
function: pytorch_krnn.CNNKRNNEncoder
|
|
- docstring: " \nParameters----------x : torch.TensorInput datan_id : torch.TensorNode\
|
|
\ indicesReturns-------torch.TensorUpdated representations"
|
|
function: pytorch_krnn.forward
|
|
- docstring: ' Build a KRNN model
|
|
|
|
Parameters----------fea_dim : intThe feature dimensioncnn_dim : intThe hidden
|
|
dimension of CNNcnn_kernel_size : intThe size of convolutional kernelsrnn_dim
|
|
: intThe hidden dimension of KRNNrnn_dups : intThe number of parallel duplicatesrnn_layers:
|
|
intThe number of RNN layers'
|
|
function: pytorch_krnn.KRNNModel
|
|
- docstring: null
|
|
function: pytorch_krnn.forward
|
|
- docstring: ' KRNN Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_krnn.KRNN
|
|
- docstring: null
|
|
function: pytorch_krnn.use_gpu
|
|
- docstring: null
|
|
function: pytorch_krnn.mse
|
|
- docstring: null
|
|
function: pytorch_krnn.loss_fn
|
|
- docstring: null
|
|
function: pytorch_krnn.metric_fn
|
|
- docstring: null
|
|
function: pytorch_krnn.get_daily_inter
|
|
- docstring: null
|
|
function: pytorch_krnn.train_epoch
|
|
- docstring: null
|
|
function: pytorch_krnn.test_epoch
|
|
- docstring: null
|
|
function: pytorch_krnn.fit
|
|
- docstring: ' Linear Model
|
|
|
|
Solve one of the following regression problems:- `ols`: min_w |y - Xw|^2_2- `nnls`:
|
|
min_w |y - Xw|^2_2, s.t. w >= 0- `ridge`: min_w |y - Xw|^2_2 + \alpha*|w|^2_2-
|
|
`lasso`: min_w |y - Xw|^2_2 + \alpha*|w|_1where `w` is the regression coefficient.'
|
|
function: linear.LinearModel
|
|
- docstring: null
|
|
function: linear.fit
|
|
- docstring: null
|
|
function: linear._fit
|
|
- docstring: null
|
|
function: linear._fit_nnls
|
|
- docstring: ' ADD Model
|
|
|
|
Parameters----------lr : floatlearning rated_feat : intinput dimensions for each
|
|
time stepmetric : strthe evaluation metric used in early stopoptimizer : stroptimizer
|
|
nameGPU : intthe GPU ID used for training'
|
|
function: pytorch_add.ADD
|
|
- docstring: null
|
|
function: pytorch_add.use_gpu
|
|
- docstring: null
|
|
function: pytorch_add.loss_pre_excess
|
|
- docstring: null
|
|
function: pytorch_add.loss_pre_market
|
|
- docstring: null
|
|
function: pytorch_add.loss_pre
|
|
- docstring: null
|
|
function: pytorch_add.loss_adv_excess
|
|
- docstring: null
|
|
function: pytorch_add.loss_adv_market
|
|
- docstring: null
|
|
function: pytorch_add.loss_adv
|
|
- docstring: null
|
|
function: pytorch_add.loss_fn
|
|
- docstring: null
|
|
function: pytorch_add.loss_rec
|
|
- docstring: null
|
|
function: pytorch_add.get_daily_inter
|
|
- docstring: null
|
|
function: pytorch_add.cal_ic_metrics
|
|
- docstring: null
|
|
function: pytorch_add.test_epoch
|
|
- docstring: null
|
|
function: pytorch_add.train_epoch
|
|
- docstring: null
|
|
function: pytorch_add.log_metrics
|
|
- docstring: null
|
|
function: pytorch_add.bootstrap_fit
|
|
- docstring: null
|
|
function: pytorch_add.gen_market_label
|
|
- docstring: null
|
|
function: pytorch_add.fit_thresh
|
|
- docstring: null
|
|
function: pytorch_add.fit
|
|
- docstring: null
|
|
function: pytorch_add.predict
|
|
- docstring: null
|
|
function: pytorch_add.ADDModel
|
|
- docstring: null
|
|
function: pytorch_add.forward
|
|
- docstring: null
|
|
function: pytorch_add.Decoder
|
|
- docstring: null
|
|
function: pytorch_add.forward
|
|
- docstring: null
|
|
function: pytorch_add.RevGradFunc
|
|
- docstring: null
|
|
function: pytorch_add.forward
|
|
- docstring: null
|
|
function: pytorch_add.backward
|
|
- docstring: " \nA gradient reversal layer.This layer has no parameters, and\
|
|
\ simply reverses the gradientin the backward pass."
|
|
function: pytorch_add.RevGrad
|
|
- docstring: null
|
|
function: pytorch_add.step_alpha
|
|
- docstring: ' XGBModel Model
|
|
|
|
self._params = {}self._params.update(kwargs)self.model = None'
|
|
function: xgboost.XGBModel
|
|
- docstring: null
|
|
function: xgboost.fit
|
|
- docstring: null
|
|
function: xgboost.predict
|
|
- docstring: ' get feature importance
|
|
|
|
Notes-------parameters reference:https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.Booster.get_score'
|
|
function: xgboost.get_feature_importance
|
|
- docstring: ' TCN Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepn_chans: intnumber
|
|
of channelsmetric: strthe evaluation metric used in early stopoptimizer : stroptimizer
|
|
nameGPU : strthe GPU ID(s) used for training'
|
|
function: pytorch_tcn.TCN
|
|
- docstring: null
|
|
function: pytorch_tcn.use_gpu
|
|
- docstring: null
|
|
function: pytorch_tcn.mse
|
|
- docstring: null
|
|
function: pytorch_tcn.loss_fn
|
|
- docstring: null
|
|
function: pytorch_tcn.metric_fn
|
|
- docstring: null
|
|
function: pytorch_tcn.train_epoch
|
|
- docstring: null
|
|
function: pytorch_tcn.test_epoch
|
|
- docstring: null
|
|
function: pytorch_tcn.fit
|
|
- docstring: null
|
|
function: pytorch_tcn.predict
|
|
- docstring: null
|
|
function: pytorch_tcn.TCNModel
|
|
- docstring: ' Double Ensemble Model
|
|
|
|
self,base_model="gbm",loss="mse",num_models=6,enable_sr=True,enable_fs=True,alpha1=1.0,alpha2=1.0,bins_sr=10,bins_fs=5,decay=None,sample_ratios=None,sub_weights=None,epochs=100,early_stopping_rounds=None,**kwargs):self.base_model
|
|
= base_model # "gbm" or "mlp", specifically, we use lgbm for "gbm"self.num_models
|
|
= num_models # the number of sub-modelsself.enable_sr = enable_srself.enable_fs
|
|
= enable_fsself.alpha1 = alpha1self.alpha2 = alpha2self.bins_sr = bins_srself.bins_fs
|
|
= bins_fsself.decay = decayif sample_ratios is None: # the default values for
|
|
sample_ratiossample_ratios = [0.8, 0.7, 0.6, 0.5, 0.4]if sub_weights is None: #
|
|
the default values for sub_weightssub_weights = [1] * self.num_modelsif not len(sample_ratios)
|
|
== bins_fs:raise ValueError("The length of sample_ratios should be equal to bins_fs.")self.sample_ratios
|
|
= sample_ratiosif not len(sub_weights) == num_models:raise ValueError("The length
|
|
of sub_weights should be equal to num_models.")self.sub_weights = sub_weightsself.epochs
|
|
= epochsself.logger = get_module_logger("DEnsembleModel")self.logger.info("Double
|
|
Ensemble Model...")self.ensemble = [] # the current ensemble model, a list contains
|
|
all the sub-modelsself.sub_features = [] # the features for each sub model in
|
|
the form of pandas.Indexself.params = {"objective": loss}self.params.update(kwargs)self.loss
|
|
= lossself.early_stopping_rounds = early_stopping_rounds'
|
|
function: double_ensemble.DEnsembleModel
|
|
- docstring: null
|
|
function: double_ensemble.fit
|
|
- docstring: null
|
|
function: double_ensemble.train_submodel
|
|
- docstring: null
|
|
function: double_ensemble._prepare_data_gbm
|
|
- docstring: " \nthe SR module of Double Ensemble:param loss_curve: the shape\
|
|
\ is NxTthe loss curve for the previous sub-model, where the element (i, t) if\
|
|
\ the error on the i-th sampleafter the t-th iteration in the training of the\
|
|
\ previous sub-model.:param loss_values: the shape is Nthe loss of the current\
|
|
\ ensemble on the i-th sample.:param k_th: the index of the current sub-model,\
|
|
\ starting from 1:return: weightsthe weights for all the samples."
|
|
function: double_ensemble.sample_reweight
|
|
- docstring: " \nthe FS module of Double Ensemble:param df_train: the shape\
|
|
\ is NxF:param loss_values: the shape is Nthe loss of the current ensemble on\
|
|
\ the i-th sample.:return: res_feat: in the form of pandas.Index"
|
|
function: double_ensemble.feature_selection
|
|
- docstring: null
|
|
function: double_ensemble.get_loss
|
|
- docstring: null
|
|
function: double_ensemble.retrieve_loss_curve
|
|
- docstring: null
|
|
function: double_ensemble.predict
|
|
- docstring: null
|
|
function: double_ensemble.predict_sub
|
|
- docstring: ' get feature importance
|
|
|
|
Notes-----parameters reference:https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Booster.html?highlight=feature_importance#lightgbm.Booster.feature_importance'
|
|
function: double_ensemble.get_feature_importance
|
|
- docstring: ' LightGBM Model for high frequency prediction
|
|
|
|
if loss not in {"mse", "binary"}:raise NotImplementedErrorself.params = {"objective":
|
|
loss, "verbosity": -1}self.params.update(kwargs)self.model = None'
|
|
function: highfreq_gdbt_model.HFLGBModel
|
|
- docstring: " \nCalcaute the signal metrics by daily level"
|
|
function: highfreq_gdbt_model._cal_signal_metrics
|
|
- docstring: " \nTest the signal in high frequency test set"
|
|
function: highfreq_gdbt_model.hf_signal_test
|
|
- docstring: null
|
|
function: highfreq_gdbt_model._prepare_data
|
|
- docstring: null
|
|
function: highfreq_gdbt_model.mapping_fn
|
|
- docstring: null
|
|
function: highfreq_gdbt_model.fit
|
|
- docstring: null
|
|
function: highfreq_gdbt_model.predict
|
|
- docstring: " \nfinetune modelParameters----------dataset : DatasetHdataset\
|
|
\ for finetuningnum_boost_round : intnumber of round to finetune modelverbose_eval\
|
|
\ : intverbose level"
|
|
function: highfreq_gdbt_model.finetune
|
|
- docstring: ' GATs Model
|
|
|
|
Parameters----------lr : floatlearning rated_feat : intinput dimensions for each
|
|
time stepmetric : strthe evaluation metric used in early stopoptimizer : stroptimizer
|
|
nameGPU : intthe GPU ID used for training'
|
|
function: pytorch_gats.GATs
|
|
- docstring: null
|
|
function: pytorch_gats.use_gpu
|
|
- docstring: null
|
|
function: pytorch_gats.mse
|
|
- docstring: null
|
|
function: pytorch_gats.loss_fn
|
|
- docstring: null
|
|
function: pytorch_gats.metric_fn
|
|
- docstring: null
|
|
function: pytorch_gats.get_daily_inter
|
|
- docstring: null
|
|
function: pytorch_gats.train_epoch
|
|
- docstring: null
|
|
function: pytorch_gats.test_epoch
|
|
- docstring: null
|
|
function: pytorch_gats.fit
|
|
- docstring: null
|
|
function: pytorch_gats.predict
|
|
- docstring: null
|
|
function: pytorch_gats.GATModel
|
|
- docstring: null
|
|
function: pytorch_gats.cal_attention
|
|
- docstring: " \nThis function is to obtain the storage size unit of a (or multiple)\
|
|
\ models.Parameters----------models_or_parameters : PyTorch model(s) or a list\
|
|
\ of parameters.unit : the storage size unit.Returns-------The number of parameters\
|
|
\ of the given model(s) or parameters."
|
|
function: pytorch_utils.count_parameters
|
|
- docstring: ' GRU Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_gru.GRU
|
|
- docstring: null
|
|
function: pytorch_gru.use_gpu
|
|
- docstring: null
|
|
function: pytorch_gru.mse
|
|
- docstring: null
|
|
function: pytorch_gru.loss_fn
|
|
- docstring: null
|
|
function: pytorch_gru.metric_fn
|
|
- docstring: null
|
|
function: pytorch_gru.train_epoch
|
|
- docstring: null
|
|
function: pytorch_gru.test_epoch
|
|
- docstring: null
|
|
function: pytorch_gru.fit
|
|
- docstring: null
|
|
function: pytorch_gru.predict
|
|
- docstring: null
|
|
function: pytorch_gru.GRUModel
|
|
- docstring: null
|
|
function: pytorch_sfm.SFM_Model
|
|
- docstring: null
|
|
function: pytorch_sfm.forward
|
|
- docstring: null
|
|
function: pytorch_sfm.init_states
|
|
- docstring: null
|
|
function: pytorch_sfm.get_constants
|
|
- docstring: ' SFM Model
|
|
|
|
Parameters----------input_dim : intinput dimensionoutput_dim : intoutput dimensionlr
|
|
: floatlearning rateoptimizer : stroptimizer nameGPU : intthe GPU ID used for
|
|
training'
|
|
function: pytorch_sfm.SFM
|
|
- docstring: null
|
|
function: pytorch_sfm.use_gpu
|
|
- docstring: null
|
|
function: pytorch_sfm.test_epoch
|
|
- docstring: null
|
|
function: pytorch_sfm.train_epoch
|
|
- docstring: null
|
|
function: pytorch_sfm.fit
|
|
- docstring: null
|
|
function: pytorch_sfm.mse
|
|
- docstring: null
|
|
function: pytorch_sfm.loss_fn
|
|
- docstring: null
|
|
function: pytorch_sfm.metric_fn
|
|
- docstring: null
|
|
function: pytorch_sfm.predict
|
|
- docstring: ' Computes and stores the average and current value
|
|
|
|
self.reset()'
|
|
function: 'pytorch_sfm.AverageMeter:'
|
|
- docstring: null
|
|
function: pytorch_sfm.reset
|
|
- docstring: null
|
|
function: tcn.Chomp1d
|
|
- docstring: null
|
|
function: tcn.forward
|
|
- docstring: null
|
|
function: tcn.TemporalBlock
|
|
- docstring: null
|
|
function: tcn.init_weights
|
|
- docstring: null
|
|
function: tcn.forward
|
|
- docstring: null
|
|
function: tcn.TemporalConvNet
|
|
- docstring: ' LSTM Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_lstm.LSTM
|
|
- docstring: null
|
|
function: pytorch_lstm.use_gpu
|
|
- docstring: null
|
|
function: pytorch_lstm.mse
|
|
- docstring: null
|
|
function: pytorch_lstm.loss_fn
|
|
- docstring: null
|
|
function: pytorch_lstm.metric_fn
|
|
- docstring: null
|
|
function: pytorch_lstm.train_epoch
|
|
- docstring: null
|
|
function: pytorch_lstm.test_epoch
|
|
- docstring: null
|
|
function: pytorch_lstm.fit
|
|
- docstring: null
|
|
function: pytorch_lstm.predict
|
|
- docstring: null
|
|
function: pytorch_lstm.LSTMModel
|
|
- docstring: " \nTRA ModelArgs:model_config (dict): model config (will be used\
|
|
\ by RNN or Transformer)tra_config (dict): TRA config (will be used by TRA)model_type\
|
|
\ (str): which backbone model to use (RNN/Transformer)lr (float): learning raten_epochs\
|
|
\ (int): number of total epochsearly_stop (int): early stop when performance not\
|
|
\ improved at this stepupdate_freq (int): gradient update frequencymax_steps_per_epoch\
|
|
\ (int): maximum number of steps in one epochlamb (float): regularization parameterrho\
|
|
\ (float): exponential decay rate for `lamb`alpha (float): fusion parameter for\
|
|
\ calculating transport loss matrixseed (int): random seedlogdir (str): local\
|
|
\ log directoryeval_train (bool): whether evaluate train set between epochseval_test\
|
|
\ (bool): whether evaluate test set between epochspretrain (bool): whether pretrain\
|
|
\ the backbone model before training TRA.Note that only TRA will be optimized\
|
|
\ after pretraininginit_state (str): model init state pathfreeze_model (bool):\
|
|
\ whether freeze backbone model parametersfreeze_predictors (bool): whether freeze\
|
|
\ predictors parameterstransport_method (str): transport method, can be none/router/oraclememory_mode\
|
|
\ (str): memory mode, the same argument for MTSDatasetH"
|
|
function: pytorch_tra.TRAModel
|
|
- docstring: null
|
|
function: pytorch_tra._init_model
|
|
- docstring: null
|
|
function: pytorch_tra.train_epoch
|
|
- docstring: null
|
|
function: pytorch_tra.test_epoch
|
|
- docstring: null
|
|
function: pytorch_tra._fit
|
|
- docstring: null
|
|
function: pytorch_tra.fit
|
|
- docstring: null
|
|
function: pytorch_tra.predict
|
|
- docstring: ' RNN Model
|
|
|
|
Args:input_size (int): input size (# features)hidden_size (int): hidden sizenum_layers
|
|
(int): number of hidden layersrnn_arch (str): rnn architectureuse_attn (bool):
|
|
whether use attention layer.we use concat attention as https://github.com/fulifeng/Adv-ALSTM/dropout
|
|
(float): dropout rate'
|
|
function: pytorch_tra.RNN
|
|
- docstring: null
|
|
function: pytorch_tra.forward
|
|
- docstring: null
|
|
function: pytorch_tra.PositionalEncoding
|
|
- docstring: null
|
|
function: pytorch_tra.forward
|
|
- docstring: ' Transformer Model
|
|
|
|
Args:input_size (int): input size (# features)hidden_size (int): hidden sizenum_layers
|
|
(int): number of transformer layersnum_heads (int): number of heads in transformerdropout
|
|
(float): dropout rate'
|
|
function: pytorch_tra.Transformer
|
|
- docstring: null
|
|
function: pytorch_tra.forward
|
|
- docstring: ' Temporal Routing Adaptor (TRA)
|
|
|
|
TRA takes historical prediction errors & latent representation as inputs,then
|
|
routes the input sample to a specific predictor for training & inference.Args:input_size
|
|
(int): input size (RNN/Transformer''s hidden size)num_states (int): number of
|
|
latent states (i.e., trading patterns)If `num_states=1`, then TRA falls back to
|
|
traditional methodshidden_size (int): hidden size of the routertau (float): gumbel
|
|
softmax temperaturesrc_info (str): information for the router'
|
|
function: pytorch_tra.TRA
|
|
- docstring: null
|
|
function: pytorch_tra.reset_parameters
|
|
- docstring: null
|
|
function: pytorch_tra.forward
|
|
- docstring: null
|
|
function: pytorch_tra.evaluate
|
|
- docstring: ' Replaces inf by maximum of tensor
|
|
|
|
mask_inf = torch.isinf(inp_tensor)ind_inf = torch.nonzero(mask_inf, as_tuple=False)if
|
|
len(ind_inf) > 0:for ind in ind_inf:if len(ind) == 2:inp_tensor[ind[0], ind[1]]
|
|
= 0elif len(ind) == 1:inp_tensor[ind[0]] = 0m = torch.max(inp_tensor)for ind in
|
|
ind_inf:if len(ind) == 2:inp_tensor[ind[0], ind[1]] = melif len(ind) == 1:inp_tensor[ind[0]]
|
|
= mreturn inp_tensor'
|
|
function: pytorch_tra.shoot_infs
|
|
- docstring: null
|
|
function: pytorch_tra.sinkhorn
|
|
- docstring: null
|
|
function: pytorch_tra.loss_fn
|
|
- docstring: null
|
|
function: pytorch_tra.minmax_norm
|
|
- docstring: " \nsample-wise transportArgs:all_preds (torch.Tensor): predictions\
|
|
\ from all predictors, [sample x states]label (torch.Tensor): label, [sample]choice\
|
|
\ (torch.Tensor): gumbel softmax choice, [sample x states]prob (torch.Tensor):\
|
|
\ router predicted probility, [sample x states]hist_loss (torch.Tensor): history\
|
|
\ loss matrix, [sample x states]count (list): sample counts for each day, empty\
|
|
\ list for sample-wise transporttransport_method (str): transportation methodalpha\
|
|
\ (float): fusion parameter for calculating transport loss matrixtraining (bool):\
|
|
\ indicate training or inference"
|
|
function: pytorch_tra.transport_sample
|
|
- docstring: " \ndaily transportArgs:all_preds (torch.Tensor): predictions from\
|
|
\ all predictors, [sample x states]label (torch.Tensor): label, [sample]choice\
|
|
\ (torch.Tensor): gumbel softmax choice, [days x states]prob (torch.Tensor): router\
|
|
\ predicted probility, [days x states]hist_loss (torch.Tensor): history loss matrix,\
|
|
\ [days x states]count (list): sample counts for each day, [days]transport_method\
|
|
\ (str): transportation methodalpha (float): fusion parameter for calculating\
|
|
\ transport loss matrixtraining (bool): indicate training or inference"
|
|
function: pytorch_tra.transport_daily
|
|
- docstring: " \nLoad state dict to provided model while ignore exceptions."
|
|
function: pytorch_tra.load_state_dict_unsafe
|
|
- docstring: null
|
|
function: pytorch_tra.load
|
|
- docstring: ' IGMTF Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_igmtf.IGMTF
|
|
- docstring: null
|
|
function: pytorch_igmtf.use_gpu
|
|
- docstring: null
|
|
function: pytorch_igmtf.mse
|
|
- docstring: null
|
|
function: pytorch_igmtf.loss_fn
|
|
- docstring: null
|
|
function: pytorch_igmtf.metric_fn
|
|
- docstring: null
|
|
function: pytorch_igmtf.get_daily_inter
|
|
- docstring: null
|
|
function: pytorch_igmtf.get_train_hidden
|
|
- docstring: null
|
|
function: pytorch_igmtf.train_epoch
|
|
- docstring: null
|
|
function: pytorch_igmtf.test_epoch
|
|
- docstring: null
|
|
function: pytorch_igmtf.fit
|
|
- docstring: null
|
|
function: pytorch_igmtf.predict
|
|
- docstring: null
|
|
function: pytorch_igmtf.IGMTFModel
|
|
- docstring: null
|
|
function: pytorch_igmtf.cal_cos_similarity
|
|
- docstring: null
|
|
function: pytorch_igmtf.sparse_dense_mul
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.TransformerModel
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.use_gpu
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.mse
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.loss_fn
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.metric_fn
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.train_epoch
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.test_epoch
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.fit
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.predict
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.PositionalEncoding
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.forward
|
|
- docstring: null
|
|
function: pytorch_transformer_ts.Transformer
|
|
- docstring: ' HIST Model
|
|
|
|
Parameters----------lr : floatlearning rated_feat : intinput dimensions for each
|
|
time stepmetric : strthe evaluation metric used in early stopoptimizer : stroptimizer
|
|
nameGPU : strthe GPU ID(s) used for training'
|
|
function: pytorch_hist.HIST
|
|
- docstring: null
|
|
function: pytorch_hist.use_gpu
|
|
- docstring: null
|
|
function: pytorch_hist.mse
|
|
- docstring: null
|
|
function: pytorch_hist.loss_fn
|
|
- docstring: null
|
|
function: pytorch_hist.metric_fn
|
|
- docstring: null
|
|
function: pytorch_hist.get_daily_inter
|
|
- docstring: null
|
|
function: pytorch_hist.train_epoch
|
|
- docstring: null
|
|
function: pytorch_hist.test_epoch
|
|
- docstring: null
|
|
function: pytorch_hist.fit
|
|
- docstring: null
|
|
function: pytorch_hist.predict
|
|
- docstring: null
|
|
function: pytorch_hist.HISTModel
|
|
- docstring: null
|
|
function: pytorch_hist.cal_cos_similarity
|
|
- docstring: ' LightGBM Model
|
|
|
|
if loss not in {"mse", "binary"}:raise NotImplementedErrorself.params = {"objective":
|
|
loss, "verbosity": -1}self.params.update(kwargs)self.early_stopping_rounds = early_stopping_roundsself.num_boost_round
|
|
= num_boost_roundself.model = None'
|
|
function: gbdt.LGBModel
|
|
- docstring: " \nThe motivation of current version is to make validation optional-\
|
|
\ train segment is necessary;"
|
|
function: gbdt._prepare_data
|
|
- docstring: null
|
|
function: gbdt.fit
|
|
- docstring: null
|
|
function: gbdt.predict
|
|
- docstring: " \nfinetune modelParameters----------dataset : DatasetHdataset\
|
|
\ for finetuningnum_boost_round : intnumber of round to finetune modelverbose_eval\
|
|
\ : intverbose level"
|
|
function: gbdt.finetune
|
|
- docstring: ' TCTS Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_tcts.TCTS
|
|
- docstring: null
|
|
function: pytorch_tcts.loss_fn
|
|
- docstring: null
|
|
function: pytorch_tcts.train_epoch
|
|
- docstring: null
|
|
function: pytorch_tcts.test_epoch
|
|
- docstring: null
|
|
function: pytorch_tcts.fit
|
|
- docstring: null
|
|
function: pytorch_tcts.training
|
|
- docstring: null
|
|
function: pytorch_tcts.predict
|
|
- docstring: null
|
|
function: pytorch_tcts.MLPModel
|
|
- docstring: null
|
|
function: pytorch_tcts.forward
|
|
- docstring: null
|
|
function: pytorch_tcts.GRUModel
|
|
- docstring: ' LSTM Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_lstm_ts.LSTM
|
|
- docstring: null
|
|
function: pytorch_lstm_ts.use_gpu
|
|
- docstring: null
|
|
function: pytorch_lstm_ts.mse
|
|
- docstring: null
|
|
function: pytorch_lstm_ts.loss_fn
|
|
- docstring: null
|
|
function: pytorch_lstm_ts.metric_fn
|
|
- docstring: null
|
|
function: pytorch_lstm_ts.train_epoch
|
|
- docstring: null
|
|
function: pytorch_lstm_ts.test_epoch
|
|
- docstring: null
|
|
function: pytorch_lstm_ts.fit
|
|
- docstring: null
|
|
function: pytorch_lstm_ts.predict
|
|
- docstring: null
|
|
function: pytorch_lstm_ts.LSTMModel
|
|
- docstring: ' ALSTM Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : intthe
|
|
GPU ID used for training'
|
|
function: pytorch_alstm.ALSTM
|
|
- docstring: null
|
|
function: pytorch_alstm.use_gpu
|
|
- docstring: null
|
|
function: pytorch_alstm.mse
|
|
- docstring: null
|
|
function: pytorch_alstm.loss_fn
|
|
- docstring: null
|
|
function: pytorch_alstm.metric_fn
|
|
- docstring: null
|
|
function: pytorch_alstm.train_epoch
|
|
- docstring: null
|
|
function: pytorch_alstm.test_epoch
|
|
- docstring: null
|
|
function: pytorch_alstm.fit
|
|
- docstring: null
|
|
function: pytorch_alstm.predict
|
|
- docstring: null
|
|
function: pytorch_alstm.ALSTMModel
|
|
- docstring: null
|
|
function: pytorch_alstm._build_model
|
|
- docstring: ' CatBoost Model
|
|
|
|
# There are more optionsif loss not in {"RMSE", "Logloss"}:raise NotImplementedErrorself._params
|
|
= {"loss_function": loss}self._params.update(kwargs)self.model = None'
|
|
function: catboost_model.CatBoostModel
|
|
- docstring: null
|
|
function: catboost_model.fit
|
|
- docstring: null
|
|
function: catboost_model.predict
|
|
- docstring: ' get feature importance
|
|
|
|
Notes-----parameters references:https://catboost.ai/docs/concepts/python-reference_catboost_get_feature_importance.html#python-reference_catboost_get_feature_importance'
|
|
function: catboost_model.get_feature_importance
|
|
- docstring: " \nTabNet model for QlibArgs:ps: probability to generate the\
|
|
\ bernoulli mask"
|
|
function: pytorch_tabnet.TabnetModel
|
|
- docstring: null
|
|
function: pytorch_tabnet.use_gpu
|
|
- docstring: null
|
|
function: pytorch_tabnet.pretrain_fn
|
|
- docstring: null
|
|
function: pytorch_tabnet.fit
|
|
- docstring: null
|
|
function: pytorch_tabnet.predict
|
|
- docstring: null
|
|
function: pytorch_tabnet.test_epoch
|
|
- docstring: null
|
|
function: pytorch_tabnet.train_epoch
|
|
- docstring: null
|
|
function: pytorch_tabnet.pretrain_epoch
|
|
- docstring: null
|
|
function: pytorch_tabnet.pretrain_test_epoch
|
|
- docstring: " \nPretrain loss function defined in the original paper, read\
|
|
\ \"Tabular self-supervised learning\" in https://arxiv.org/pdf/1908.07442.pdf"
|
|
function: pytorch_tabnet.pretrain_loss_fn
|
|
- docstring: null
|
|
function: pytorch_tabnet.loss_fn
|
|
- docstring: null
|
|
function: pytorch_tabnet.metric_fn
|
|
- docstring: null
|
|
function: pytorch_tabnet.mse
|
|
- docstring: " \nFinuetuneModel for adding a layer by the end"
|
|
function: pytorch_tabnet.FinetuneModel
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: null
|
|
function: pytorch_tabnet.DecoderStep
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: " \nTabNet decoder that is used in pre-training"
|
|
function: pytorch_tabnet.TabNet_Decoder
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: " \nTabNet AKA the original encoderArgs:n_d: dimension of the\
|
|
\ features used to calculate the final resultsn_a: dimension of the features input\
|
|
\ to the attention transformer of the next stepn_shared: numbr of shared steps\
|
|
\ in feature transformer(optional)n_ind: number of independent steps in feature\
|
|
\ transformern_steps: number of steps of pass through tabbetrelax coefficient:virtual\
|
|
\ batch size:"
|
|
function: pytorch_tabnet.TabNet
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: " \nGhost Batch Normalizationan efficient way of doing batch normalizationArgs:vbs:\
|
|
\ virtual batch size"
|
|
function: pytorch_tabnet.GBN
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: " \nGLU block that extracts only the most essential informationArgs:vbs:\
|
|
\ virtual batch size"
|
|
function: pytorch_tabnet.GLU
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: " \nArgs:relax: relax coefficient. The greater it is, we canuse the\
|
|
\ same features more. When it is set to 1we can use every feature only once"
|
|
function: pytorch_tabnet.AttentionTransformer
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: null
|
|
function: pytorch_tabnet.FeatureTransformer
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: " \nOne step for the TabNet"
|
|
function: pytorch_tabnet.DecisionStep
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: null
|
|
function: pytorch_tabnet.make_ix_like
|
|
- docstring: " \nSparseMax function for replacing reLU"
|
|
function: pytorch_tabnet.SparsemaxFunction
|
|
- docstring: null
|
|
function: pytorch_tabnet.forward
|
|
- docstring: null
|
|
function: pytorch_tabnet.backward
|
|
- docstring: ' ALSTM Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : intthe
|
|
GPU ID used for training'
|
|
function: pytorch_alstm_ts.ALSTM
|
|
- docstring: null
|
|
function: pytorch_alstm_ts.use_gpu
|
|
- docstring: null
|
|
function: pytorch_alstm_ts.mse
|
|
- docstring: null
|
|
function: pytorch_alstm_ts.loss_fn
|
|
- docstring: null
|
|
function: pytorch_alstm_ts.metric_fn
|
|
- docstring: null
|
|
function: pytorch_alstm_ts.train_epoch
|
|
- docstring: null
|
|
function: pytorch_alstm_ts.test_epoch
|
|
- docstring: null
|
|
function: pytorch_alstm_ts.fit
|
|
- docstring: null
|
|
function: pytorch_alstm_ts.predict
|
|
- docstring: null
|
|
function: pytorch_alstm_ts.ALSTMModel
|
|
- docstring: null
|
|
function: pytorch_alstm_ts._build_model
|
|
- docstring: ' GRU Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_gru_ts.GRU
|
|
- docstring: null
|
|
function: pytorch_gru_ts.use_gpu
|
|
- docstring: null
|
|
function: pytorch_gru_ts.mse
|
|
- docstring: null
|
|
function: pytorch_gru_ts.loss_fn
|
|
- docstring: null
|
|
function: pytorch_gru_ts.metric_fn
|
|
- docstring: null
|
|
function: pytorch_gru_ts.train_epoch
|
|
- docstring: null
|
|
function: pytorch_gru_ts.test_epoch
|
|
- docstring: null
|
|
function: pytorch_gru_ts.fit
|
|
- docstring: null
|
|
function: pytorch_gru_ts.predict
|
|
- docstring: null
|
|
function: pytorch_gru_ts.GRUModel
|
|
- docstring: ' Build a Sandwich model
|
|
|
|
Parameters----------fea_dim : intThe feature dimensioncnn_dim_1 : intThe hidden
|
|
dimension of the first CNNcnn_dim_2 : intThe hidden dimension of the second CNNcnn_kernel_size
|
|
: intThe size of convolutional kernelsrnn_dim_1 : intThe hidden dimension of the
|
|
first KRNNrnn_dim_2 : intThe hidden dimension of the second KRNNrnn_dups : intThe
|
|
number of parallel duplicatesrnn_layers: intThe number of RNN layers'
|
|
function: pytorch_sandwich.SandwichModel
|
|
- docstring: null
|
|
function: pytorch_sandwich.forward
|
|
- docstring: ' Sandwich Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_sandwich.Sandwich
|
|
- docstring: null
|
|
function: pytorch_sandwich.use_gpu
|
|
- docstring: null
|
|
function: pytorch_sandwich.mse
|
|
- docstring: null
|
|
function: pytorch_sandwich.loss_fn
|
|
- docstring: null
|
|
function: pytorch_sandwich.metric_fn
|
|
- docstring: null
|
|
function: pytorch_sandwich.train_epoch
|
|
- docstring: null
|
|
function: pytorch_sandwich.test_epoch
|
|
- docstring: null
|
|
function: pytorch_sandwich.fit
|
|
- docstring: ' ADARNN Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_adarnn.ADARNN
|
|
- docstring: null
|
|
function: pytorch_adarnn.use_gpu
|
|
- docstring: null
|
|
function: pytorch_adarnn.train_AdaRNN
|
|
- docstring: ' pred is a pandas dataframe that has two attributes: score (pred)
|
|
and label (real)
|
|
|
|
res = {}ic = pred.groupby(level="datetime").apply(lambda x: x.label.corr(x.score))rank_ic
|
|
= pred.groupby(level="datetime").apply(lambda x: x.label.corr(x.score, method="spearman"))res["ic"]
|
|
= ic.mean()res["icir"] = ic.mean() / ic.std()res["ric"] = rank_ic.mean()res["ricir"]
|
|
= rank_ic.mean() / rank_ic.std()res["mse"] = -(pred["label"] - pred["score"]).mean()res["loss"]
|
|
= res["mse"]return res'
|
|
function: pytorch_adarnn.calc_all_metrics
|
|
- docstring: null
|
|
function: pytorch_adarnn.test_epoch
|
|
- docstring: null
|
|
function: pytorch_adarnn.log_metrics
|
|
- docstring: null
|
|
function: pytorch_adarnn.fit
|
|
- docstring: null
|
|
function: pytorch_adarnn.predict
|
|
- docstring: null
|
|
function: pytorch_adarnn.infer
|
|
- docstring: null
|
|
function: pytorch_adarnn.transform_type
|
|
- docstring: null
|
|
function: pytorch_adarnn.data_loader
|
|
- docstring: null
|
|
function: pytorch_adarnn.get_stock_loader
|
|
- docstring: null
|
|
function: pytorch_adarnn.get_index
|
|
- docstring: " \nmodel_type: 'Boosting', 'AdaRNN'"
|
|
function: pytorch_adarnn.AdaRNN
|
|
- docstring: null
|
|
function: pytorch_adarnn.init_layers
|
|
- docstring: null
|
|
function: pytorch_adarnn.forward_pre_train
|
|
- docstring: null
|
|
function: pytorch_adarnn.gru_features
|
|
- docstring: null
|
|
function: pytorch_adarnn.process_gate_weight
|
|
- docstring: null
|
|
function: pytorch_adarnn.get_features
|
|
- docstring: null
|
|
function: pytorch_adarnn.forward_Boosting
|
|
- docstring: null
|
|
function: pytorch_adarnn.update_weight_Boosting
|
|
- docstring: null
|
|
function: pytorch_adarnn.predict
|
|
- docstring: " \nSupported loss_type: mmd(mmd_lin), mmd_rbf, coral, cosine,\
|
|
\ kl, js, mine, adv"
|
|
function: 'pytorch_adarnn.TransferLoss:'
|
|
- docstring: ' Compute adaptation loss
|
|
|
|
Arguments:X {tensor} -- source matrixY {tensor} -- target matrixReturns:[tensor]
|
|
-- transfer loss'
|
|
function: pytorch_adarnn.compute
|
|
- docstring: null
|
|
function: pytorch_adarnn.cosine
|
|
- docstring: null
|
|
function: pytorch_adarnn.ReverseLayerF
|
|
- docstring: null
|
|
function: pytorch_adarnn.forward
|
|
- docstring: null
|
|
function: pytorch_adarnn.backward
|
|
- docstring: null
|
|
function: pytorch_adarnn.Discriminator
|
|
- docstring: null
|
|
function: pytorch_adarnn.forward
|
|
- docstring: null
|
|
function: pytorch_adarnn.adv
|
|
- docstring: null
|
|
function: pytorch_adarnn.CORAL
|
|
- docstring: null
|
|
function: pytorch_adarnn.MMD_loss
|
|
- docstring: null
|
|
function: pytorch_adarnn.guassian_kernel
|
|
- docstring: null
|
|
function: pytorch_adarnn.linear_mmd
|
|
- docstring: null
|
|
function: pytorch_adarnn.forward
|
|
- docstring: null
|
|
function: pytorch_adarnn.Mine_estimator
|
|
- docstring: null
|
|
function: pytorch_adarnn.forward
|
|
- docstring: null
|
|
function: pytorch_adarnn.Mine
|
|
- docstring: null
|
|
function: pytorch_adarnn.forward
|
|
- docstring: null
|
|
function: pytorch_adarnn.pairwise_dist
|
|
- docstring: null
|
|
function: pytorch_adarnn.pairwise_dist_np
|
|
- docstring: null
|
|
function: pytorch_adarnn.pa
|
|
- docstring: null
|
|
function: pytorch_adarnn.kl_div
|
|
- docstring: ' TCN Model
|
|
|
|
Parameters----------d_feat : intinput dimension for each time stepmetric: strthe
|
|
evaluation metric used in early stopoptimizer : stroptimizer nameGPU : strthe
|
|
GPU ID(s) used for training'
|
|
function: pytorch_tcn_ts.TCN
|
|
- docstring: null
|
|
function: pytorch_tcn_ts.use_gpu
|
|
- docstring: null
|
|
function: pytorch_tcn_ts.mse
|
|
- docstring: null
|
|
function: pytorch_tcn_ts.loss_fn
|
|
- docstring: null
|
|
function: pytorch_tcn_ts.metric_fn
|
|
- docstring: null
|
|
function: pytorch_tcn_ts.train_epoch
|
|
- docstring: null
|
|
function: pytorch_tcn_ts.test_epoch
|
|
- docstring: null
|
|
function: pytorch_tcn_ts.fit
|
|
- docstring: null
|
|
function: pytorch_tcn_ts.predict
|
|
- docstring: null
|
|
function: pytorch_tcn_ts.TCNModel
|
|
- docstring: null
|
|
function: pytorch_transformer.TransformerModel
|
|
- docstring: null
|
|
function: pytorch_transformer.use_gpu
|
|
- docstring: null
|
|
function: pytorch_transformer.mse
|
|
- docstring: null
|
|
function: pytorch_transformer.loss_fn
|
|
- docstring: null
|
|
function: pytorch_transformer.metric_fn
|
|
- docstring: null
|
|
function: pytorch_transformer.train_epoch
|
|
- docstring: null
|
|
function: pytorch_transformer.test_epoch
|
|
- docstring: null
|
|
function: pytorch_transformer.fit
|
|
- docstring: null
|
|
function: pytorch_transformer.predict
|
|
- docstring: null
|
|
function: pytorch_transformer.PositionalEncoding
|
|
- docstring: null
|
|
function: pytorch_transformer.forward
|
|
- docstring: null
|
|
function: pytorch_transformer.Transformer
|
|
- docstring: " \nThe motivation of Rolling Module- It only focus **offlinely**\
|
|
\ turn a specific task to rollinng- To make the implementation easier, following\
|
|
\ factors are ignored.- The tasks is dependent (e.g. time series).Related modules\
|
|
\ and difference from me:- MetaController: It is learning how to handle a task\
|
|
\ (e.g. learning to learn).- But rolling is about how to split a single task into\
|
|
\ tasks in time series and run them.- OnlineStrategy: It is focusing on serving\
|
|
\ a model, the model can be updated time dependently in time.- Rolling is much\
|
|
\ simpler and is only for testing rolling models offline. It does not want to\
|
|
\ share the interface with OnlineStrategy.The code about rolling is shared in\
|
|
\ `task_generator` & `RollingGen` level between me and the above modulesBut it\
|
|
\ is for different purpose, so other parts are not shared... code-block:: shell#\
|
|
\ here is an typical use case of the module.python -m qlib.contrib.rolling.base\
|
|
\ --conf_path <path to the yaml> run**NOTE**before running the example, please\
|
|
\ clean your previous results with following command- `rm -r mlruns`- Because\
|
|
\ it is very hard to permanently delete a experiment (it will be moved into .trash\
|
|
\ and raise error when creating experiment with same name)."
|
|
function: 'base.Rolling:'
|
|
- docstring: null
|
|
function: base._raw_conf
|
|
- docstring: " \nDue to the data processing part in original rolling is slow.\
|
|
\ So we have toThis class tries to add more feature"
|
|
function: base._replace_hanler_with_cache
|
|
- docstring: null
|
|
function: base._update_start_end_time
|
|
- docstring: " \nThe basic task may not be the exactly same as the config from\
|
|
\ `conf_path` from __init__ due to- some parameters could be overriding by some\
|
|
\ parameters from __init__- user could implementing sublcass to change it for\
|
|
\ higher performance"
|
|
function: base.basic_task
|
|
- docstring: ' return a batch of tasks for rolling.
|
|
|
|
task = self.basic_task()task_l = task_generator(task, RollingGen(step=self.step,
|
|
trunc_days=self.horizon + 1)) # the last two days should be truncated to avoid
|
|
information leakagefor t in task_l:# when we rolling tasks. No further analyis
|
|
is needed.# analyis are postponed to the final ensemble.t["record"] = ["qlib.workflow.record_temp.SignalRecord"]return
|
|
task_l'
|
|
function: base.get_task_list
|
|
- docstring: null
|
|
function: base._train_rolling_tasks
|
|
- docstring: null
|
|
function: base._ens_rolling
|
|
- docstring: " \nEvaluate the combined rolling results"
|
|
function: base._update_rolling_rec
|
|
- docstring: " \nIt is a rolling based on DDG-DA**NOTE**before running the example,\
|
|
\ please clean your previous results with following command- `rm -r mlruns`"
|
|
function: ddgda.DDGDA
|
|
- docstring: " \nsome task are use for special purpose.For example:- GBDT for\
|
|
\ calculating feature importance- Linear or GBDT for calculating similarity- Datset\
|
|
\ (well processed) that aligned to Linear that for meta learning"
|
|
function: ddgda._adjust_task
|
|
- docstring: null
|
|
function: ddgda._get_feature_importance
|
|
- docstring: " \nDump data for training meta model.The meta model will be trained\
|
|
\ upon the proxy forecasting model.This dataset is for the proxy forecasting model."
|
|
function: ddgda._dump_data_for_proxy_model
|
|
- docstring: null
|
|
function: ddgda._internal_data_path
|
|
- docstring: " \nDump data for training meta model.This function will dump\
|
|
\ the input data for meta model"
|
|
function: ddgda._dump_meta_ipt
|
|
- docstring: " \ntraining a meta model based on a simplified linear proxy model;"
|
|
function: ddgda._train_meta_model
|
|
- docstring: null
|
|
function: ddgda._task_path
|
|
- docstring: " \nLeverage meta-model for inference:- Given- baseline tasks-\
|
|
\ input for meta model(internal data)- meta model (its learnt knowledge on proxy\
|
|
\ forecasting model is expected to transfer to normal forecasting model)"
|
|
function: ddgda.get_task_list
|
|
- docstring: " \nSimulator that resets with ``__init__``, and transits with ``step(action)``.To\
|
|
\ make the data-flow clear, we make the following restrictions to Simulator:1.\
|
|
\ The only way to modify the inner status of a simulator is by using ``step(action)``.2.\
|
|
\ External modules can *read* the status of a simulator by using ``simulator.get_state()``,and\
|
|
\ check whether the simulator is in the ending state by calling ``simulator.done()``.A\
|
|
\ simulator is defined to be bounded with three types:- *InitialStateType* that\
|
|
\ is the type of the data used to create the simulator.- *StateType* that is the\
|
|
\ type of the **status** (state) of the simulator.- *ActType* that is the type\
|
|
\ of the **action**, which is the input received in each step.Different simulators\
|
|
\ might share the same StateType. For example, when they are dealing with the\
|
|
\ same task,but with different simulation implementation. With the same type,\
|
|
\ they can safely share other components in the MDP.Simulators are ephemeral.\
|
|
\ The lifecycle of a simulator starts with an initial state, and ends with the\
|
|
\ trajectory.In another word, when the trajectory ends, simulator is recycled.If\
|
|
\ simulators want to share context between (e.g., for speed-up purposes),this\
|
|
\ could be done by accessing the weak reference of environment wrapper.Attributes----------envA\
|
|
\ reference of env-wrapper, which could be useful in some corner cases.Simulators\
|
|
\ are discouraged to use this, because it's prone to induce errors."
|
|
function: simulator.Simulator
|
|
- docstring: ' Receives an action of ActType.
|
|
|
|
Simulator should update its internal state, and return None.The updated state
|
|
can be retrieved with ``simulator.get_state()``.'
|
|
function: simulator.step
|
|
- docstring: null
|
|
function: simulator.get_state
|
|
- docstring: ' Check whether the simulator is in a "done" state.
|
|
|
|
When simulator is in a "done" state,it should no longer receives any ``step``
|
|
request.As simulators are ephemeral, to reset the simulator,the old one should
|
|
be destroyed and a new simulator can be created.'
|
|
function: simulator.done
|
|
- docstring: " \nReward calculation component that takes a single argument: state\
|
|
\ of simulator. Returns a real number: reward.Subclass should implement ``reward(simulator_state)``\
|
|
\ to implement their own reward calculation recipe."
|
|
function: reward.Reward
|
|
- docstring: ' Implement this method for your own reward.
|
|
|
|
raise NotImplementedError("Implement reward calculation recipe in `reward()`.")'
|
|
function: reward.reward
|
|
- docstring: null
|
|
function: reward.log
|
|
- docstring: ' Combination of multiple reward.
|
|
|
|
self.rewards = rewards'
|
|
function: reward.RewardCombination
|
|
- docstring: ' Override this class to collect customized auxiliary information
|
|
from environment.
|
|
|
|
env: Optional[EnvWrapper] = None@finalreturn self.collect(simulator_state)'
|
|
function: aux_info.AuxiliaryInfoCollector
|
|
- docstring: ' Override this for customized auxiliary info.
|
|
|
|
Usually useful in Multi-agent RL.Parameters----------simulator_stateRetrieved
|
|
with ``simulator.get_state()``.Returns-------Auxiliary information.'
|
|
function: aux_info.collect
|
|
- docstring: ' Interpreter is a media between states produced by simulators and
|
|
states needed by RL policies.
|
|
|
|
Interpreters are two-way:1. From simulator state to policy state (aka observation),
|
|
see :class:`StateInterpreter`.2. From policy action to action accepted by simulator,
|
|
see :class:`ActionInterpreter`.Inherit one of the two sub-classes to define your
|
|
own interpreter.This super-class is only used for isinstance check.Interpreters
|
|
are recommended to be stateless, meaning that storing temporary information with
|
|
``self.xxx``in interpreter is anti-pattern. In future, we might support register
|
|
some interpreter-relatedstates by calling ``self.env.register_state()``, but it''s
|
|
not planned for first iteration.'
|
|
function: 'interpreter.Interpreter:'
|
|
- docstring: ' State Interpreter that interpret execution result of qlib executor
|
|
into rl env state
|
|
|
|
@property'
|
|
function: interpreter.StateInterpreter
|
|
- docstring: null
|
|
function: interpreter.observation_space
|
|
- docstring: ' Validate whether an observation belongs to the pre-defined observation
|
|
space.
|
|
|
|
_gym_space_contains(self.observation_space, obs)'
|
|
function: interpreter.validate
|
|
- docstring: ' Interpret the state of simulator.
|
|
|
|
Parameters----------simulator_stateRetrieved with ``simulator.get_state()``.Returns-------State
|
|
needed by policy. Should conform with the state space defined in ``observation_space``.'
|
|
function: interpreter.interpret
|
|
- docstring: ' Action Interpreter that interpret rl agent action into qlib orders
|
|
|
|
@property'
|
|
function: interpreter.ActionInterpreter
|
|
- docstring: null
|
|
function: interpreter.action_space
|
|
- docstring: ' Validate whether an action belongs to the pre-defined action
|
|
space.
|
|
|
|
_gym_space_contains(self.action_space, action)'
|
|
function: interpreter.validate
|
|
- docstring: ' Convert the policy action to simulator action.
|
|
|
|
Parameters----------simulator_stateRetrieved with ``simulator.get_state()``.actionRaw
|
|
action given by policy.Returns-------The action needed by simulator,'
|
|
function: interpreter.interpret
|
|
- docstring: ' Strengthened version of gym.Space.contains.
|
|
|
|
Giving more diagnostic information on why validation fails.Throw exception rather
|
|
than returning true or false.'
|
|
function: interpreter._gym_space_contains
|
|
- docstring: ' Strategy used to generate a trade decision with exactly one order.
|
|
|
|
self,order: Order,trade_range: TradeRange | None = None,) -> None:super().__init__()self._order
|
|
= orderself._trade_range = trade_range'
|
|
function: single_order.SingleOrderStrategy
|
|
- docstring: ' Main process (producer) produces data and stores them in a queue.
|
|
|
|
Sub-processes (consumers) can retrieve the data-points from the queue.Data-points
|
|
are generated via reading items from ``dataset``.:class:`DataQueue` is ephemeral.
|
|
You must create a new DataQueuewhen the ``repeat`` is exhausted.See the documents
|
|
of :class:`qlib.rl.utils.FiniteVectorEnv` for more background.Parameters----------datasetThe
|
|
dataset to read data from. Must implement ``__len__`` and ``__getitem__``.repeatIterate
|
|
over the data-points for how many times. Use ``-1`` to iterate forever.shuffleIf
|
|
``shuffle`` is true, the items will be read in random order.producer_num_workersConcurrent
|
|
workers for data-loading.queue_maxsizeMaximum items to put into queue before it
|
|
jams.Examples-------->>> data_queue = DataQueue(my_dataset)>>> with data_queue:... ...In
|
|
worker:>>> for data in data_queue:... print(data)'
|
|
function: data_queue.DataQueue
|
|
- docstring: null
|
|
function: data_queue.cleanup
|
|
- docstring: null
|
|
function: data_queue.get
|
|
- docstring: null
|
|
function: data_queue.put
|
|
- docstring: null
|
|
function: data_queue.mark_as_done
|
|
- docstring: null
|
|
function: data_queue.done
|
|
- docstring: null
|
|
function: data_queue.activate
|
|
- docstring: null
|
|
function: data_queue._consumer
|
|
- docstring: null
|
|
function: finite_env.fill_invalid
|
|
- docstring: null
|
|
function: finite_env.is_invalid
|
|
- docstring: ' The NaN observation that indicates the environment receives no seed.
|
|
|
|
We assume that obs is complex and there must be something like float.Otherwise
|
|
this logic doesn''t work.'
|
|
function: finite_env.generate_nan_observation
|
|
- docstring: ' Check whether obs is generated by :func:`generate_nan_observation`.
|
|
|
|
return is_invalid(obs)'
|
|
function: finite_env.check_nan_observation
|
|
- docstring: ' To allow the paralleled env workers consume a single DataQueue until
|
|
it''s exhausted.
|
|
|
|
See `tianshou issue #322 <https://github.com/thu-ml/tianshou/issues/322>`_.The
|
|
requirement is to make every possible seed (stored in :class:`qlib.rl.utils.DataQueue`
|
|
in our case)consumed by exactly one environment. This is not possible by tianshou''s
|
|
native VectorEnv and Collector,because tianshou is unaware of this "exactly one"
|
|
constraint, and might launch extra workers.Consider a corner case, where concurrency
|
|
is 2, but there is only one seed in DataQueue.The reset of two workers must be
|
|
both called according to the logic in collect.The returned results of two workers
|
|
are collected, regardless of what they are.The problem is, one of the reset result
|
|
must be invalid, or repeated,because there''s only one need in queue, and collector
|
|
isn''t aware of such situation.Luckily, we can hack the vector env, and make a
|
|
protocol between single env and vector env.The single environment (should be :class:`qlib.rl.utils.EnvWrapper`
|
|
in our case) is responsible forreading from queue, and generate a special observation
|
|
when the queue is exhausted. The special obsis called "nan observation", because
|
|
simply using none causes problems in shared-memory vector env.:class:`FiniteVectorEnv`
|
|
then read the observations from all workers, and select those non-nanobservation.
|
|
It also maintains an ``_alive_env_ids`` to track which workers should never becalled
|
|
again. When also the environments are exhausted, it will raise StopIteration exception.The
|
|
usage of this vector env in collector are two parts:1. If the data queue is finite
|
|
(usually when inference), collector should collect "infinity" number ofepisodes,
|
|
until the vector env exhausts by itself.2. If the data queue is infinite (usually
|
|
in training), collector can set number of episodes / steps.In this case, data
|
|
would be randomly ordered, and some repetitions wouldn''t matter.One extra function
|
|
of this vector env is that it has a logger that explicitly collects logsfrom child
|
|
workers. See :class:`qlib.rl.utils.LogWriter`.'
|
|
function: finite_env.FiniteVectorEnv
|
|
- docstring: null
|
|
function: finite_env._reset_alive_envs
|
|
- docstring: null
|
|
function: finite_env._set_default_obs
|
|
- docstring: null
|
|
function: finite_env._set_default_info
|
|
- docstring: null
|
|
function: finite_env._set_default_rew
|
|
- docstring: null
|
|
function: finite_env._get_default_obs
|
|
- docstring: null
|
|
function: finite_env._get_default_info
|
|
- docstring: null
|
|
function: finite_env._get_default_rew
|
|
- docstring: null
|
|
function: finite_env._postproc_env_obs
|
|
- docstring: ' Guard the collector. Recommended to guard every collect.
|
|
|
|
This guard is for two purposes.1. Catch and ignore the StopIteration exception,
|
|
which is the stopping signalthrown by FiniteEnv to let tianshou know that ``collector.collect()``
|
|
should exit.2. Notify the loggers that the collect is ready / done what it''s
|
|
ready / done.Examples-------->>> with finite_env.collector_guard():... collector.collect(n_episode=INF)'
|
|
function: finite_env.collector_guard
|
|
- docstring: null
|
|
function: finite_env.reset
|
|
- docstring: null
|
|
function: finite_env.step
|
|
- docstring: null
|
|
function: finite_env.FiniteDummyVectorEnv
|
|
- docstring: null
|
|
function: finite_env.FiniteSubprocVectorEnv
|
|
- docstring: null
|
|
function: finite_env.FiniteShmemVectorEnv
|
|
- docstring: ' Helper function to create a vector env. Can be used to replace usual
|
|
VectorEnv.
|
|
|
|
For example, once you wrote: ::DummyVectorEnv([lambda: gym.make(task) for _ in
|
|
range(env_num)])Now you can replace it with: ::finite_env_factory(lambda: gym.make(task),
|
|
"dummy", env_num, my_logger)By doing such replacement, you have two additional
|
|
features enabled (compared to normal VectorEnv):1. The vector env will check for
|
|
NaN observation and kill the worker when its found.See :class:`FiniteVectorEnv`
|
|
for why we need this.2. A logger to explicit collect logs from environment workers.Parameters----------env_factoryCallable
|
|
to instantiate one single ``gym.Env``.All concurrent workers will have the same
|
|
``env_factory``.env_typedummy or subproc or shmem. Corresponding to`parallelism
|
|
in tianshou <https://tianshou.readthedocs.io/en/master/api/tianshou.env.html#vectorenv>`_.concurrencyConcurrent
|
|
environment workers.loggerLog writers.Warnings--------Please do not use lambda
|
|
expression here for ``env_factory`` as it may create incorrectly-shared instances.Don''t
|
|
do: ::vectorize_env(lambda: EnvWrapper(...), ...)Please do: ::'
|
|
function: finite_env.vectorize_env
|
|
- docstring: ' The type of dict that is used in the 4th return value of ``env.step()``.
|
|
|
|
aux_info: dictAny information depends on auxiliary info collector.'
|
|
function: env_wrapper.InfoDict
|
|
- docstring: " \nThis is the status data structure used in EnvWrapper.The fields\
|
|
\ here are in the semantics of RL.For example, ``obs`` means the observation fed\
|
|
\ into policy.``action`` means the raw action returned by policy."
|
|
function: env_wrapper.EnvWrapperStatus
|
|
- docstring: ' Qlib-based RL environment, subclassing ``gym.Env``.
|
|
|
|
A wrapper of components, including simulator, state-interpreter, action-interpreter,
|
|
reward.This is what the framework of simulator - interpreter - policy looks like
|
|
in RL training.All the components other than policy needs to be assembled into
|
|
a single object called "environment".The "environment" are replicated into multiple
|
|
workers, and (at least in tianshou''s implementation),one single policy (agent)
|
|
plays against a batch of environments.Parameters----------simulator_fnA callable
|
|
that is the simulator factory.When ``seed_iterator`` is present, the factory should
|
|
take one argument,that is the seed (aka initial state).Otherwise, it should take
|
|
zero argument.state_interpreterState-observation converter.action_interpreterPolicy-simulator
|
|
action converter.seed_iteratorAn iterable of seed. With the help of :class:`qlib.rl.utils.DataQueue`,environment
|
|
workers in different processes can share one ``seed_iterator``.reward_fnA callable
|
|
that accepts the StateType and returns a float (at least in single-agent case).aux_info_collectorCollect
|
|
auxiliary information. Could be useful in MARL.loggerLog collector that collects
|
|
the logs. The collected logs are sent back to main process,via the return value
|
|
of ``env.step()``.Attributes----------status : EnvWrapperStatusStatus indicator.
|
|
All terms are in *RL language*.It can be used if users care about data on the
|
|
RL side.Can be none when no trajectory is available.'
|
|
function: env_wrapper.EnvWrapper
|
|
- docstring: null
|
|
function: env_wrapper.action_space
|
|
- docstring: null
|
|
function: env_wrapper.observation_space
|
|
- docstring: " \nTry to get a state from state queue, and init the simulator\
|
|
\ with this state.If the queue is exhausted, generate an invalid (nan) observation."
|
|
function: env_wrapper.reset
|
|
- docstring: ' Environment step.
|
|
|
|
See the code along with comments to get a sequence of things happening here.'
|
|
function: env_wrapper.step
|
|
- docstring: ' Log-levels for RL training.
|
|
|
|
The behavior of handling each log level depends on the implementation of :class:`LogWriter`.'
|
|
function: log.LogLevel
|
|
- docstring: ' Logs are first collected in each environment worker,
|
|
|
|
and then aggregated to stream at the central thread in vector env.In :class:`LogCollector`,
|
|
every metric is added to a dict, which needs to be ``reset()`` at each step.The
|
|
dict is sent via the ``info`` in ``env.step()``, and decoded by the :class:`LogWriter`
|
|
at vector env.``min_loglevel`` is for optimization purposes: to avoid too much
|
|
traffic on networks / in pipe.'
|
|
function: 'log.LogCollector:'
|
|
- docstring: ' Clear all collected contents.
|
|
|
|
self._logged = {}'
|
|
function: log.reset
|
|
- docstring: null
|
|
function: log._add_metric
|
|
- docstring: ' Add a string with name into logged contents.
|
|
|
|
if loglevel < self._min_loglevel:returnif not isinstance(string, str):raise TypeError(f"{string}
|
|
is not a string.")self._add_metric(name, string, loglevel)'
|
|
function: log.add_string
|
|
- docstring: ' Add a scalar with name into logged contents.
|
|
|
|
Scalar will be converted into a float.'
|
|
function: log.add_scalar
|
|
- docstring: ' Add an array with name into logging.
|
|
|
|
if loglevel < self._min_loglevel:returnif not isinstance(array, (np.ndarray, pd.DataFrame,
|
|
pd.Series)):raise TypeError(f"{array} is not one of ndarray, DataFrame and Series.")self._add_metric(name,
|
|
array, loglevel)'
|
|
function: log.add_array
|
|
- docstring: ' Log something with any type.
|
|
|
|
As it''s an "any" object, the only LogWriter accepting it is pickle.Therefore,
|
|
pickle must be able to serialize it.'
|
|
function: log.add_any
|
|
- docstring: null
|
|
function: log.logs
|
|
- docstring: ' Base class for log writers, triggered at every reset and step by
|
|
finite env.
|
|
|
|
What to do with a specific log depends on the implementation of subclassing :class:`LogWriter`.The
|
|
general principle is that, it should handle logs above its loglevel (inclusive),and
|
|
discard logs that are not acceptable. For instance, console loggers obviously
|
|
can''t handle an image.'
|
|
function: log.LogWriter
|
|
- docstring: ' Clear all the metrics for a fresh start.
|
|
|
|
To make the logger instance reusable.'
|
|
function: log.clear
|
|
- docstring: ' Save the states of the logger to a dict.
|
|
|
|
return {"episode_count": self.episode_count,"step_count": self.step_count,"global_step":
|
|
self.global_step,"global_episode": self.global_episode,"active_env_ids": self.active_env_ids,"episode_lengths":
|
|
self.episode_lengths,"episode_rewards": self.episode_rewards,"episode_logs": self.episode_logs,}'
|
|
function: log.state_dict
|
|
- docstring: ' Load the states of current logger from a dict.
|
|
|
|
self.episode_count = state_dict["episode_count"]self.step_count = state_dict["step_count"]self.global_step
|
|
= state_dict["global_step"]self.global_episode = state_dict["global_episode"]#
|
|
These are runtime infos.# Though they are loaded, I don''t think it really helps.self.active_env_ids
|
|
= state_dict["active_env_ids"]self.episode_lengths = state_dict["episode_lengths"]self.episode_rewards
|
|
= state_dict["episode_rewards"]self.episode_logs = state_dict["episode_logs"]@staticmethod'
|
|
function: log.load_state_dict
|
|
- docstring: ' Aggregation function from step-wise to episode-wise.
|
|
|
|
If it''s a sequence of float, take the mean.Otherwise, take the first element.If
|
|
a name is specified and,- if it''s ``reward``, the reduction will be sum.'
|
|
function: log.aggregation
|
|
- docstring: ' This is triggered at the end of each trajectory.
|
|
|
|
Parameters----------lengthLength of this trajectory.rewardsA list of rewards at
|
|
each step of this episode.contentsLogged contents for every step.'
|
|
function: log.log_episode
|
|
- docstring: ' This is triggered at each step.
|
|
|
|
Parameters----------rewardReward for this step.contentsLogged contents for this
|
|
step.'
|
|
function: log.log_step
|
|
- docstring: ' Callback for finite env, on each step.
|
|
|
|
# Update counterself.global_step += 1self.step_count += 1self.active_env_ids.add(env_id)self.episode_lengths[env_id]
|
|
+= 1# TODO: reward can be a list of list for MARLself.episode_rewards[env_id].append(rew)values:
|
|
Dict[str, Any] = {}for key, (loglevel, value) in info["log"].items():if loglevel
|
|
>= self.loglevel: # FIXME: this is actually incorrect (see last FIXME)values[key]
|
|
= valueself.episode_logs[env_id].append(values)self.log_step(rew, values)if done:#
|
|
Update counterself.global_episode += 1self.episode_count += 1self.log_episode(self.episode_lengths[env_id],
|
|
self.episode_rewards[env_id], self.episode_logs[env_id])'
|
|
function: log.on_env_step
|
|
- docstring: ' Callback for finite env.
|
|
|
|
Reset episode statistics. Nothing task-specific is logged here because of`a limitation
|
|
of tianshou <https://github.com/thu-ml/tianshou/issues/605>`__.'
|
|
function: log.on_env_reset
|
|
- docstring: ' When all environments are ready to run.
|
|
|
|
Usually, loggers should be reset here.'
|
|
function: log.on_env_all_ready
|
|
- docstring: ' All done. Time for cleanup.
|
|
|
|
'
|
|
function: log.on_env_all_done
|
|
- docstring: ' Keep all numbers in memory.
|
|
|
|
Objects that can''t be aggregated like strings, tensors, images can''t be stored
|
|
in the buffer.To persist them, please use :class:`PickleWriter`.Every time, Log
|
|
buffer receives a new metric, the callback is triggered,which is useful when tracking
|
|
metrics inside a trainer.Parameters----------callbackA callback receiving three
|
|
arguments:- on_episode: Whether it''s called at the end of an episode- on_collect:
|
|
Whether it''s called at the end of a collect- log_buffer: the :class:`LogBbuffer`
|
|
objectNo return value is expected.'
|
|
function: log.LogBuffer
|
|
- docstring: null
|
|
function: log.state_dict
|
|
- docstring: null
|
|
function: log.load_state_dict
|
|
- docstring: null
|
|
function: log.clear
|
|
- docstring: null
|
|
function: log.log_episode
|
|
- docstring: null
|
|
function: log.on_env_all_done
|
|
- docstring: ' Retrieve the numeric metrics of the latest episode.
|
|
|
|
if self._latest_metrics is None:raise ValueError("No episode metrics available
|
|
yet.")return self._latest_metrics'
|
|
function: log.episode_metrics
|
|
- docstring: ' Retrieve the aggregated metrics of the latest collect.
|
|
|
|
return {name: value / self.episode_count for name, value in self._aggregated_metrics.items()}'
|
|
function: log.collect_metrics
|
|
- docstring: ' Write log messages to console periodically.
|
|
|
|
It tracks an average meter for each metric, which is the average value since last
|
|
``clear()`` till now.The display format for each metric is ``<name> <latest_value>
|
|
(<average_value>)``.Non-single-number metrics are auto skipped.'
|
|
function: log.ConsoleWriter
|
|
- docstring: null
|
|
function: log.clear
|
|
- docstring: null
|
|
function: log.log_episode
|
|
- docstring: null
|
|
function: log.generate_log_message
|
|
- docstring: ' Dump all episode metrics to a ``result.csv``.
|
|
|
|
This is not the correct implementation. It''s only used for first iteration.'
|
|
function: log.CsvWriter
|
|
- docstring: null
|
|
function: log.clear
|
|
- docstring: null
|
|
function: log.log_episode
|
|
- docstring: null
|
|
function: log.on_env_all_done
|
|
- docstring: ' Dump logs to pickle files.
|
|
|
|
'
|
|
function: log.PickleWriter
|
|
- docstring: ' Write logs to event files that can be visualized with tensorboard.
|
|
|
|
'
|
|
function: log.TensorboardWriter
|
|
- docstring: " \nRaw market data that is often used in backtesting (thus called\
|
|
\ BacktestData).Base class for all types of backtest data. Currently, each type\
|
|
\ of simulator has its corresponding backtestdata type."
|
|
function: 'base.BaseIntradayBacktestData:'
|
|
- docstring: null
|
|
function: base.get_deal_price
|
|
- docstring: null
|
|
function: base.get_volume
|
|
- docstring: null
|
|
function: base.get_time_index
|
|
- docstring: ' Processed market data after data cleanup and feature engineering.
|
|
|
|
It contains both processed data for "today" and "yesterday", as some algorithmsmight
|
|
use the market information of the previous day to assist decision making.'
|
|
function: 'base.BaseIntradayProcessedData:'
|
|
- docstring: ' Provider of processed data
|
|
|
|
'
|
|
function: 'base.ProcessedDataProvider:'
|
|
- docstring: null
|
|
function: native.get_ticks_slice
|
|
- docstring: ' Backtest data for Qlib simulator
|
|
|
|
self,order: Order,exchange: Exchange,ticks_index: pd.DatetimeIndex,ticks_for_order:
|
|
pd.DatetimeIndex,) -> None:self._order = orderself._exchange = exchangeself._start_time
|
|
= ticks_for_order[0]self._end_time = ticks_for_order[-1]self.ticks_index = ticks_indexself.ticks_for_order
|
|
= ticks_for_orderself._deal_price = cast(pd.Series,self._exchange.get_deal_price(self._order.stock_id,self._start_time,self._end_time,direction=self._order.direction,method=None,),)self._volume
|
|
= cast(pd.Series,self._exchange.get_volume(self._order.stock_id,self._start_time,self._end_time,method=None,),)return
|
|
(f"Order: {self._order}, Exchange: {self._exchange}, "f"Start time: {self._start_time},
|
|
End time: {self._end_time}")return len(self._deal_price)'
|
|
function: native.IntradayBacktestData
|
|
- docstring: null
|
|
function: native.get_deal_price
|
|
- docstring: null
|
|
function: native.get_volume
|
|
- docstring: null
|
|
function: native.get_time_index
|
|
- docstring: ' Backtest data from dataframe
|
|
|
|
self.df = dfself.price_column = price_columnself.volume_column = volume_columnwith
|
|
pd.option_context("memory_usage", False, "display.max_info_columns", 1, "display.large_repr",
|
|
"info"):return f"{self.__class__.__name__}({self.df})"return len(self.df)'
|
|
function: native.DataframeIntradayBacktestData
|
|
- docstring: null
|
|
function: native.get_deal_price
|
|
- docstring: null
|
|
function: native.get_volume
|
|
- docstring: null
|
|
function: native.get_time_index
|
|
- docstring: null
|
|
function: native.load_backtest_data
|
|
- docstring: ' Subclass of IntradayProcessedData. Used to handle handler (bin format)
|
|
style data.
|
|
|
|
self,data_dir: Path,stock_id: str,date: pd.Timestamp,feature_columns_today: List[str],feature_columns_yesterday:
|
|
List[str],backtest: bool = False,index_only: bool = False,) -> None:'
|
|
function: native.HandlerIntradayProcessedData
|
|
- docstring: null
|
|
function: native._drop_stock_id
|
|
- docstring: null
|
|
function: native.load_handler_intraday_processed_data
|
|
- docstring: null
|
|
function: native.HandlerProcessedDataProvider
|
|
- docstring: ' Initialize necessary resource to launch the workflow, including
|
|
data direction, feature columns, etc..
|
|
|
|
Parameters----------qlib_config:Qlib configuration.Example::{"provider_uri_day":
|
|
DATA_ROOT_DIR / "qlib_1d","provider_uri_1min": DATA_ROOT_DIR / "qlib_1min","feature_root_dir":
|
|
DATA_ROOT_DIR / "qlib_handler_stock","feature_columns_today": ["$open", "$high",
|
|
"$low", "$close", "$vwap", "$bid", "$ask", "$volume","$bidV", "$bidV1", "$bidV3",
|
|
"$bidV5", "$askV", "$askV1", "$askV3", "$askV5",],"feature_columns_yesterday":
|
|
["$open_1", "$high_1", "$low_1", "$close_1", "$vwap_1", "$bid_1", "$ask_1", "$volume_1","$bidV_1",
|
|
"$bidV1_1", "$bidV3_1", "$bidV5_1", "$askV_1", "$askV1_1", "$askV3_1", "$askV5_1",],}'
|
|
function: integration.init_qlib
|
|
- docstring: null
|
|
function: pickle_styled._infer_processed_data_column_names
|
|
- docstring: null
|
|
function: pickle_styled._find_pickle
|
|
- docstring: null
|
|
function: pickle_styled._read_pickle
|
|
- docstring: ' Backtest data for simple simulator
|
|
|
|
self,data_dir: Path | str,stock_id: str,date: pd.Timestamp,deal_price: DealPriceType
|
|
= "close",order_dir: int | None = None,) -> None:super(SimpleIntradayBacktestData,
|
|
self).__init__()backtest = _read_pickle((data_dir if isinstance(data_dir, Path)
|
|
else Path(data_dir)) / stock_id)backtest = backtest.loc[pd.IndexSlice[stock_id,
|
|
:, date]]# No longer need for pandas >= 1.4# backtest = backtest.droplevel([0,
|
|
2])self.data: pd.DataFrame = backtestself.deal_price_type: DealPriceType = deal_priceself.order_dir
|
|
= order_dirwith pd.option_context("memory_usage", False, "display.max_info_columns",
|
|
1, "display.large_repr", "info"):return f"{self.__class__.__name__}({self.data})"return
|
|
len(self.data)'
|
|
function: pickle_styled.SimpleIntradayBacktestData
|
|
- docstring: ' Return a pandas series that can be indexed with time.
|
|
|
|
See :attribute:`DealPriceType` for details.'
|
|
function: pickle_styled.get_deal_price
|
|
- docstring: ' Return a volume series that can be indexed with time.
|
|
|
|
return self.data["$volume0"]'
|
|
function: pickle_styled.get_volume
|
|
- docstring: null
|
|
function: pickle_styled.get_time_index
|
|
- docstring: ' Subclass of IntradayProcessedData. Used to handle pickle-styled
|
|
data.
|
|
|
|
self,data_dir: Path | str,stock_id: str,date: pd.Timestamp,feature_dim: int,time_index:
|
|
pd.Index,) -> None:proc = _read_pickle((data_dir if isinstance(data_dir, Path)
|
|
else Path(data_dir)) / stock_id)# We have to infer the names here because,# unfortunately
|
|
they are not included in the original data.cnames = _infer_processed_data_column_names(feature_dim)time_length:
|
|
int = len(time_index)try:# new data formatproc = proc.loc[pd.IndexSlice[stock_id,
|
|
:, date]]assert len(proc) == time_length and len(proc.columns) == feature_dim
|
|
* 2proc_today = proc[cnames]proc_yesterday = proc[[f"{c}_1" for c in cnames]].rename(columns=lambda
|
|
c: c[:-2])except (IndexError, KeyError):# legacy dataproc = proc.loc[pd.IndexSlice[stock_id,
|
|
date]]assert time_length * feature_dim * 2 == len(proc)proc_today = proc.to_numpy()[:
|
|
time_length * feature_dim].reshape((time_length, feature_dim))proc_yesterday =
|
|
proc.to_numpy()[time_length * feature_dim :].reshape((time_length, feature_dim))proc_today
|
|
= pd.DataFrame(proc_today, index=time_index, columns=cnames)proc_yesterday = pd.DataFrame(proc_yesterday,
|
|
index=time_index, columns=cnames)self.today: pd.DataFrame = proc_todayself.yesterday:
|
|
pd.DataFrame = proc_yesterdayassert len(self.today.columns) == len(self.yesterday.columns)
|
|
== feature_dimassert len(self.today) == len(self.yesterday) == time_lengthwith
|
|
pd.option_context("memory_usage", False, "display.max_info_columns", 1, "display.large_repr",
|
|
"info"):return f"{self.__class__.__name__}({self.today}, {self.yesterday})"@lru_cache(maxsize=100) #
|
|
100 * 50K = 5MB'
|
|
function: pickle_styled.PickleIntradayProcessedData
|
|
- docstring: null
|
|
function: pickle_styled.load_simple_intraday_backtest_data
|
|
- docstring: null
|
|
function: pickle_styled.load_pickle_intraday_processed_data
|
|
- docstring: null
|
|
function: pickle_styled.PickleProcessedDataProvider
|
|
- docstring: null
|
|
function: pickle_styled.get_data
|
|
- docstring: null
|
|
function: backtest._get_multi_level_executor_config
|
|
- docstring: null
|
|
function: backtest._convert_indicator_to_dataframe
|
|
- docstring: ' Generate backtest reports
|
|
|
|
Parameters----------decisions:List of trade decisions.report_indicatorsList of
|
|
indicator reports.Returns-------'
|
|
function: backtest._generate_report
|
|
- docstring: ' Run backtest in a single thread with SingleAssetOrderExecution simulator.
|
|
The orders will be executed day by day.
|
|
|
|
A new simulator will be created and used for every single-day order.Parameters----------backtest_config:Backtest
|
|
configorders:Orders to be executed. Example format:datetime instrument amount direction0 2020-06-01 INST 600.0 01 2020-06-02 INST 700.0 1...splitMethod
|
|
to split orders. If it is "stock", split orders by stock. If it is "day", split
|
|
orders by date.cash_limitLimitation of cash.generate_reportWhether to generate
|
|
reports.Returns-------If generate_report is True, return execution records and
|
|
the generated report. Otherwise, return only records.'
|
|
function: backtest.single_with_simulator
|
|
- docstring: ' Run backtest in a single thread with collect_data_loop.
|
|
|
|
Parameters----------backtest_config:Backtest configorders:Orders to be executed.
|
|
Example format:datetime instrument amount direction0 2020-06-01 INST 600.0 01 2020-06-02 INST 700.0 1...splitMethod
|
|
to split orders. If it is "stock", split orders by stock. If it is "day", split
|
|
orders by date.cash_limitLimitation of cash.generate_reportWhether to generate
|
|
reports.Returns-------If generate_report is True, return execution records and
|
|
the generated report. Otherwise, return only records.'
|
|
function: backtest.single_with_collect_data_loop
|
|
- docstring: null
|
|
function: train_onpolicy.seed_everything
|
|
- docstring: null
|
|
function: train_onpolicy._read_orders
|
|
- docstring: null
|
|
function: train_onpolicy.LazyLoadDataset
|
|
- docstring: null
|
|
function: train_onpolicy.train_and_test
|
|
- docstring: null
|
|
function: train_onpolicy._simulator_factory_simple
|
|
- docstring: null
|
|
function: naive_config_parser.merge_a_into_b
|
|
- docstring: null
|
|
function: naive_config_parser.check_file_exist
|
|
- docstring: null
|
|
function: naive_config_parser.parse_backtest_config
|
|
- docstring: null
|
|
function: naive_config_parser._convert_all_list_to_tuple
|
|
- docstring: null
|
|
function: utils.dataframe_append
|
|
- docstring: null
|
|
function: utils.price_advantage
|
|
- docstring: ' The network architecture proposed in `OPD <https://seqml.github.io/opd/opd_aaai21_supplement.pdf>`_.
|
|
|
|
At every time step the input of policy network is divided into two parts,the public
|
|
variables and the private variables. which are handled by ``raw_rnn``and ``pri_rnn``
|
|
in this network, respectively.One minor difference is that, in this implementation,
|
|
we don''t assume the direction to be fixed.Thus, another ``dire_fc`` is added
|
|
to produce an extra direction-related feature.'
|
|
function: network.Recurrent
|
|
- docstring: null
|
|
function: network._init_extra_branches
|
|
- docstring: null
|
|
function: network._source_features
|
|
- docstring: " \nInput should be a dict (at least) containing:- data_processed:\
|
|
\ [N, T, C]- cur_step: [N] (int)- cur_time: [N] (int)- position_history: [N,\
|
|
\ S] (S is number of steps)- target: [N]- num_step: [N] (int)- acquiring: [N]\
|
|
\ (0 or 1)"
|
|
function: network.forward
|
|
- docstring: null
|
|
function: network.Attention
|
|
- docstring: ' Encourage higher PAs, but penalize stacking all the amounts within
|
|
a very short time.
|
|
|
|
Formally, for each time step, the reward is :math:`(PA_t * vol_t / target - vol_t^2
|
|
* penalty)`.Parameters----------penaltyThe penalty for large volume in a short
|
|
time.scaleThe weight used to scale up or down the reward.'
|
|
function: reward.PAPenaltyReward
|
|
- docstring: null
|
|
function: reward.reward
|
|
- docstring: ' Reward proposed by paper "An End-to-End Optimal Trade Execution
|
|
Framework based on Proximal Policy Optimization".
|
|
|
|
Parameters----------max_stepMaximum number of steps.start_time_indexFirst time
|
|
index that allowed to trade.end_time_indexLast time index that allowed to trade.'
|
|
function: reward.PPOReward
|
|
- docstring: null
|
|
function: strategy._get_all_timestamps
|
|
- docstring: ' Fill missing data.
|
|
|
|
Parameters----------original_dataOriginal data without missing values.fill_methodMethod
|
|
used to fill the missing data.Returns-------The filled data.'
|
|
function: strategy.fill_missing_data
|
|
- docstring: " \nMaintain states of the environment. SAOEStateAdapter accepts execution\
|
|
\ results and update its internal stateaccording to the execution results with\
|
|
\ additional information acquired from executors & exchange. For example,it gets\
|
|
\ the dealt order amount from execution results, and get the corresponding market\
|
|
\ price / volume fromexchange.Example usage::adapter = SAOEStateAdapter(...)adapter.update(...)state\
|
|
\ = adapter.saoe_state"
|
|
function: 'strategy.SAOEStateAdapter:'
|
|
- docstring: null
|
|
function: strategy._next_time
|
|
- docstring: null
|
|
function: strategy.update
|
|
- docstring: ' Generate metrics once the upper level execution is done
|
|
|
|
self.metrics = self._collect_single_order_metric(self.order,self.backtest_data.ticks_index[0], #
|
|
start timeself.history_exec["market_volume"],self.history_exec["market_price"],self.history_steps["amount"].sum(),self.history_exec["deal_amount"],)'
|
|
function: strategy.generate_metrics_after_done
|
|
- docstring: null
|
|
function: strategy._collect_multi_order_metric
|
|
- docstring: null
|
|
function: strategy._collect_single_order_metric
|
|
- docstring: null
|
|
function: strategy.saoe_state
|
|
- docstring: ' RL-based strategies that use SAOEState as state.
|
|
|
|
self,policy: BasePolicy,outer_trade_decision: BaseTradeDecision | None = None,level_infra:
|
|
LevelInfrastructure | None = None,common_infra: CommonInfrastructure | None =
|
|
None,data_granularity: int = 1,**kwargs: Any,) -> None:super(SAOEStrategy, self).__init__(policy=policy,outer_trade_decision=outer_trade_decision,level_infra=level_infra,common_infra=common_infra,**kwargs,)self._data_granularity
|
|
= data_granularityself.adapter_dict: Dict[tuple, SAOEStateAdapter] = {}self._last_step_range
|
|
= (0, 0)'
|
|
function: strategy.SAOEStrategy
|
|
- docstring: null
|
|
function: strategy._create_qlib_backtest_adapter
|
|
- docstring: null
|
|
function: strategy.reset
|
|
- docstring: null
|
|
function: strategy.get_saoe_state_by_order
|
|
- docstring: null
|
|
function: strategy.post_upper_level_exe_step
|
|
- docstring: null
|
|
function: strategy.post_exe_step
|
|
- docstring: " \nFor SAOEStrategy, we need to update the `self._last_step_range`\
|
|
\ every time a decision is generated.This operation should be invisible to developers,\
|
|
\ so we implement it in `generate_trade_decision()`The concrete logic to generate\
|
|
\ decisions should be implemented in `_generate_trade_decision()`.In other words,\
|
|
\ all subclass of `SAOEStrategy` should overwrite `_generate_trade_decision()`\
|
|
\ instead of`generate_trade_decision()`."
|
|
function: strategy.generate_trade_decision
|
|
- docstring: null
|
|
function: strategy._generate_trade_decision
|
|
- docstring: ' Proxy strategy that uses SAOEState. It is called a ''proxy'' strategy
|
|
because it does not make any decisions
|
|
|
|
by itself. Instead, when the strategy is required to generate a decision, it will
|
|
yield the environment''sinformation and let the outside agents to make the decision.
|
|
Please refer to `_generate_trade_decision` formore details.'
|
|
function: strategy.ProxySAOEStrategy
|
|
- docstring: null
|
|
function: strategy._generate_trade_decision
|
|
- docstring: null
|
|
function: strategy.reset
|
|
- docstring: ' (SAOE)state based strategy with (Int)preters.
|
|
|
|
self,policy: dict | BasePolicy,state_interpreter: dict | StateInterpreter,action_interpreter:
|
|
dict | ActionInterpreter,network: dict | torch.nn.Module | None = None,outer_trade_decision:
|
|
BaseTradeDecision | None = None,level_infra: LevelInfrastructure | None = None,common_infra:
|
|
CommonInfrastructure | None = None,**kwargs: Any,) -> None:super(SAOEIntStrategy,
|
|
self).__init__(policy=policy,outer_trade_decision=outer_trade_decision,level_infra=level_infra,common_infra=common_infra,**kwargs,)self._state_interpreter:
|
|
StateInterpreter = init_instance_by_config(state_interpreter,accept_types=StateInterpreter,)self._action_interpreter:
|
|
ActionInterpreter = init_instance_by_config(action_interpreter,accept_types=ActionInterpreter,)if
|
|
isinstance(policy, dict):assert network is not Noneif isinstance(network, dict):network["kwargs"].update({"obs_space":
|
|
self._state_interpreter.observation_space,})network_inst = init_instance_by_config(network)else:network_inst
|
|
= networkpolicy["kwargs"].update({"obs_space": self._state_interpreter.observation_space,"action_space":
|
|
self._action_interpreter.action_space,"network": network_inst,})self._policy =
|
|
init_instance_by_config(policy)elif isinstance(policy, BasePolicy):self._policy
|
|
= policyelse:raise ValueError(f"Unsupported policy type: {type(policy)}.")if self._policy
|
|
is not None:self._policy.eval()'
|
|
function: strategy.SAOEIntStrategy
|
|
- docstring: null
|
|
function: strategy.reset
|
|
- docstring: null
|
|
function: strategy._generate_trade_details
|
|
- docstring: ' Tianshou''s BasePolicy with empty ``learn`` and ``process_fn``.
|
|
|
|
This could be moved outside in future.'
|
|
function: policy.NonLearnablePolicy
|
|
- docstring: null
|
|
function: policy.learn
|
|
- docstring: null
|
|
function: policy.process_fn
|
|
- docstring: ' Forward returns a batch full of 1.
|
|
|
|
Useful when implementing some baselines (e.g., TWAP).'
|
|
function: policy.AllOne
|
|
- docstring: null
|
|
function: policy.forward
|
|
- docstring: null
|
|
function: policy.PPOActor
|
|
- docstring: null
|
|
function: policy.forward
|
|
- docstring: null
|
|
function: policy.PPOCritic
|
|
- docstring: null
|
|
function: policy.forward
|
|
- docstring: ' A wrapper of tianshou PPOPolicy.
|
|
|
|
Differences:- Auto-create actor and critic network. Supports discrete action space
|
|
only.- Dedup common parameters between actor network and critic network(not sure
|
|
whether this is included in latest tianshou or not).- Support a ``weight_file``
|
|
that supports loading checkpoint.- Some parameters'' default values are different
|
|
from original.'
|
|
function: policy.PPO
|
|
- docstring: ' A wrapper of tianshou DQNPolicy.
|
|
|
|
Differences:- Auto-create model network. Supports discrete action space only.-
|
|
Support a ``weight_file`` that supports loading checkpoint.'
|
|
function: policy.DQN
|
|
- docstring: null
|
|
function: policy.auto_device
|
|
- docstring: null
|
|
function: policy.set_weight
|
|
- docstring: ' Single-asset order execution (SAOE) simulator.
|
|
|
|
As there''s no "calendar" in the simple simulator, ticks are used to trade.A tick
|
|
is a record (a line) in the pickle-styled data file.Each tick is considered as
|
|
a individual trading opportunity.If such fine granularity is not needed, use ``ticks_per_step``
|
|
tolengthen the ticks for each step.In each step, the traded amount are "equally"
|
|
separated to each tick,then bounded by volume maximum execution volume (i.e.,
|
|
``vol_threshold``),and if it''s the last step, try to ensure all the amount to
|
|
be executed.Parameters----------orderThe seed to start an SAOE simulator is an
|
|
order.data_dirPath to load backtest data.feature_columns_todayColumns of today''s
|
|
feature.feature_columns_yesterdayColumns of yesterday''s feature.data_granularityNumber
|
|
of ticks between consecutive data entries.ticks_per_stepHow many ticks per step.vol_thresholdMaximum
|
|
execution volume (divided by market execution volume).'
|
|
function: simulator_simple.SingleAssetOrderExecutionSimple
|
|
- docstring: null
|
|
function: simulator_simple.get_backtest_data
|
|
- docstring: ' Execute one step or SAOE.
|
|
|
|
Parameters----------amountThe amount you wish to deal. The simulator doesn''t
|
|
guarantee all the amount to be successfully dealt.'
|
|
function: simulator_simple.step
|
|
- docstring: null
|
|
function: simulator_simple.get_state
|
|
- docstring: null
|
|
function: simulator_simple.done
|
|
- docstring: ' The "current time" (``cur_time``) for next step.
|
|
|
|
# Look for next time on time indexcurrent_loc = self.ticks_index.get_loc(self.cur_time)next_loc
|
|
= current_loc + self.ticks_per_step# Calibrate the next location to multiple of
|
|
ticks_per_step.# This is to make sure that:# as long as ticks_per_step is a multiple
|
|
of something, each step won''t cross morning and afternoon.next_loc = next_loc
|
|
- next_loc % self.ticks_per_stepif next_loc < len(self.ticks_index) and self.ticks_index[next_loc]
|
|
< self.order.end_time:return self.ticks_index[next_loc]else:return self.order.end_time'
|
|
function: simulator_simple._next_time
|
|
- docstring: ' The "duration" of this step (step that is about to happen).
|
|
|
|
return self._next_time() - self.cur_time'
|
|
function: simulator_simple._cur_duration
|
|
- docstring: " \nSplit the volume in each step into minutes, considering possible\
|
|
\ constraints.This follows TWAP strategy."
|
|
function: simulator_simple._split_exec_vol
|
|
- docstring: null
|
|
function: simulator_simple._metrics_collect
|
|
- docstring: null
|
|
function: simulator_simple._get_ticks_slice
|
|
- docstring: null
|
|
function: simulator_simple._dataframe_append
|
|
- docstring: ' Single-asset order execution (SAOE) simulator which is implemented
|
|
based on Qlib backtest tools.
|
|
|
|
Parameters----------orderThe seed to start an SAOE simulator is an order.executor_configExecutor
|
|
configurationexchange_configExchange configurationqlib_configConfiguration used
|
|
to initialize Qlib. If it is None, Qlib will not be initialized.cash_limit:Cash
|
|
limit.'
|
|
function: simulator_qlib.SingleAssetOrderExecution
|
|
- docstring: null
|
|
function: simulator_qlib.reset
|
|
- docstring: null
|
|
function: simulator_qlib._get_adapter
|
|
- docstring: null
|
|
function: simulator_qlib.twap_price
|
|
- docstring: ' Iterate the _collect_data_loop until we get the next yield SAOEStrategy.
|
|
|
|
assert self._collect_data_loop is not Noneobj = next(self._collect_data_loop)
|
|
if action is None else self._collect_data_loop.send(action)while not isinstance(obj,
|
|
SAOEStrategy):if isinstance(obj, BaseTradeDecision):self.decisions.append(obj)obj
|
|
= next(self._collect_data_loop) if action is None else self._collect_data_loop.send(action)assert
|
|
isinstance(obj, SAOEStrategy)return obj'
|
|
function: simulator_qlib._iter_strategy
|
|
- docstring: ' Execute one step or SAOE.
|
|
|
|
Parameters----------action (float):The amount you wish to deal. The simulator
|
|
doesn''t guarantee all the amount to be successfully dealt.'
|
|
function: simulator_qlib.step
|
|
- docstring: null
|
|
function: simulator_qlib.get_state
|
|
- docstring: ' To 32-bit numeric types. Recursively.
|
|
|
|
if isinstance(value, pd.DataFrame):return value.to_numpy()if isinstance(value,
|
|
(float, np.floating)) or (isinstance(value, np.ndarray) and value.dtype.kind ==
|
|
"f"):return np.array(value, dtype=np.float32)elif isinstance(value, (int, bool,
|
|
np.integer)) or (isinstance(value, np.ndarray) and value.dtype.kind == "i"):return
|
|
np.array(value, dtype=np.int32)elif isinstance(value, dict):return {k: canonicalize(v)
|
|
for k, v in value.items()}else:return value'
|
|
function: interpreter.canonicalize
|
|
- docstring: null
|
|
function: interpreter.FullHistoryObs
|
|
- docstring: ' Dummy interpreter for policies that do not need inputs (for example,
|
|
AllOne).
|
|
|
|
'
|
|
function: interpreter.DummyStateInterpreter
|
|
- docstring: null
|
|
function: interpreter.interpret
|
|
- docstring: null
|
|
function: interpreter.observation_space
|
|
- docstring: ' The observation of all the history, including today (until this
|
|
moment), and yesterday.
|
|
|
|
Parameters----------max_stepTotal number of steps (an upper-bound estimation).
|
|
For example, 390min / 30min-per-step = 13 steps.data_ticksEqual to the total number
|
|
of records. For example, in SAOE per minute,the total ticks is the length of day
|
|
in minutes.data_dimNumber of dimensions in data.processed_data_providerProvider
|
|
of the processed data.'
|
|
function: interpreter.FullHistoryStateInterpreter
|
|
- docstring: null
|
|
function: interpreter.interpret
|
|
- docstring: null
|
|
function: interpreter.observation_space
|
|
- docstring: null
|
|
function: interpreter._mask_future_info
|
|
- docstring: null
|
|
function: interpreter.CurrentStateObs
|
|
- docstring: ' The observation of current step.
|
|
|
|
Used when policy only depends on the latest state, but not history.The key list
|
|
is not full. You can add more if more information is needed by your policy.'
|
|
function: interpreter.CurrentStepStateInterpreter
|
|
- docstring: null
|
|
function: interpreter.observation_space
|
|
- docstring: null
|
|
function: interpreter.interpret
|
|
- docstring: ' Convert a discrete policy action to a continuous action, then multiplied
|
|
by ``order.amount``.
|
|
|
|
Parameters----------valuesIt can be a list of length $L$: $[a_1, a_2, \\ldots,
|
|
a_L]$.Then when policy givens decision $x$, $a_x$ times order amount is the output.It
|
|
can also be an integer $n$, in which case the list of length $n+1$ is auto-generated,i.e.,
|
|
$[0, 1/n, 2/n, \\ldots, n/n]$.max_stepTotal number of steps (an upper-bound estimation).
|
|
For example, 390min / 30min-per-step = 13 steps.'
|
|
function: interpreter.CategoricalActionInterpreter
|
|
- docstring: null
|
|
function: interpreter.action_space
|
|
- docstring: null
|
|
function: interpreter.interpret
|
|
- docstring: ' Convert a continuous ratio to deal amount.
|
|
|
|
The ratio is relative to TWAP on the remainder of the day.For example, there are
|
|
5 steps left, and the left position is 300.With TWAP strategy, in each position,
|
|
60 should be traded.When this interpreter receives action $a$, its output is $60
|
|
\\cdot a$.'
|
|
function: interpreter.TwapRelativeActionInterpreter
|
|
- docstring: null
|
|
function: interpreter.action_space
|
|
- docstring: null
|
|
function: interpreter.interpret
|
|
- docstring: null
|
|
function: interpreter._to_int32
|
|
- docstring: ' Metrics for SAOE accumulated for a "period".
|
|
|
|
It could be accumulated for a day, or a period of time (e.g., 30min), or calculated
|
|
separately for every minute.Warnings--------The type hints are for single elements.
|
|
In lots of times, they can be vectorized.For example, ``market_volume`` could
|
|
be a list of float (or ndarray) rather tahn a single float.'
|
|
function: state.SAOEMetrics
|
|
- docstring: ' Data structure holding a state for SAOE simulator.
|
|
|
|
order: OrderThe order we are dealing with.'
|
|
function: state.SAOEState
|
|
- docstring: " \nUtility to train a policy on a particular task.Different from\
|
|
\ traditional DL trainer, the iteration of this trainer is \"collect\",rather\
|
|
\ than \"epoch\", or \"mini-batch\".In each collect, :class:`Collector` collects\
|
|
\ a number of policy-env interactions, and accumulatesthem into a replay buffer.\
|
|
\ This buffer is used as the \"data\" to train the policy.At the end of each collect,\
|
|
\ the policy is *updated* several times.The API has some resemblence with `PyTorch\
|
|
\ Lightning <https://pytorch-lightning.readthedocs.io/>`__,but it's essentially\
|
|
\ different because this trainer is built for RL applications, and thusmost configurations\
|
|
\ are under RL context.We are still looking for ways to incorporate existing trainer\
|
|
\ libraries, because it looks likebig efforts to build a trainer as powerful as\
|
|
\ those libraries, and also, that's not our primary goal.It's essentially different`tianshou's\
|
|
\ built-in trainers <https://tianshou.readthedocs.io/en/master/api/tianshou.trainer.html>`__,as\
|
|
\ it's far much more complicated than that.Parameters----------max_itersMaximum\
|
|
\ iterations before stopping.val_every_n_itersPerform validation every n iterations\
|
|
\ (i.e., training collects).loggerLogger to record the backtest results. Logger\
|
|
\ must be present becausewithout logger, all information will be lost.finite_env_typeType\
|
|
\ of finite env implementation.concurrencyParallel workers.fast_dev_runCreate\
|
|
\ a subset for debugging.How this is implemented depends on the implementation\
|
|
\ of training vessel.For :class:`~qlib.rl.vessel.TrainingVessel`, if greater than\
|
|
\ zero,a random subset sized ``fast_dev_run`` will be usedinstead of ``train_initial_states``\
|
|
\ and ``val_initial_states``."
|
|
function: 'trainer.Trainer:'
|
|
- docstring: ' Initialize the whole training process.
|
|
|
|
The states here should be synchronized with state_dict.'
|
|
function: trainer.initialize
|
|
- docstring: ' Initialize one iteration / collect.
|
|
|
|
self.metrics = {}'
|
|
function: trainer.initialize_iter
|
|
- docstring: ' Putting every states of current training into a dict, at best
|
|
effort.
|
|
|
|
It doesn''t try to handle all the possible kinds of states in the middle of one
|
|
training collect.For most cases at the end of each iteration, things should be
|
|
usually correct.Note that it''s also intended behavior that replay buffer data
|
|
in the collector will be lost.'
|
|
function: trainer.state_dict
|
|
- docstring: null
|
|
function: trainer.get_policy_state_dict
|
|
- docstring: ' Load all states into current trainer.
|
|
|
|
self.vessel.load_state_dict(state_dict["vessel"])for name, callback in self.named_callbacks().items():callback.load_state_dict(state_dict["callbacks"][name])for
|
|
name, logger in self.named_loggers().items():logger.load_state_dict(state_dict["loggers"][name])self.should_stop
|
|
= state_dict["should_stop"]self.current_iter = state_dict["current_iter"]self.current_episode
|
|
= state_dict["current_episode"]self.current_stage = state_dict["current_stage"]self.metrics
|
|
= state_dict["metrics"]'
|
|
function: trainer.load_state_dict
|
|
- docstring: ' Retrieve a collection of callbacks where each one has a name.
|
|
|
|
Useful when saving checkpoints.'
|
|
function: trainer.named_callbacks
|
|
- docstring: ' Retrieve a collection of loggers where each one has a name.
|
|
|
|
Useful when saving checkpoints.'
|
|
function: trainer.named_loggers
|
|
- docstring: ' Train the RL policy upon the defined simulator.
|
|
|
|
Parameters----------vesselA bundle of all elements used in training.ckpt_pathLoad
|
|
a pre-trained / paused training checkpoint.'
|
|
function: trainer.fit
|
|
- docstring: ' Test the RL policy against the simulator.
|
|
|
|
The simulator will be fed with data generated in ``test_seed_iterator``.Parameters----------vesselA
|
|
bundle of all related elements.'
|
|
function: trainer.test
|
|
- docstring: ' Create a vectorized environment from iterator and the training
|
|
vessel.
|
|
|
|
'
|
|
function: trainer.venv_from_iterator
|
|
- docstring: null
|
|
function: trainer.env_factory
|
|
- docstring: null
|
|
function: trainer._metrics_callback
|
|
- docstring: null
|
|
function: trainer._call_callback_hooks
|
|
- docstring: null
|
|
function: trainer._min_loglevel
|
|
- docstring: ' Make any object a (possibly dummy) context manager.
|
|
|
|
if isinstance(obj, AbstractContextManager):# obj has __enter__ and __exit__with
|
|
obj as ctx:yield ctxelse:yield obj'
|
|
function: trainer._wrap_context
|
|
- docstring: ' Train a policy with the parallelism provided by RL framework.
|
|
|
|
Experimental API. Parameters might change shortly.Parameters----------simulator_fnCallable
|
|
receiving initial seed, returning a simulator.state_interpreterInterprets the
|
|
state of simulators.action_interpreterInterprets the policy actions.initial_statesInitial
|
|
states to iterate over. Every state will be run exactly once.policyPolicy to train
|
|
against.rewardReward function.vessel_kwargsKeyword arguments passed to :class:`TrainingVessel`,
|
|
like ``episode_per_iter``.trainer_kwargsKeyword arguments passed to :class:`Trainer`,
|
|
like ``finite_env_type``, ``concurrency``.'
|
|
function: api.train
|
|
- docstring: ' Backtest with the parallelism provided by RL framework.
|
|
|
|
Experimental API. Parameters might change shortly.Parameters----------simulator_fnCallable
|
|
receiving initial seed, returning a simulator.state_interpreterInterprets the
|
|
state of simulators.action_interpreterInterprets the policy actions.initial_statesInitial
|
|
states to iterate over. Every state will be run exactly once.policyPolicy to test
|
|
against.loggerLogger to record the backtest results. Logger must be present becausewithout
|
|
logger, all information will be lost.rewardOptional reward function. For backtest,
|
|
this is for testing the rewardsand logging them only.finite_env_typeType of finite
|
|
env implementation.concurrencyParallel workers.'
|
|
function: api.backtest
|
|
- docstring: ' Base class of all callbacks.
|
|
|
|
'
|
|
function: 'callbacks.Callback:'
|
|
- docstring: ' Called before the whole fit process begins.
|
|
|
|
'
|
|
function: callbacks.on_fit_start
|
|
- docstring: ' Called after the whole fit process ends.
|
|
|
|
'
|
|
function: callbacks.on_fit_end
|
|
- docstring: ' Called when each collect for training begins.
|
|
|
|
'
|
|
function: callbacks.on_train_start
|
|
- docstring: ' Called when the training ends.
|
|
|
|
To access all outputs produced during training, cache the data in either trainer
|
|
and vessel,and post-process them in this hook.'
|
|
function: callbacks.on_train_end
|
|
- docstring: ' Called when every run for validation begins.
|
|
|
|
'
|
|
function: callbacks.on_validate_start
|
|
- docstring: ' Called when the validation ends.
|
|
|
|
'
|
|
function: callbacks.on_validate_end
|
|
- docstring: ' Called when every run of testing begins.
|
|
|
|
'
|
|
function: callbacks.on_test_start
|
|
- docstring: ' Called when the testing ends.
|
|
|
|
'
|
|
function: callbacks.on_test_end
|
|
- docstring: ' Called when every iteration (i.e., collect) starts.
|
|
|
|
'
|
|
function: callbacks.on_iter_start
|
|
- docstring: ' Called upon every end of iteration.
|
|
|
|
This is called **after** the bump of ``current_iter``,when the previous iteration
|
|
is considered complete.'
|
|
function: callbacks.on_iter_end
|
|
- docstring: ' Get a state dict of the callback for pause and resume.
|
|
|
|
'
|
|
function: callbacks.state_dict
|
|
- docstring: ' Resume the callback from a saved state dict.
|
|
|
|
'
|
|
function: callbacks.load_state_dict
|
|
- docstring: ' Stop training when a monitored metric has stopped improving.
|
|
|
|
The earlystopping callback will be triggered each time validation ends.It will
|
|
examine the metrics produced in validation,and get the metric with name ``monitor`
|
|
(``monitor`` is ``reward`` by default),to check whether it''s no longer increasing
|
|
/ decreasing.It takes ``min_delta`` and ``patience`` if applicable.If it''s found
|
|
to be not increasing / decreasing any more.``trainer.should_stop`` will be set
|
|
to true,and the training terminates.Implementation reference: https://github.com/keras-team/keras/blob/v2.9.0/keras/callbacks.py#L1744-L1893'
|
|
function: callbacks.EarlyStopping
|
|
- docstring: null
|
|
function: callbacks.state_dict
|
|
- docstring: null
|
|
function: callbacks.load_state_dict
|
|
- docstring: null
|
|
function: callbacks.on_fit_start
|
|
- docstring: null
|
|
function: callbacks.on_validate_end
|
|
- docstring: null
|
|
function: callbacks.get_monitor_value
|
|
- docstring: null
|
|
function: callbacks._is_improvement
|
|
- docstring: ' Dump training metrics to file.
|
|
|
|
self.dirpath = dirpathself.dirpath.mkdir(exist_ok=True, parents=True)self.train_records:
|
|
List[dict] = []self.valid_records: List[dict] = []'
|
|
function: callbacks.MetricsWriter
|
|
- docstring: null
|
|
function: callbacks.on_train_end
|
|
- docstring: null
|
|
function: callbacks.on_validate_end
|
|
- docstring: ' Save checkpoints periodically for persistence and recovery.
|
|
|
|
Reference: https://github.com/PyTorchLightning/pytorch-lightning/blob/bfa8b7be/pytorch_lightning/callbacks/model_checkpoint.pyParameters----------dirpathDirectory
|
|
to save the checkpoint file.filenameCheckpoint filename. Can contain named formatting
|
|
options to be auto-filled.For example: ``{iter:03d}-{reward:.2f}.pth``.Supported
|
|
argument names are:- iter (int)- metrics in ``trainer.metrics``- time string,
|
|
in the format of ``%Y%m%d%H%M%S``save_latestSave the latest checkpoint in ``latest.pth``.If
|
|
``link``, ``latest.pth`` will be created as a softlink.If ``copy``, ``latest.pth``
|
|
will be stored as an individual copy.Set to none to disable this.every_n_itersCheckpoints
|
|
are saved at the end of every n iterations of training,after validation if applicable.time_intervalMaximum
|
|
time (seconds) before checkpoints save again.save_on_fit_endSave one last checkpoint
|
|
at the end to fit.Do nothing if a checkpoint is already saved there.'
|
|
function: callbacks.Checkpoint
|
|
- docstring: null
|
|
function: callbacks.on_fit_end
|
|
- docstring: null
|
|
function: callbacks.on_iter_end
|
|
- docstring: null
|
|
function: callbacks._save_checkpoint
|
|
- docstring: null
|
|
function: vessel.SeedIteratorNotAvailable
|
|
- docstring: ' A ship that contains simulator, interpreter, and policy, will be
|
|
sent to trainer.
|
|
|
|
This class controls algorithm-related parts of training, while trainer is responsible
|
|
for runtime part.The ship also defines the most important logic of the core training
|
|
part,and (optionally) some callbacks to insert customized logics at specific events.'
|
|
function: vessel.TrainingVesselBase
|
|
- docstring: null
|
|
function: vessel.assign_trainer
|
|
- docstring: ' Override this to create a seed iterator for training.
|
|
|
|
If the iterable is a context manager, the whole training will be invoked in the
|
|
with-block,and the iterator will be automatically closed after the training is
|
|
done.'
|
|
function: vessel.train_seed_iterator
|
|
- docstring: ' Override this to create a seed iterator for validation.
|
|
|
|
raise SeedIteratorNotAvailable("Seed iterator for validation is not available.")'
|
|
function: vessel.val_seed_iterator
|
|
- docstring: ' Override this to create a seed iterator for testing.
|
|
|
|
raise SeedIteratorNotAvailable("Seed iterator for testing is not available.")'
|
|
function: vessel.test_seed_iterator
|
|
- docstring: ' Implement this to train one iteration. In RL, one iteration
|
|
usually refers to one collect.
|
|
|
|
raise NotImplementedError()'
|
|
function: vessel.train
|
|
- docstring: ' Implement this to validate the policy once.
|
|
|
|
raise NotImplementedError()'
|
|
function: vessel.validate
|
|
- docstring: ' Implement this to evaluate the policy on test environment once.
|
|
|
|
raise NotImplementedError()'
|
|
function: vessel.test
|
|
- docstring: null
|
|
function: vessel.log
|
|
- docstring: null
|
|
function: vessel.log_dict
|
|
- docstring: ' Return a checkpoint of current vessel state.
|
|
|
|
return {"policy": self.policy.state_dict()}'
|
|
function: vessel.state_dict
|
|
- docstring: ' Restore a checkpoint from a previously saved state dict.
|
|
|
|
self.policy.load_state_dict(state_dict["policy"])'
|
|
function: vessel.load_state_dict
|
|
- docstring: ' The default implementation of training vessel.
|
|
|
|
``__init__`` accepts a sequence of initial states so that iterator can be created.``train``,
|
|
``validate``, ``test`` each do one collect (and also update in train).By default,
|
|
the train initial states will be repeated infinitely during training,and collector
|
|
will control the number of episodes for each iteration.In validation and testing,
|
|
the val / test initial states will be used exactly once.Extra hyper-parameters
|
|
(only used in train) include:- ``buffer_size``: Size of replay buffer.- ``episode_per_iter``:
|
|
Episodes per collect at training. Can be overridden by fast dev run.- ``update_kwargs``:
|
|
Keyword arguments appearing in ``policy.update``.For example, ``dict(repeat=10,
|
|
batch_size=64)``.'
|
|
function: vessel.TrainingVessel
|
|
- docstring: null
|
|
function: vessel.train_seed_iterator
|
|
- docstring: null
|
|
function: vessel.val_seed_iterator
|
|
- docstring: null
|
|
function: vessel.test_seed_iterator
|
|
- docstring: ' Create a collector and collects ``episode_per_iter`` episodes.
|
|
|
|
Update the policy on the collected replay buffer.'
|
|
function: vessel.train
|
|
- docstring: null
|
|
function: vessel.validate
|
|
- docstring: null
|
|
function: vessel.test
|
|
- docstring: " \nThis is the Records Template class that enables user to generate\
|
|
\ experiment results such as IC andbacktest in a certain format."
|
|
function: 'record_temp.RecordTemp:'
|
|
- docstring: null
|
|
function: record_temp.get_path
|
|
- docstring: " \nIt behaves the same as self.recorder.save_objects.But it is\
|
|
\ an easier interface because users don't have to care about `get_path` and `artifact_path`"
|
|
function: record_temp.save
|
|
- docstring: null
|
|
function: record_temp.recorder
|
|
- docstring: " \nGenerate certain records such as IC, backtest etc., and save\
|
|
\ them.Parameters----------kwargsReturn------"
|
|
function: record_temp.generate
|
|
- docstring: " \nIt behaves the same as self.recorder.load_object.But it is\
|
|
\ an easier interface because users don't have to care about `get_path` and `artifact_path`Parameters----------name\
|
|
\ : strthe name for the file to be load.parents : boolEach recorder has different\
|
|
\ `artifact_path`.So parents recursively find the path in parentsSub classes has\
|
|
\ higher priorityReturn------The stored records."
|
|
function: record_temp.load
|
|
- docstring: " \nList the supported artifacts.Users don't have to consider\
|
|
\ self.get_pathReturn------A list of all the supported artifacts."
|
|
function: record_temp.list
|
|
- docstring: " \nCheck if the records is properly generated and saved.It is\
|
|
\ useful in following examples- checking if the dependant files complete before\
|
|
\ generating new things.- checking if the final files is completedParameters----------include_self\
|
|
\ : boolis the file generated by self includedparents : boolwill we check parentsRaise------FileNotFoundErrorwhether\
|
|
\ the records are stored properly."
|
|
function: record_temp.check
|
|
- docstring: null
|
|
function: record_temp._get_arts
|
|
- docstring: null
|
|
function: record_temp.analyse
|
|
- docstring: " \nThis is the Signal Record class that generates the signal prediction.\
|
|
\ This class inherits the ``RecordTemp`` class."
|
|
function: record_temp.SignalRecord
|
|
- docstring: null
|
|
function: record_temp.generate_label
|
|
- docstring: null
|
|
function: record_temp.generate
|
|
- docstring: null
|
|
function: record_temp.list
|
|
- docstring: ' Automatically checking record template
|
|
|
|
self.skip_existing = skip_existingsuper().__init__(recorder=recorder)'
|
|
function: record_temp.ACRecordTemp
|
|
- docstring: ' automatically checking the files and then run the concrete generating
|
|
task
|
|
|
|
if self.skip_existing:try:self.check(include_self=True, parents=False)except FileNotFoundError:pass #
|
|
continue to generating metricselse:logger.info("The results has previously generated,
|
|
Generation skipped.")returntry:self.check()except FileNotFoundError:logger.warning("The
|
|
dependent data does not exists. Generation skipped.")returnreturn self._generate(*args,
|
|
**kwargs)'
|
|
function: record_temp.generate
|
|
- docstring: null
|
|
function: record_temp._generate
|
|
- docstring: " \nThis is the Signal Analysis Record class that generates the analysis\
|
|
\ results such as IC and IR. This class inherits the ``RecordTemp`` class."
|
|
function: record_temp.HFSignalRecord
|
|
- docstring: null
|
|
function: record_temp.generate
|
|
- docstring: null
|
|
function: record_temp.list
|
|
- docstring: " \nThis is the Signal Analysis Record class that generates the analysis\
|
|
\ results such as IC and IR.This class inherits the ``RecordTemp`` class."
|
|
function: record_temp.SigAnaRecord
|
|
- docstring: " \nParameters----------label : Optional[pd.DataFrame]Label should\
|
|
\ be a dataframe."
|
|
function: record_temp._generate
|
|
- docstring: null
|
|
function: record_temp.list
|
|
- docstring: " \nThis is the Portfolio Analysis Record class that generates the\
|
|
\ analysis results such as those of backtest. This class inherits the ``RecordTemp``\
|
|
\ class.The following files will be stored in recorder- report_normal.pkl & positions_normal.pkl:-\
|
|
\ The return report and detailed positions of the backtest, returned by `qlib/contrib/evaluate.py:backtest`-\
|
|
\ port_analysis.pkl : The risk analysis of your portfolio, returned by `qlib/contrib/evaluate.py:risk_analysis`"
|
|
function: record_temp.PortAnaRecord
|
|
- docstring: null
|
|
function: record_temp._get_report_freq
|
|
- docstring: null
|
|
function: record_temp._generate
|
|
- docstring: " \nMethod for handling the experiment when any unusual program ending\
|
|
\ occurs.The `atexit` handler should be put in the last, since, as long as the\
|
|
\ program ends, it will be called.Thus, if any exception or user interruption\
|
|
\ occurs beforehand, we should handle them first. Once `R` isended, another call\
|
|
\ of `R.end_exp` will not take effect.Limitations:- If pdb is used in your program,\
|
|
\ excepthook will not be triggered when it ends. The status will be finished"
|
|
function: utils.experiment_exit_handler
|
|
- docstring: " \nEnd an experiment with status to be \"FAILED\". This exception\
|
|
\ tries to catch those uncaught exceptionand end the experiment automatically.Parametersexc_type:\
|
|
\ Exception typevalue: Exception's valuetb: Exception's traceback"
|
|
function: utils.experiment_exception_hook
|
|
- docstring: null
|
|
function: cli.get_path_list
|
|
- docstring: " \nConfigure the `sys` sectionParameters----------config : dictconfiguration\
|
|
\ of the workflow.config_path : strpath of the configuration"
|
|
function: cli.sys_config
|
|
- docstring: " \nThis is a Qlib CLI entrance.User can run the whole Quant research\
|
|
\ workflow defined by a configure file- the code is located here ``qlib/workflow/cli.py`User\
|
|
\ can specify a base_config file in your workflow.yml file by adding \"BASE_CONFIG_PATH\"\
|
|
.Qlib will load the configuration in BASE_CONFIG_PATH first, and the user only\
|
|
\ needs to update the custom fieldsin their own workflow.yml file.For examples:qlib_init:provider_uri:\
|
|
\ \"~/.qlib/qlib_data/cn_data\"region: cnBASE_CONFIG_PATH: \"workflow_config_lightgbm_Alpha158_csi500.yaml\"\
|
|
market: csi300"
|
|
function: cli.workflow
|
|
- docstring: " \nThis is the `ExpManager` class for managing experiments. The API\
|
|
\ is designed similar to mlflow.(The link: https://mlflow.org/docs/latest/python_api/mlflow.html)The\
|
|
\ `ExpManager` is expected to be a singleton (btw, we can have multiple `Experiment`s\
|
|
\ with different uri. user can get different experiments from different uri, and\
|
|
\ then compare records of them). Global Config (i.e. `C`) is also a singleton.So\
|
|
\ we try to align them together. They share the same variable, which is called\
|
|
\ **default uri**. Please refer to `ExpManager.default_uri` for details of variable\
|
|
\ sharing.When the user starts an experiment, the user may want to set the uri\
|
|
\ to a specific uri (it will override **default uri** during this period), and\
|
|
\ then unset the **specific uri** and fallback to the **default uri**. `ExpManager._active_exp_uri`\
|
|
\ is that **specific uri**."
|
|
function: 'expm.ExpManager:'
|
|
- docstring: " \nStart an experiment. This method includes first get_or_create\
|
|
\ an experiment, and thenset it to be active.Maintaining `_active_exp_uri` is\
|
|
\ included in start_exp, remaining implementation should be included in _end_exp\
|
|
\ in subclassParameters----------experiment_id : strid of the active experiment.experiment_name\
|
|
\ : strname of the active experiment.recorder_id : strid of the recorder to be\
|
|
\ started.recorder_name : strname of the recorder to be started.uri : strthe current\
|
|
\ tracking URI.resume : booleanwhether to resume the experiment and recorder.Returns-------An\
|
|
\ active experiment."
|
|
function: expm.start_exp
|
|
- docstring: ' Please refer to the doc of `start_exp`
|
|
|
|
raise NotImplementedError(f"Please implement the `start_exp` method.")'
|
|
function: expm._start_exp
|
|
- docstring: " \nEnd an active experiment.Maintaining `_active_exp_uri` is\
|
|
\ included in end_exp, remaining implementation should be included in _end_exp\
|
|
\ in subclassParameters----------experiment_name : strname of the active experiment.recorder_status\
|
|
\ : strthe status of the active recorder of the experiment."
|
|
function: expm.end_exp
|
|
- docstring: null
|
|
function: expm._end_exp
|
|
- docstring: " \nCreate an experiment.Parameters----------experiment_name :\
|
|
\ strthe experiment name, which must be unique.Returns-------An experiment object.Raise-----ExpAlreadyExistError"
|
|
function: expm.create_exp
|
|
- docstring: " \nGet a pandas DataFrame of records that fit the search criteria\
|
|
\ of the experiment.Inputs are the search criteria user want to apply.Returns-------A\
|
|
\ pandas.DataFrame of records, where each metric, parameter, and tagare expanded\
|
|
\ into their own columns named metrics.*, params.*, and tags.*respectively. For\
|
|
\ records that don't have a particular metric, parameter, or tag, theirvalue will\
|
|
\ be (NumPy) Nan, None, or None respectively."
|
|
function: expm.search_records
|
|
- docstring: " \nRetrieve an experiment. This method includes getting an active\
|
|
\ experiment, and get_or_create a specific experiment.When user specify experiment\
|
|
\ id and name, the method will try to return the specific experiment.When user\
|
|
\ does not provide recorder id or name, the method will try to return the current\
|
|
\ active experiment.The `create` argument determines whether the method will automatically\
|
|
\ create a new experiment accordingto user's specification if the experiment hasn't\
|
|
\ been created before.* If `create` is True:* If `active experiment` exists:*\
|
|
\ no id or name specified, return the active experiment.* if id or name is specified,\
|
|
\ return the specified experiment. If no such exp found, create a new experiment\
|
|
\ with given id or name. If `start` is set to be True, the experiment is set to\
|
|
\ be active.* If `active experiment` not exists:* no id or name specified, create\
|
|
\ a default experiment.* if id or name is specified, return the specified experiment.\
|
|
\ If no such exp found, create a new experiment with given id or name. If `start`\
|
|
\ is set to be True, the experiment is set to be active.* Else If `create` is\
|
|
\ False:* If `active experiment` exists:* no id or name specified, return the\
|
|
\ active experiment.* if id or name is specified, return the specified experiment.\
|
|
\ If no such exp found, raise Error.* If `active experiment` not exists:* no\
|
|
\ id or name specified. If the default experiment exists, return it, otherwise,\
|
|
\ raise Error.* if id or name is specified, return the specified experiment. If\
|
|
\ no such exp found, raise Error.Parameters----------experiment_id : strid of\
|
|
\ the experiment to return.experiment_name : strname of the experiment to return.create\
|
|
\ : booleancreate the experiment it if hasn't been created before.start : booleanstart\
|
|
\ the new experiment if one is created.Returns-------An experiment object."
|
|
function: expm.get_exp
|
|
- docstring: " \nMethod for getting or creating an experiment. It will try\
|
|
\ to first get a valid experiment, if exception occurs, it willautomatically create\
|
|
\ a new experiment based on the given id and name."
|
|
function: expm._get_or_create_exp
|
|
- docstring: " \nGet specific experiment by name or id. If it does not exist,\
|
|
\ raise ValueError.Parameters----------experiment_id :The id of experimentexperiment_name\
|
|
\ :The name of experimentReturns-------Experiment:The searched experimentRaises------ValueError"
|
|
function: expm._get_exp
|
|
- docstring: " \nDelete an experiment.Parameters----------experiment_id :\
|
|
\ strthe experiment id.experiment_name : strthe experiment name."
|
|
function: expm.delete_exp
|
|
- docstring: " \nGet the default tracking URI from qlib.config.C"
|
|
function: expm.default_uri
|
|
- docstring: null
|
|
function: expm.default_uri
|
|
- docstring: " \nGet the default tracking URI or current URI.Returns-------The\
|
|
\ tracking URI string."
|
|
function: expm.uri
|
|
- docstring: " \nList all the existing experiments.Returns-------A dictionary\
|
|
\ (name -> experiment) of experiments information that being stored."
|
|
function: expm.list_experiments
|
|
- docstring: " \nUse mlflow to implement ExpManager."
|
|
function: expm.MLflowExpManager
|
|
- docstring: null
|
|
function: expm.client
|
|
- docstring: null
|
|
function: expm._start_exp
|
|
- docstring: null
|
|
function: expm._end_exp
|
|
- docstring: null
|
|
function: expm.create_exp
|
|
- docstring: " \nMethod for getting or creating an experiment. It will try\
|
|
\ to first get a valid experiment, if exception occurs, it willraise errors."
|
|
function: expm._get_exp
|
|
- docstring: null
|
|
function: expm.search_records
|
|
- docstring: null
|
|
function: expm.delete_exp
|
|
- docstring: " \nThis is the `Experiment` class for each experiment being run.\
|
|
\ The API is designed similar to mlflow.(The link: https://mlflow.org/docs/latest/python_api/mlflow.html)"
|
|
function: 'exp.Experiment:'
|
|
- docstring: null
|
|
function: exp.info
|
|
- docstring: " \nStart the experiment and set it to be active. This method\
|
|
\ will also start a new recorder.Parameters----------recorder_id : strthe id of\
|
|
\ the recorder to be created.recorder_name : strthe name of the recorder to be\
|
|
\ created.resume : boolwhether to resume the first recorderReturns-------An active\
|
|
\ recorder."
|
|
function: exp.start
|
|
- docstring: " \nEnd the experiment.Parameters----------recorder_status : strthe\
|
|
\ status the recorder to be set with when ending (SCHEDULED, RUNNING, FINISHED,\
|
|
\ FAILED)."
|
|
function: exp.end
|
|
- docstring: " \nCreate a recorder for each experiment.Parameters----------recorder_name\
|
|
\ : strthe name of the recorder to be created.Returns-------A recorder object."
|
|
function: exp.create_recorder
|
|
- docstring: " \nGet a pandas DataFrame of records that fit the search criteria\
|
|
\ of the experiment.Inputs are the search criteria user want to apply.Returns-------A\
|
|
\ pandas.DataFrame of records, where each metric, parameter, and tagare expanded\
|
|
\ into their own columns named metrics.*, params.*, and tags.*respectively. For\
|
|
\ records that don't have a particular metric, parameter, or tag, theirvalue will\
|
|
\ be (NumPy) Nan, None, or None respectively."
|
|
function: exp.search_records
|
|
- docstring: " \nCreate a recorder for each experiment.Parameters----------recorder_id\
|
|
\ : strthe id of the recorder to be deleted."
|
|
function: exp.delete_recorder
|
|
- docstring: " \nRetrieve a Recorder for user. When user specify recorder id\
|
|
\ and name, the method will try to return thespecific recorder. When user does\
|
|
\ not provide recorder id or name, the method will try to return the currentactive\
|
|
\ recorder. The `create` argument determines whether the method will automatically\
|
|
\ create a new recorderaccording to user's specification if the recorder hasn't\
|
|
\ been created before.* If `create` is True:* If `active recorder` exists:* no\
|
|
\ id or name specified, return the active recorder.* if id or name is specified,\
|
|
\ return the specified recorder. If no such exp found, create a new recorder with\
|
|
\ given id or name. If `start` is set to be True, the recorder is set to be active.*\
|
|
\ If `active recorder` not exists:* no id or name specified, create a new recorder.*\
|
|
\ if id or name is specified, return the specified experiment. If no such exp\
|
|
\ found, create a new recorder with given id or name. If `start` is set to be\
|
|
\ True, the recorder is set to be active.* Else If `create` is False:* If `active\
|
|
\ recorder` exists:* no id or name specified, return the active recorder.* if\
|
|
\ id or name is specified, return the specified recorder. If no such exp found,\
|
|
\ raise Error.* If `active recorder` not exists:* no id or name specified, raise\
|
|
\ Error.* if id or name is specified, return the specified recorder. If no such\
|
|
\ exp found, raise Error.Parameters----------recorder_id : strthe id of the recorder\
|
|
\ to be deleted.recorder_name : strthe name of the recorder to be deleted.create\
|
|
\ : booleancreate the recorder if it hasn't been created before.start : booleanstart\
|
|
\ the new recorder if one is **created**.Returns-------A recorder object."
|
|
function: exp.get_recorder
|
|
- docstring: " \nMethod for getting or creating a recorder. It will try to\
|
|
\ first get a valid recorder, if exception occurs, it willautomatically create\
|
|
\ a new recorder based on the given id and name."
|
|
function: exp._get_or_create_rec
|
|
- docstring: " \nGet specific recorder by name or id. If it does not exist,\
|
|
\ raise ValueErrorParameters----------recorder_id :The id of recorderrecorder_name\
|
|
\ :The name of recorderReturns-------Recorder:The searched recorderRaises------ValueError"
|
|
function: exp._get_recorder
|
|
- docstring: " \nList all the existing recorders of this experiment. Please\
|
|
\ first get the experiment instance before calling this method.If user want to\
|
|
\ use the method `R.list_recorders()`, please refer to the related API document\
|
|
\ in `QlibRecorder`.flt_kwargs : dictfilter recorders by conditionse.g. list_recorders(status=Recorder.STATUS_FI)Returns-------The\
|
|
\ return type depends on `rtype`if `rtype` == \"dict\":A dictionary (id -> recorder)\
|
|
\ of recorder information that being stored.elif `rtype` == \"list\":A list of\
|
|
\ Recorder."
|
|
function: exp.list_recorders
|
|
- docstring: " \nUse mlflow to implement Experiment."
|
|
function: exp.MLflowExperiment
|
|
- docstring: null
|
|
function: exp.start
|
|
- docstring: null
|
|
function: exp.end
|
|
- docstring: null
|
|
function: exp.create_recorder
|
|
- docstring: " \nMethod for getting or creating a recorder. It will try to\
|
|
\ first get a valid recorder, if exception occurs, it willraise errors.Quoting\
|
|
\ docs of search_runs from MLflow> The default ordering is to sort by start_time\
|
|
\ DESC, then run_id."
|
|
function: exp._get_recorder
|
|
- docstring: null
|
|
function: exp.search_records
|
|
- docstring: null
|
|
function: exp.delete_recorder
|
|
- docstring: " \nQuoting docs of search_runs> The default ordering is to sort\
|
|
\ by start_time DESC, then run_id.Parameters----------max_results : intthe number\
|
|
\ limitation of the results'status : strthe criteria based on status to filter\
|
|
\ results.`None` indicates no filtering.filter_string : strmlflow supported filter\
|
|
\ string like 'params.\"my_param\"=\"a\" and tags.\"my_tag\"=\"b\"', use this\
|
|
\ will help to reduce too much run number."
|
|
function: exp.list_recorders
|
|
- docstring: " \nThis is the `Recorder` class for logging the experiments. The\
|
|
\ API is designed similar to mlflow.(The link: https://mlflow.org/docs/latest/python_api/mlflow.html)The\
|
|
\ status of the recorder can be SCHEDULED, RUNNING, FINISHED, FAILED."
|
|
function: 'recorder.Recorder:'
|
|
- docstring: null
|
|
function: recorder.info
|
|
- docstring: null
|
|
function: recorder.set_recorder_name
|
|
- docstring: " \nSave objects such as prediction file or model checkpoints\
|
|
\ to the artifact URI. Usercan save object through keywords arguments (name:value).Please\
|
|
\ refer to the docs of qlib.workflow:R.save_objectsParameters----------local_path\
|
|
\ : strif provided, them save the file or directory to the artifact URI.artifact_path=None\
|
|
\ : strthe relative path for the artifact to be stored in the URI."
|
|
function: recorder.save_objects
|
|
- docstring: " \nLoad objects such as prediction file or model checkpoints.Parameters----------name\
|
|
\ : strname of the file to be loaded.Returns-------The saved object."
|
|
function: recorder.load_object
|
|
- docstring: " \nStart running or resuming the Recorder. The return value can\
|
|
\ be used as a context manager within a `with` block;otherwise, you must call\
|
|
\ end_run() to terminate the current run. (See `ActiveRun` class in mlflow)Returns-------An\
|
|
\ active running object (e.g. mlflow.ActiveRun object)."
|
|
function: recorder.start_run
|
|
- docstring: " \nEnd an active Recorder."
|
|
function: recorder.end_run
|
|
- docstring: " \nLog a batch of params for the current run.Parameters----------keyword\
|
|
\ argumentskey, value pair to be logged as parameters."
|
|
function: recorder.log_params
|
|
- docstring: " \nLog multiple metrics for the current run.Parameters----------keyword\
|
|
\ argumentskey, value pair to be logged as metrics."
|
|
function: recorder.log_metrics
|
|
- docstring: " \nLog a local file or directory as an artifact of the currently\
|
|
\ active run.Parameters----------local_path : strPath to the file to write.artifact_path\
|
|
\ : Optional[str]If provided, the directory in ``artifact_uri`` to write to."
|
|
function: recorder.log_artifact
|
|
- docstring: " \nLog a batch of tags for the current run.Parameters----------keyword\
|
|
\ argumentskey, value pair to be logged as tags."
|
|
function: recorder.set_tags
|
|
- docstring: " \nDelete some tags from a run.Parameters----------keys : series\
|
|
\ of strs of the keysall the name of the tag to be deleted."
|
|
function: recorder.delete_tags
|
|
- docstring: " \nList all the artifacts of a recorder.Parameters----------artifact_path\
|
|
\ : strthe relative path for the artifact to be stored in the URI.Returns-------A\
|
|
\ list of artifacts information (name, path, etc.) that being stored."
|
|
function: recorder.list_artifacts
|
|
- docstring: " \nDownload an artifact file or directory from a run to a local\
|
|
\ directory if applicable,and return a local path for it.Parameters----------path\
|
|
\ : strRelative source path to the desired artifact.dst_path : Optional[str]Absolute\
|
|
\ path of the local filesystem destination directory to which todownload the specified\
|
|
\ artifacts. This directory must already exist.If unspecified, the artifacts will\
|
|
\ either be downloaded to a newuniquely-named directory on the local filesystem.Returns-------strLocal\
|
|
\ path of desired artifact."
|
|
function: recorder.download_artifact
|
|
- docstring: " \nList all the metrics of a recorder.Returns-------A dictionary\
|
|
\ of metrics that being stored."
|
|
function: recorder.list_metrics
|
|
- docstring: " \nList all the params of a recorder.Returns-------A dictionary\
|
|
\ of params that being stored."
|
|
function: recorder.list_params
|
|
- docstring: " \nList all the tags of a recorder.Returns-------A dictionary\
|
|
\ of tags that being stored."
|
|
function: recorder.list_tags
|
|
- docstring: " \nUse mlflow to implement a Recorder.Due to the fact that mlflow\
|
|
\ will only log artifact from a file or directory, we decide touse file manager\
|
|
\ to help maintain the objects in the project.Instead of using mlflow directly,\
|
|
\ we use another interface wrapping mlflow to log experiments.Though it takes\
|
|
\ extra efforts, but it brings users benefits due to following reasons.- It will\
|
|
\ be more convenient to change the experiment logging backend without changing\
|
|
\ any code in upper level- We can provide more convenience to automatically do\
|
|
\ some extra things and make interface easier. For examples:- Automatically logging\
|
|
\ the uncommitted code- Automatically logging part of environment variables- User\
|
|
\ can control several different runs by just creating different Recorder (in mlflow,\
|
|
\ you always have to switch artifact_uri and pass in run ids frequently)"
|
|
function: recorder.MLflowRecorder
|
|
- docstring: null
|
|
function: recorder.uri
|
|
- docstring: null
|
|
function: recorder.artifact_uri
|
|
- docstring: " \nThis function will return the directory path of this recorder."
|
|
function: recorder.get_local_dir
|
|
- docstring: null
|
|
function: recorder.start_run
|
|
- docstring: " \nMlflow only log the commit id of the current repo. But usually,\
|
|
\ user will have a lot of uncommitted changes.So this tries to automatically to\
|
|
\ log them all."
|
|
function: recorder._log_uncommitted_code
|
|
- docstring: null
|
|
function: recorder.end_run
|
|
- docstring: null
|
|
function: recorder.save_objects
|
|
- docstring: " \nLoad object such as prediction file or model checkpoint in\
|
|
\ mlflow.Args:name (str): the object nameunpickler: Supporting using custom unpicklerRaises:LoadObjectError:\
|
|
\ if raise some exceptions when load the objectReturns:object: the saved object\
|
|
\ in mlflow."
|
|
function: recorder.load_object
|
|
- docstring: null
|
|
function: recorder.log_params
|
|
- docstring: null
|
|
function: recorder.log_metrics
|
|
- docstring: null
|
|
function: recorder.log_artifact
|
|
- docstring: null
|
|
function: recorder.set_tags
|
|
- docstring: null
|
|
function: recorder.delete_tags
|
|
- docstring: null
|
|
function: recorder.get_artifact_uri
|
|
- docstring: null
|
|
function: recorder.list_artifacts
|
|
- docstring: null
|
|
function: recorder.download_artifact
|
|
- docstring: null
|
|
function: recorder.list_metrics
|
|
- docstring: null
|
|
function: recorder.list_params
|
|
- docstring: " \nA global system that helps to manage the experiments."
|
|
function: '__init__.QlibRecorder:'
|
|
- docstring: " \nMethod to start an experiment. This method can only be called\
|
|
\ within a Python's `with` statement. Here is the example code:.. code-block::\
|
|
\ Python# start new experiment and recorderwith R.start(experiment_name='test',\
|
|
\ recorder_name='recorder_1'):model.fit(dataset)R.log...... # further operations#\
|
|
\ resume previous experiment and recorderwith R.start(experiment_name='test',\
|
|
\ recorder_name='recorder_1', resume=True): # if users want to resume recorder,\
|
|
\ they have to specify the exact same name for experiment and recorder.... # further\
|
|
\ operationsParameters----------experiment_id : strid of the experiment one wants\
|
|
\ to start.experiment_name : strname of the experiment one wants to start.recorder_id\
|
|
\ : strid of the recorder under the experiment one wants to start.recorder_name\
|
|
\ : strname of the recorder under the experiment one wants to start.uri : strThe\
|
|
\ tracking uri of the experiment, where all the artifacts/metrics etc. will be\
|
|
\ stored.The default uri is set in the qlib.config. Note that this uri argument\
|
|
\ will not change the one defined in the config file.Therefore, the next time\
|
|
\ when users call this function in the same experiment,they have to also specify\
|
|
\ this argument with the same value. Otherwise, inconsistent uri may occur.resume\
|
|
\ : boolwhether to resume the specific recorder with given name under the given\
|
|
\ experiment."
|
|
function: __init__.start
|
|
- docstring: " \nLower level method for starting an experiment. When use this\
|
|
\ method, one should end the experiment manuallyand the status of the recorder\
|
|
\ may not be handled properly. Here is the example code:.. code-block:: PythonR.start_exp(experiment_name='test',\
|
|
\ recorder_name='recorder_1')... # further operationsR.end_exp('FINISHED') or\
|
|
\ R.end_exp(Recorder.STATUS_S)Parameters----------experiment_id : strid of the\
|
|
\ experiment one wants to start.experiment_name : strthe name of the experiment\
|
|
\ to be startedrecorder_id : strid of the recorder under the experiment one wants\
|
|
\ to start.recorder_name : strname of the recorder under the experiment one wants\
|
|
\ to start.uri : strthe tracking uri of the experiment, where all the artifacts/metrics\
|
|
\ etc. will be stored.The default uri are set in the qlib.config.resume : boolwhether\
|
|
\ to resume the specific recorder with given name under the given experiment.Returns-------An\
|
|
\ experiment instance being started."
|
|
function: __init__.start_exp
|
|
- docstring: " \nMethod for ending an experiment manually. It will end the\
|
|
\ current active experiment, as well as itsactive recorder with the specified\
|
|
\ `status` type. Here is the example code of the method:.. code-block:: PythonR.start_exp(experiment_name='test')...\
|
|
\ # further operationsR.end_exp('FINISHED') or R.end_exp(Recorder.STATUS_S)Parameters----------status\
|
|
\ : strThe status of a recorder, which can be SCHEDULED, RUNNING, FINISHED, FAILED."
|
|
function: __init__.end_exp
|
|
- docstring: " \nGet a pandas DataFrame of records that fit the search criteria.The\
|
|
\ arguments of this function are not set to be rigid, and they will be different\
|
|
\ with different implementation of``ExpManager`` in ``Qlib``. ``Qlib`` now provides\
|
|
\ an implementation of ``ExpManager`` with mlflow, and here is theexample code\
|
|
\ of the method with the ``MLflowExpManager``:.. code-block:: PythonR.log_metrics(m=2.50,\
|
|
\ step=0)records = R.search_records([experiment_id], order_by=[\"metrics.m DESC\"\
|
|
])Parameters----------experiment_ids : listlist of experiment IDs.filter_string\
|
|
\ : strfilter query string, defaults to searching all runs.run_view_type : intone\
|
|
\ of enum values ACTIVE_ONLY, DELETED_ONLY, or ALL (e.g. in mlflow.entities.ViewType).max_results\
|
|
\ : intthe maximum number of runs to put in the dataframe.order_by : listlist\
|
|
\ of columns to order by (e.g., \u201Cmetrics.rmse\u201D).Returns-------A pandas.DataFrame\
|
|
\ of records, where each metric, parameter, and tagare expanded into their own\
|
|
\ columns named metrics.*, params.*, and tags.*respectively. For records that\
|
|
\ don't have a particular metric, parameter, or tag, theirvalue will be (NumPy)\
|
|
\ Nan, None, or None respectively."
|
|
function: __init__.search_records
|
|
- docstring: " \nMethod for listing all the existing experiments (except for\
|
|
\ those being deleted.).. code-block:: Pythonexps = R.list_experiments()Returns-------A\
|
|
\ dictionary (name -> experiment) of experiments information that being stored."
|
|
function: __init__.list_experiments
|
|
- docstring: " \nMethod for listing all the recorders of experiment with given\
|
|
\ id or name.If user doesn't provide the id or name of the experiment, this method\
|
|
\ will try to retrieve the default experiment andlist all the recorders of the\
|
|
\ default experiment. If the default experiment doesn't exist, the method will\
|
|
\ firstcreate the default experiment, and then create a new recorder under it.\
|
|
\ (More information about the default experimentcan be found `here <../component/recorder.html#qlib.workflow.exp.Experiment>`__).Here\
|
|
\ is the example code:.. code-block:: Pythonrecorders = R.list_recorders(experiment_name='test')Parameters----------experiment_id\
|
|
\ : strid of the experiment.experiment_name : strname of the experiment.Returns-------A\
|
|
\ dictionary (id -> recorder) of recorder information that being stored."
|
|
function: __init__.list_recorders
|
|
- docstring: " \nMethod for retrieving an experiment with given id or name.\
|
|
\ Once the `create` argument is set toTrue, if no valid experiment is found, this\
|
|
\ method will create one for you. Otherwise, it willonly retrieve a specific experiment\
|
|
\ or raise an Error.- If '`create`' is True:- If `active experiment` exists:-\
|
|
\ no id or name specified, return the active experiment.- if id or name is specified,\
|
|
\ return the specified experiment. If no such exp found, create a new experiment\
|
|
\ with given id or name.- If `active experiment` not exists:- no id or name specified,\
|
|
\ create a default experiment, and the experiment is set to be active.- if id\
|
|
\ or name is specified, return the specified experiment. If no such exp found,\
|
|
\ create a new experiment with given name or the default experiment.- Else If\
|
|
\ '`create`' is False:- If `active experiment` exists:- no id or name specified,\
|
|
\ return the active experiment.- if id or name is specified, return the specified\
|
|
\ experiment. If no such exp found, raise Error.- If `active experiment` not exists:-\
|
|
\ no id or name specified. If the default experiment exists, return it, otherwise,\
|
|
\ raise Error.- if id or name is specified, return the specified experiment. If\
|
|
\ no such exp found, raise Error.Here are some use cases:.. code-block:: Python#\
|
|
\ Case 1with R.start('test'):exp = R.get_exp()recorders = exp.list_recorders()#\
|
|
\ Case 2with R.start('test'):exp = R.get_exp(experiment_name='test1')# Case 3exp\
|
|
\ = R.get_exp() -> a default experiment.# Case 4exp = R.get_exp(experiment_name='test')#\
|
|
\ Case 5exp = R.get_exp(create=False) -> the default experiment if exists.Parameters----------experiment_id\
|
|
\ : strid of the experiment.experiment_name : strname of the experiment.create\
|
|
\ : booleanan argument determines whether the method will automatically create\
|
|
\ a new experimentaccording to user's specification if the experiment hasn't been\
|
|
\ created before.start : boolwhen start is True,if the experiment has not started(not\
|
|
\ activated), it will startIt is designed for R.log_params to auto start experimentsReturns-------An\
|
|
\ experiment instance with given id or name."
|
|
function: __init__.get_exp
|
|
- docstring: " \nMethod for deleting the experiment with given id or name.\
|
|
\ At least one of id or name must be given,otherwise, error will occur.Here is\
|
|
\ the example code:.. code-block:: PythonR.delete_exp(experiment_name='test')Parameters----------experiment_id\
|
|
\ : strid of the experiment.experiment_name : strname of the experiment."
|
|
function: __init__.delete_exp
|
|
- docstring: " \nMethod for retrieving the uri of current experiment manager.Here\
|
|
\ is the example code:.. code-block:: Pythonuri = R.get_uri()Returns-------The\
|
|
\ uri of current experiment manager."
|
|
function: __init__.get_uri
|
|
- docstring: " \nMethod to reset the **default** uri of current experiment\
|
|
\ manager.NOTE:- When the uri is refer to a file path, please using the absolute\
|
|
\ path instead of strings like \"~/mlruns/\"The backend don't support strings\
|
|
\ like this."
|
|
function: __init__.set_uri
|
|
- docstring: " \nTemporarily set the exp_manager's **default_uri** to uriNOTE:-\
|
|
\ Please refer to the NOTE in the `set_uri`Parameters----------uri : Textthe temporal\
|
|
\ uri"
|
|
function: __init__.uri_context
|
|
- docstring: " \nMethod for retrieving a recorder.- If `active recorder` exists:-\
|
|
\ no id or name specified, return the active recorder.- if id or name is specified,\
|
|
\ return the specified recorder.- If `active recorder` not exists:- no id or name\
|
|
\ specified, raise Error.- if id or name is specified, and the corresponding experiment_name\
|
|
\ must be given, return the specified recorder. Otherwise, raise Error.The recorder\
|
|
\ can be used for further process such as `save_object`, `load_object`, `log_params`,`log_metrics`,\
|
|
\ etc.Here are some use cases:.. code-block:: Python# Case 1with R.start(experiment_name='test'):recorder\
|
|
\ = R.get_recorder()# Case 2with R.start(experiment_name='test'):recorder = R.get_recorder(recorder_id='2e7a4efd66574fa49039e00ffaefa99d')#\
|
|
\ Case 3recorder = R.get_recorder() -> Error# Case 4recorder = R.get_recorder(recorder_id='2e7a4efd66574fa49039e00ffaefa99d')\
|
|
\ -> Error# Case 5recorder = R.get_recorder(recorder_id='2e7a4efd66574fa49039e00ffaefa99d',\
|
|
\ experiment_name='test')Here are some things users may concern- Q: What recorder\
|
|
\ will it return if multiple recorder meets the query (e.g. query with experiment_name)-\
|
|
\ A: If mlflow backend is used, then the recorder with the latest `start_time`\
|
|
\ will be returned. Because MLflow's `search_runs` function guarantee itParameters----------recorder_id\
|
|
\ : strid of the recorder.recorder_name : strname of the recorder.experiment_name\
|
|
\ : strname of the experiment.Returns-------A recorder instance."
|
|
function: __init__.get_recorder
|
|
- docstring: " \nMethod for deleting the recorders with given id or name. At\
|
|
\ least one of id or name must be given,otherwise, error will occur.Here is the\
|
|
\ example code:.. code-block:: PythonR.delete_recorder(recorder_id='2e7a4efd66574fa49039e00ffaefa99d')Parameters----------recorder_id\
|
|
\ : strid of the experiment.recorder_name : strname of the experiment."
|
|
function: __init__.delete_recorder
|
|
- docstring: " \nMethod for saving objects as artifacts in the experiment to\
|
|
\ the uri. It supports either savingfrom a local file/directory, or directly saving\
|
|
\ objects. User can use valid python's keywords argumentsto specify the object\
|
|
\ to be saved as well as its name (name: value).In summary, this API is designs\
|
|
\ for saving **objects** to **the experiments management backend path**,1. Qlib\
|
|
\ provide two methods to specify **objects**- Passing in the object directly by\
|
|
\ passing with `**kwargs` (e.g. R.save_objects(trained_model=model))- Passing\
|
|
\ in the local path to the object, i.e. `local_path` parameter.2. `artifact_path`\
|
|
\ represents the **the experiments management backend path**- If `active recorder`\
|
|
\ exists: it will save the objects through the active recorder.- If `active recorder`\
|
|
\ not exists: the system will create a default experiment, and a new recorder\
|
|
\ and save objects under it... note::If one wants to save objects with a specific\
|
|
\ recorder. It is recommended to first get the specific recorder through `get_recorder`\
|
|
\ API and use the recorder the save objects. The supported arguments are the same\
|
|
\ as this method.Here are some use cases:.. code-block:: Python# Case 1with R.start(experiment_name='test'):pred\
|
|
\ = model.predict(dataset)R.save_objects(**{\"pred.pkl\": pred}, artifact_path='prediction')rid\
|
|
\ = R.get_recorder().id...R.get_recorder(recorder_id=rid).load_object(\"prediction/pred.pkl\"\
|
|
) # after saving objects, you can load the previous object with this api# Case\
|
|
\ 2with R.start(experiment_name='test'):R.save_objects(local_path='results/pred.pkl',\
|
|
\ artifact_path=\"prediction\")rid = R.get_recorder().id...R.get_recorder(recorder_id=rid).load_object(\"\
|
|
prediction/pred.pkl\") # after saving objects, you can load the previous object\
|
|
\ with this apiParameters----------local_path : strif provided, them save the\
|
|
\ file or directory to the artifact URI.artifact_path : strthe relative path for\
|
|
\ the artifact to be stored in the URI.**kwargs: Dict[Text, Any]the object to\
|
|
\ be saved.For example, `{\"pred.pkl\": pred}`"
|
|
function: __init__.save_objects
|
|
- docstring: " \nMethod for loading an object from artifacts in the experiment\
|
|
\ in the uri."
|
|
function: __init__.load_object
|
|
- docstring: " \nMethod for logging parameters during an experiment. In addition\
|
|
\ to using ``R``, one can also log to a specific recorder after getting it with\
|
|
\ `get_recorder` API.- If `active recorder` exists: it will log parameters through\
|
|
\ the active recorder.- If `active recorder` not exists: the system will create\
|
|
\ a default experiment as well as a new recorder, and log parameters under it.Here\
|
|
\ are some use cases:.. code-block:: Python# Case 1with R.start('test'):R.log_params(learning_rate=0.01)#\
|
|
\ Case 2R.log_params(learning_rate=0.01)Parameters----------keyword argument:name1=value1,\
|
|
\ name2=value2, ..."
|
|
function: __init__.log_params
|
|
- docstring: " \nMethod for logging metrics during an experiment. In addition\
|
|
\ to using ``R``, one can also log to a specific recorder after getting it with\
|
|
\ `get_recorder` API.- If `active recorder` exists: it will log metrics through\
|
|
\ the active recorder.- If `active recorder` not exists: the system will create\
|
|
\ a default experiment as well as a new recorder, and log metrics under it.Here\
|
|
\ are some use cases:.. code-block:: Python# Case 1with R.start('test'):R.log_metrics(train_loss=0.33,\
|
|
\ step=1)# Case 2R.log_metrics(train_loss=0.33, step=1)Parameters----------keyword\
|
|
\ argument:name1=value1, name2=value2, ..."
|
|
function: __init__.log_metrics
|
|
- docstring: " \nLog a local file or directory as an artifact of the currently\
|
|
\ active run- If `active recorder` exists: it will set tags through the active\
|
|
\ recorder.- If `active recorder` not exists: the system will create a default\
|
|
\ experiment as well as a new recorder, and set the tags under it.Parameters----------local_path\
|
|
\ : strPath to the file to write.artifact_path : Optional[str]If provided, the\
|
|
\ directory in ``artifact_uri`` to write to."
|
|
function: __init__.log_artifact
|
|
- docstring: " \nDownload an artifact file or directory from a run to a local\
|
|
\ directory if applicable,and return a local path for it.Parameters----------path\
|
|
\ : strRelative source path to the desired artifact.dst_path : Optional[str]Absolute\
|
|
\ path of the local filesystem destination directory to which todownload the specified\
|
|
\ artifacts. This directory must already exist.If unspecified, the artifacts will\
|
|
\ either be downloaded to a newuniquely-named directory on the local filesystem.Returns-------strLocal\
|
|
\ path of desired artifact."
|
|
function: __init__.download_artifact
|
|
- docstring: " \nMethod for setting tags for a recorder. In addition to using\
|
|
\ ``R``, one can also set the tag to a specific recorder after getting it with\
|
|
\ `get_recorder` API.- If `active recorder` exists: it will set tags through the\
|
|
\ active recorder.- If `active recorder` not exists: the system will create a\
|
|
\ default experiment as well as a new recorder, and set the tags under it.Here\
|
|
\ are some use cases:.. code-block:: Python# Case 1with R.start('test'):R.set_tags(release_version=\"\
|
|
2.2.0\")# Case 2R.set_tags(release_version=\"2.2.0\")Parameters----------keyword\
|
|
\ argument:name1=value1, name2=value2, ..."
|
|
function: __init__.set_tags
|
|
- docstring: " \nWrapper class for QlibRecorder, which detects whether users reinitialize\
|
|
\ qlib when already starting an experiment."
|
|
function: __init__.RecorderWrapper
|
|
- docstring: " \nOnlineTool will manage `online` models in an experiment that includes\
|
|
\ the model recorders."
|
|
function: 'utils.OnlineTool:'
|
|
- docstring: " \nSet `tag` to the model to sign whether online.Args:tag (str):\
|
|
\ the tags in `ONLINE_TAG`, `OFFLINE_TAG`recorder (Union[list,object]): the model's\
|
|
\ recorder"
|
|
function: utils.set_online_tag
|
|
- docstring: " \nGiven a model recorder and return its online tag.Args:recorder\
|
|
\ (Object): the model's recorderReturns:str: the online tag"
|
|
function: utils.get_online_tag
|
|
- docstring: " \nOffline all models and set the recorders to 'online'.Args:recorder\
|
|
\ (Union[list,object]):the recorder you want to reset to 'online'."
|
|
function: utils.reset_online_tag
|
|
- docstring: " \nGet current `online` modelsReturns:list: a list of `online`\
|
|
\ models."
|
|
function: utils.online_models
|
|
- docstring: " \nUpdate the predictions of `online` models to to_date.Args:to_date\
|
|
\ (pd.Timestamp): the pred before this date will be updated. None for updating\
|
|
\ to the latest."
|
|
function: utils.update_online_pred
|
|
- docstring: " \nThe implementation of OnlineTool based on (R)ecorder."
|
|
function: utils.OnlineToolR
|
|
- docstring: " \nSet `tag` to the model's recorder to sign whether online.Args:tag\
|
|
\ (str): the tags in `ONLINE_TAG`, `NEXT_ONLINE_TAG`, `OFFLINE_TAG`recorder (Union[Recorder,\
|
|
\ List]): a list of Recorder or an instance of Recorder"
|
|
function: utils.set_online_tag
|
|
- docstring: " \nGiven a model recorder and return its online tag.Args:recorder\
|
|
\ (Recorder): an instance of recorderReturns:str: the online tag"
|
|
function: utils.get_online_tag
|
|
- docstring: " \nOffline all models and set the recorders to 'online'.Args:recorder\
|
|
\ (Union[Recorder, List]):the recorder you want to reset to 'online'.exp_name\
|
|
\ (str): the experiment name. If None, then use default_exp_name."
|
|
function: utils.reset_online_tag
|
|
- docstring: " \nGet current `online` modelsArgs:exp_name (str): the experiment\
|
|
\ name. If None, then use default_exp_name.Returns:list: a list of `online` models."
|
|
function: utils.online_models
|
|
- docstring: " \nUpdate the predictions of online models to to_date.Args:to_date\
|
|
\ (pd.Timestamp): the pred before this date will be updated. None for updating\
|
|
\ to latest time in Calendar.exp_name (str): the experiment name. If None, then\
|
|
\ use default_exp_name."
|
|
function: utils.update_online_pred
|
|
- docstring: " \nRecorder Model Dataset Loader"
|
|
function: 'update.RMDLoader:'
|
|
- docstring: " \nLoad, config and setup dataset.This dataset is for inference.Args:start_time\
|
|
\ :the start_time of underlying dataend_time :the end_time of underlying datasegments\
|
|
\ : dictthe segments config for datasetDue to the time series dataset (TSDatasetH),\
|
|
\ the test segments maybe different from start_time and end_timeunprepared_dataset:\
|
|
\ Optional[DatasetH]if user don't want to load dataset from recorder, please specify\
|
|
\ user's datasetReturns:DatasetH: the instance of DatasetH"
|
|
function: update.get_dataset
|
|
- docstring: null
|
|
function: update.get_model
|
|
- docstring: " \nUpdate a specific recorders"
|
|
function: update.RecordUpdater
|
|
- docstring: " \nUpdate info for specific recorder"
|
|
function: update.update
|
|
- docstring: " \nDataset-Based Updater- Providing updating feature for Updating\
|
|
\ data based on Qlib DatasetAssumption- Based on Qlib dataset- The data to be\
|
|
\ updated is a multi-level index pd.DataFrame. For example label, prediction...\
|
|
\ code-block::LABEL0datetime instrument2021-05-10 SH600000 0.006965SH600004\
|
|
\ 0.003407... ...2021-05-28 SZ300498 0.015748SZ300676\
|
|
\ -0.001321"
|
|
function: update.DSBasedUpdater
|
|
- docstring: " \nLoad dataset- if unprepared_dataset is specified, then prepare\
|
|
\ the dataset directly- Otherwise,Separating this function will make it easier\
|
|
\ to reuse the datasetReturns:DatasetH: the instance of DatasetH"
|
|
function: update.prepare_data
|
|
- docstring: " \nParameters----------dataset : DatasetHDatasetH: the instance\
|
|
\ of DatasetH. None for prepare it again.write : boolwill the the write action\
|
|
\ be executedret_new : boolwill the updated data be returnedReturns-------Optional[object]the\
|
|
\ updated dataset"
|
|
function: update.update
|
|
- docstring: " \nreturn the updated data based on the given datasetThe difference\
|
|
\ between `get_update_data` and `update`- `update_date` only include some data\
|
|
\ specific feature- `update` include some general routine steps(e.g. prepare dataset,\
|
|
\ checking)"
|
|
function: update.get_update_data
|
|
- docstring: null
|
|
function: update._replace_range
|
|
- docstring: " \nUpdate the prediction in the Recorder"
|
|
function: update.PredUpdater
|
|
- docstring: null
|
|
function: update.get_update_data
|
|
- docstring: " \nUpdate the label in the recorderAssumption- The label is generated\
|
|
\ from record_temp.SignalRecord."
|
|
function: update.LabelUpdater
|
|
- docstring: " \nOnlineStrategy is working with `Online Manager <#Online Manager>`_,\
|
|
\ responding to how the tasks are generated, the models are updated and signals\
|
|
\ are prepared."
|
|
function: 'strategy.OnlineStrategy:'
|
|
- docstring: " \nAfter the end of a routine, check whether we need to prepare\
|
|
\ and train some new tasks based on cur_time (None for latest)..Return the new\
|
|
\ tasks waiting for training.You can find the last online models by OnlineTool.online_models."
|
|
function: strategy.prepare_tasks
|
|
- docstring: " \nSelect some models from trained models and set them to online\
|
|
\ models.This is a typical implementation to online all trained models, you can\
|
|
\ override it to implement the complex method.You can find the last online models\
|
|
\ by OnlineTool.online_models if you still need them.NOTE: Reset all online models\
|
|
\ to trained models. If there are no trained models, then do nothing.**NOTE**:Current\
|
|
\ implementation is very naive. Here is a more complex situation which is more\
|
|
\ closer to thepractical scenarios.1. Train new models at the day before `test_start`\
|
|
\ (at time stamp `T`)2. Switch models at the `test_start` (at time timestamp `T\
|
|
\ + 1` typically)Args:models (list): a list of models.cur_time (pd.Dataframe):\
|
|
\ current time from OnlineManger. None for the latest.Returns:List[object]: a\
|
|
\ list of online models."
|
|
function: strategy.prepare_online_models
|
|
- docstring: " \nGenerate a series of tasks firstly and return them."
|
|
function: strategy.first_tasks
|
|
- docstring: " \nGet the instance of `Collector <../advanced/task_management.html#Task\
|
|
\ Collecting>`_ to collect different results of this strategy.For example:1) collect\
|
|
\ predictions in Recorder2) collect signals in a txt fileReturns:Collector"
|
|
function: strategy.get_collector
|
|
- docstring: " \nThis example strategy always uses the latest rolling model sas\
|
|
\ online models."
|
|
function: strategy.RollingStrategy
|
|
- docstring: " \nGet the instance of `Collector <../advanced/task_management.html#Task\
|
|
\ Collecting>`_ to collect results. The returned collector must distinguish results\
|
|
\ in different models.Assumption: the models can be distinguished based on the\
|
|
\ model name and rolling test segments.If you do not want this assumption, please\
|
|
\ implement your method or use another rec_key_func.Args:rec_key_func (Callable):\
|
|
\ a function to get the key of a recorder. If None, use recorder id.rec_filter_func\
|
|
\ (Callable, optional): filter the recorder by return True or False. Defaults\
|
|
\ to None.artifacts_key (List[str], optional): the artifacts key you want to get.\
|
|
\ If None, get all artifacts."
|
|
function: strategy.get_collector
|
|
- docstring: null
|
|
function: strategy.rec_key
|
|
- docstring: " \nUse rolling_gen to generate different tasks based on task_template.Returns:List[dict]:\
|
|
\ a list of tasks"
|
|
function: strategy.first_tasks
|
|
- docstring: " \nPrepare new tasks based on cur_time (None for the latest).You\
|
|
\ can find the last online models by OnlineToolR.online_models.Returns:List[dict]:\
|
|
\ a list of new tasks."
|
|
function: strategy.prepare_tasks
|
|
- docstring: " \nList latest recorder form rec_listArgs:rec_list (List[Recorder]):\
|
|
\ a list of RecorderReturns:List[Recorder], pd.Timestamp: the latest recorders\
|
|
\ and their test end time"
|
|
function: strategy._list_latest
|
|
- docstring: " \nOnlineManager can manage online models with `Online Strategy <#Online\
|
|
\ Strategy>`_.It also provides a history recording of which models are online\
|
|
\ at what time."
|
|
function: manager.OnlineManager
|
|
- docstring: " \nShould the workflow to postpone the following actions to the\
|
|
\ end (in delay_prepare)- trainer.end_train- prepare_signalsPostpone these actions\
|
|
\ is to support simulating/backtest online strategies without time dependencies.All\
|
|
\ the actions can be done parallelly at the end."
|
|
function: manager._postpone_action
|
|
- docstring: " \nGet tasks from every strategy's first_tasks method and train\
|
|
\ them.If using DelayTrainer, it can finish training all together after every\
|
|
\ strategy's first_tasks.Args:strategies (List[OnlineStrategy]): the strategies\
|
|
\ list (need this param when adding strategies). None for use default strategies.model_kwargs\
|
|
\ (dict): the params for `prepare_online_models`"
|
|
function: manager.first_train
|
|
- docstring: " \nTypical update process for every strategy and record the online\
|
|
\ history.The typical update process after a routine, such as day by day or month\
|
|
\ by month.The process is: Update predictions -> Prepare tasks -> Prepare online\
|
|
\ models -> Prepare signals.If using DelayTrainer, it can finish training all\
|
|
\ together after every strategy's prepare_tasks.Args:cur_time (Union[str,pd.Timestamp],\
|
|
\ optional): run routine method in this time. Defaults to None.task_kwargs (dict):\
|
|
\ the params for `prepare_tasks`model_kwargs (dict): the params for `prepare_online_models`signal_kwargs\
|
|
\ (dict): the params for `prepare_signals`"
|
|
function: manager.routine
|
|
- docstring: " \nGet the instance of `Collector <../advanced/task_management.html#Task\
|
|
\ Collecting>`_ to collect results from every strategy.This collector can be a\
|
|
\ basis as the signals preparation.Args:**kwargs: the params for get_collector.Returns:MergeCollector:\
|
|
\ the collector to merge other collectors."
|
|
function: manager.get_collector
|
|
- docstring: " \nAdd some new strategies to OnlineManager.Args:strategy (Union[OnlineStrategy,\
|
|
\ List[OnlineStrategy]]): a list of OnlineStrategy"
|
|
function: manager.add_strategy
|
|
- docstring: " \nAfter preparing the data of the last routine (a box in box-plot)\
|
|
\ which means the end of the routine, we can prepare trading signals for the next\
|
|
\ routine.NOTE: Given a set prediction, all signals before these prediction end\
|
|
\ times will be prepared well.Even if the latest signal already exists, the latest\
|
|
\ calculation result will be overwritten... note::Given a prediction of a certain\
|
|
\ time, all signals before this time will be prepared well.Args:prepare_func (Callable,\
|
|
\ optional): Get signals from a dict after collecting. Defaults to AverageEnsemble(),\
|
|
\ the results collected by MergeCollector must be {xxx:pred}.over_write (bool,\
|
|
\ optional): If True, the new signals will overwrite. If False, the new signals\
|
|
\ will append to the end of signals. Defaults to False.Returns:pd.DataFrame: the\
|
|
\ signals."
|
|
function: manager.prepare_signals
|
|
- docstring: " \nGet prepared online signals.Returns:Union[pd.Series, pd.DataFrame]:\
|
|
\ pd.Series for only one signals every datetime.pd.DataFrame for multiple signals,\
|
|
\ for example, buy and sell operations use different trading signals."
|
|
function: manager.get_signals
|
|
- docstring: " \nStarting from the current time, this method will simulate\
|
|
\ every routine in OnlineManager until the end time.Considering the parallel training,\
|
|
\ the models and signals can be prepared after all routine simulating.The delay\
|
|
\ training way can be ``DelayTrainer`` and the delay preparing signals way can\
|
|
\ be ``delay_prepare``.Args:end_time: the time the simulation will endfrequency:\
|
|
\ the calendar frequencytask_kwargs (dict): the params for `prepare_tasks`model_kwargs\
|
|
\ (dict): the params for `prepare_online_models`signal_kwargs (dict): the params\
|
|
\ for `prepare_signals`Returns:Union[pd.Series, pd.DataFrame]: pd.Series for only\
|
|
\ one signals every datetime.pd.DataFrame for multiple signals, for example, buy\
|
|
\ and sell operations use different trading signals."
|
|
function: manager.simulate
|
|
- docstring: " \nPrepare all models and signals if something is waiting for\
|
|
\ preparation.Args:model_kwargs: the params for `end_train`signal_kwargs: the\
|
|
\ params for `prepare_signals`"
|
|
function: manager.delay_prepare
|
|
- docstring: " \nGet database in MongoDB, which means you need to declare the address\
|
|
\ and the name of a database at first.For example:Using qlib.init():.. code-block::\
|
|
\ pythonmongo_conf = {\"task_url\": task_url, # your MongoDB url\"task_db_name\"\
|
|
: task_db_name, # database name}qlib.init(..., mongo=mongo_conf)After qlib.init():..\
|
|
\ code-block:: pythonC[\"mongo\"] = {\"task_url\" : \"mongodb://localhost:27017/\"\
|
|
,\"task_db_name\" : \"rolling_db\"}Returns:Database: the Database instance"
|
|
function: utils.get_mongodb
|
|
- docstring: " \nList all recorders which can pass the filter in an experiment.Args:experiment\
|
|
\ (str or Experiment): the name of an Experiment or an instancerec_filter_func\
|
|
\ (Callable, optional): return True to retain the given recorder. Defaults to\
|
|
\ None.Returns:dict: a dict {rid: recorder} after filtering."
|
|
function: utils.list_recorders
|
|
- docstring: " \nFind appropriate date and adjust date."
|
|
function: 'utils.TimeAdjuster:'
|
|
- docstring: " \nSet end time. None for use calendar's end time.Args:end_time"
|
|
function: utils.set_end_time
|
|
- docstring: " \nGet datetime by index.Parameters----------idx : intindex of\
|
|
\ the calendar"
|
|
function: utils.get
|
|
- docstring: " \nReturn the max calendar datetime"
|
|
function: utils.max
|
|
- docstring: " \nAlign the index of time_point in the calendar.Parameters----------time_pointtp_type\
|
|
\ : strReturns-------index : int"
|
|
function: utils.align_idx
|
|
- docstring: " \nCalculate the trading day interval (time_point_A - time_point_B)Args:time_point_A\
|
|
\ : time_point_Atime_point_B : time_point_B (is the past of time_point_A)Returns:int:\
|
|
\ the interval between A and B"
|
|
function: utils.cal_interval
|
|
- docstring: " \nAlign time_point to trade date of calendarArgs:time_pointTime\
|
|
\ pointtp_type : strtime point type (`\"start\"`, `\"end\"`)Returns:pd.Timestamp"
|
|
function: utils.align_time
|
|
- docstring: " \nAlign the given date to the trade datefor example:.. code-block::\
|
|
\ pythoninput: {'train': ('2008-01-01', '2014-12-31'), 'valid': ('2015-01-01',\
|
|
\ '2016-12-31'), 'test': ('2017-01-01', '2020-08-01')}output: {'train': (Timestamp('2008-01-02\
|
|
\ 00:00:00'), Timestamp('2014-12-31 00:00:00')),'valid': (Timestamp('2015-01-05\
|
|
\ 00:00:00'), Timestamp('2016-12-30 00:00:00')),'test': (Timestamp('2017-01-03\
|
|
\ 00:00:00'), Timestamp('2020-07-31 00:00:00'))}Parameters----------segmentReturns-------Union[dict,\
|
|
\ tuple]: the start and end trade date (pd.Timestamp) between the given start\
|
|
\ and end date."
|
|
function: utils.align_seg
|
|
- docstring: " \nTruncate the segment based on the test_start dateParameters----------segment\
|
|
\ : tupletime segmenttest_startdays : intThe trading days to be truncatedthe data\
|
|
\ in this segment may need 'days' data`days` are based on the `test_start`.For\
|
|
\ example, if the label contains the information of 2 days in the near future,\
|
|
\ the prediction horizon 1 day.(e.g. the prediction target is `Ref($close, -2)/Ref($close,\
|
|
\ -1) - 1`)the days should be 2 + 1 == 3 days.Returns---------tuple: new segment"
|
|
function: utils.truncate
|
|
- docstring: null
|
|
function: utils._add_step
|
|
- docstring: " \nShift the datatime of segmentIf there are None (which indicates\
|
|
\ unbounded index) in the segment, this method will return None.Parameters----------seg\
|
|
\ :datetime segmentstep : introlling steprtype : strrolling type (\"sliding\"\
|
|
\ or \"expanding\")Returns--------tuple: new segmentRaises------KeyError:shift\
|
|
\ will raise error if the index(both start and end) is out of self.cal"
|
|
function: utils.shift
|
|
- docstring: " \nReplace the handler in task with a cache handler.It will automatically\
|
|
\ cache the file and save it in cache_dir.>>> import qlib>>> qlib.auto_init()>>>\
|
|
\ import datetime>>> # it is simplified task>>> task = {\"dataset\": {\"kwargs\"\
|
|
:{'handler': {'class': 'Alpha158', 'module_path': 'qlib.contrib.data.handler',\
|
|
\ 'kwargs': {'start_time': datetime.date(2008, 1, 1), 'end_time': datetime.date(2020,\
|
|
\ 8, 1), 'fit_start_time': datetime.date(2008, 1, 1), 'fit_end_time': datetime.date(2014,\
|
|
\ 12, 31), 'instruments': 'CSI300'}}}}}>>> new_task = replace_task_handler_with_cache(task)>>>\
|
|
\ print(new_task){'dataset': {'kwargs': {'handler': 'file...Alpha158.3584f5f8b4.pkl'}}}"
|
|
function: utils.replace_task_handler_with_cache
|
|
- docstring: ' The collector to collect different results
|
|
|
|
pickle_backend = "dill" # use dill to dump user method'
|
|
function: collect.Collector
|
|
- docstring: " \nCollect the results and return a dict like {key: things}Returns:dict:\
|
|
\ the dict after collecting.For example:{\"prediction\": pd.Series}{\"IC\": {\"\
|
|
Xgboost\": pd.Series, \"LSTM\": pd.Series}}..."
|
|
function: collect.collect
|
|
- docstring: " \nDo a series of processing to the dict returned by collect\
|
|
\ and return a dict like {key: things}For example, you can group and ensemble.Args:collected_dict\
|
|
\ (dict): the dict return by `collect`process_list (list or Callable): the list\
|
|
\ of processors or the instance of a processor to process dict.The processor order\
|
|
\ is the same as the list order.For example: [Group1(..., Ensemble1()), Group2(...,\
|
|
\ Ensemble2())]Returns:dict: the dict after processing."
|
|
function: collect.process_collect
|
|
- docstring: " \nA collector to collect the results of other CollectorsFor example:We\
|
|
\ have 2 collector, which named A and B.A can collect {\"prediction\": pd.Series}\
|
|
\ and B can collect {\"IC\": {\"Xgboost\": pd.Series, \"LSTM\": pd.Series}}.Then\
|
|
\ after this class's collect, we can collect {\"A_prediction\": pd.Series, \"\
|
|
B_IC\": {\"Xgboost\": pd.Series, \"LSTM\": pd.Series}}..."
|
|
function: collect.MergeCollector
|
|
- docstring: " \nCollect all results of collector_dict and change the outermost\
|
|
\ key to a recombination key.Returns:dict: the dict after collecting."
|
|
function: collect.collect
|
|
- docstring: " \nInit RecorderCollector.Args:experiment:(Experiment or str):\
|
|
\ an instance of an Experiment or the name of an Experiment(Callable): an callable\
|
|
\ function, which returns a list of experimentsprocess_list (list or Callable):\
|
|
\ the list of processors or the instance of a processor to process dict.rec_key_func\
|
|
\ (Callable): a function to get the key of a recorder. If None, use recorder id.rec_filter_func\
|
|
\ (Callable, optional): filter the recorder by return True or False. Defaults\
|
|
\ to None.artifacts_path (dict, optional): The artifacts name and its path in\
|
|
\ Recorder. Defaults to {\"pred\": \"pred.pkl\", \"IC\": \"sig_analysis/ic.pkl\"\
|
|
}.artifacts_key (str or List, optional): the artifacts key you want to get. If\
|
|
\ None, get all artifacts.list_kwargs (str): arguments for list_recorders function.status\
|
|
\ (Iterable): only collect recorders with specific status. None indicating collecting\
|
|
\ all the recorders"
|
|
function: collect.RecorderCollector
|
|
- docstring: null
|
|
function: collect.rec_key_func
|
|
- docstring: " \nCollect different artifacts based on recorder after filtering.Args:artifacts_key\
|
|
\ (str or List, optional): the artifacts key you want to get. If None, use the\
|
|
\ default.rec_filter_func (Callable, optional): filter the recorder by return\
|
|
\ True or False. If None, use the default.only_exist (bool, optional): if only\
|
|
\ collect the artifacts when a recorder really has.If True, the recorder with\
|
|
\ exception when loading will not be collected. But if False, it will raise the\
|
|
\ exception.Returns:dict: the dict after collected like {artifact: {rec_key: object}}"
|
|
function: collect.collect
|
|
- docstring: " \nGet experiment nameReturns:str: experiment name"
|
|
function: collect.get_exp_name
|
|
- docstring: " \nTaskManagerHere is what will a task looks like when it created\
|
|
\ by TaskManager.. code-block:: python{'def': pickle serialized task definition.\
|
|
\ using pickle will make it easier'filter': json-like data. This is for filtering\
|
|
\ the tasks.'status': 'waiting' | 'running' | 'done''res': pickle serialized task\
|
|
\ result,}The tasks manager assumes that you will only update the tasks you fetched.The\
|
|
\ mongo fetch one and update will make it date updating secure.This class can\
|
|
\ be used as a tool from commandline. Here are several examples.You can view the\
|
|
\ help of manage module with the following commands:python -m qlib.workflow.task.manage\
|
|
\ -h # show manual of manage module CLIpython -m qlib.workflow.task.manage wait\
|
|
\ -h # show manual of the wait command of manage.. code-block:: shellpython -m\
|
|
\ qlib.workflow.task.manage -t <pool_name> waitpython -m qlib.workflow.task.manage\
|
|
\ -t <pool_name> task_stat.. note::Assumption: the data in MongoDB was encoded\
|
|
\ and the data out of MongoDB was decodedHere are four status which are:STATUS_WAITING:\
|
|
\ waiting for trainingSTATUS_RUNNING: trainingSTATUS_PART_DONE: finished some\
|
|
\ step and waiting for next stepSTATUS_DONE: all work done"
|
|
function: 'manage.TaskManager:'
|
|
- docstring: " \nList the all collection(task_pool) of the db.Returns:list"
|
|
function: manage.list
|
|
- docstring: null
|
|
function: manage._encode_task
|
|
- docstring: " \n_decode_task is Serialization tool.Mongodb needs JSON, so\
|
|
\ it needs to convert Python objects into JSON objects through pickleParameters----------task\
|
|
\ : dicttask informationReturns-------dictJSON required by mongodb"
|
|
function: manage._decode_task
|
|
- docstring: null
|
|
function: manage._dict_to_str
|
|
- docstring: " \nIf the query includes any `_id`, then it needs `ObjectId`\
|
|
\ to decode.For example, when using TrainerRM, it needs query `{\"_id\": {\"$in\"\
|
|
: _id_list}}`. Then we need to `ObjectId` every `_id` in `_id_list`.Args:query\
|
|
\ (dict): query dict. Defaults to {}.Returns:dict: the query after decoding."
|
|
function: manage._decode_query
|
|
- docstring: " \nUse a new task to replace a old oneArgs:task: old tasknew_task:\
|
|
\ new task"
|
|
function: manage.replace_task
|
|
- docstring: " \nInsert a task.Args:task: the task waiting for insertReturns:pymongo.results.InsertOneResult"
|
|
function: manage.insert_task
|
|
- docstring: " \nInsert a task to task_poolParameters----------task_def: dictthe\
|
|
\ task definitionReturns-------pymongo.results.InsertOneResult"
|
|
function: manage.insert_task_def
|
|
- docstring: " \nIf the tasks in task_def_l are new, then insert new tasks\
|
|
\ into the task_pool, and record inserted_id.If a task is not new, then just query\
|
|
\ its _id.Parameters----------task_def_l: lista list of taskdry_run: boolif insert\
|
|
\ those new tasks to task poolprint_nt: boolif print new taskReturns-------List[str]a\
|
|
\ list of the _id of task_def_l"
|
|
function: manage.create_task
|
|
- docstring: " \nUse query to fetch tasks.Args:query (dict, optional): query\
|
|
\ dict. Defaults to {}.status (str, optional): [description]. Defaults to STATUS_WAITING.Returns:dict:\
|
|
\ a task(document in collection) after decoding"
|
|
function: manage.fetch_task
|
|
- docstring: " \nFetch task from task_pool using query with contextmanagerParameters----------query:\
|
|
\ dictthe dict of queryReturns-------dict: a task(document in collection) after\
|
|
\ decoding"
|
|
function: manage.safe_fetch_task
|
|
- docstring: null
|
|
function: manage.task_fetcher_iter
|
|
- docstring: " \nQuery task in collection.This function may raise exception\
|
|
\ `pymongo.errors.CursorNotFound: cursor id not found` if it takes too long to\
|
|
\ iterate the generatorpython -m qlib.workflow.task.manage -t <your task pool>\
|
|
\ query '{\"_id\": \"615498be837d0053acbc5d58\"}'Parameters----------query: dictthe\
|
|
\ dict of querydecode: boolReturns-------dict: a task(document in collection)\
|
|
\ after decoding"
|
|
function: manage.query
|
|
- docstring: " \nUse _id to query task.Args:_id (str): _id of a documentReturns:dict:\
|
|
\ a task(document in collection) after decoding"
|
|
function: manage.re_query
|
|
- docstring: " \nCommit the result to task['res'].Args:task ([type]): [description]res\
|
|
\ (object): the result you want to savestatus (str, optional): STATUS_WAITING,\
|
|
\ STATUS_RUNNING, STATUS_DONE, STATUS_PART_DONE. Defaults to STATUS_DONE."
|
|
function: manage.commit_task_res
|
|
- docstring: " \nReturn a task to status. Always using in error handling.Args:task\
|
|
\ ([type]): [description]status (str, optional): STATUS_WAITING, STATUS_RUNNING,\
|
|
\ STATUS_DONE, STATUS_PART_DONE. Defaults to STATUS_WAITING."
|
|
function: manage.return_task
|
|
- docstring: " \nRemove the task using queryParameters----------query: dictthe\
|
|
\ dict of query"
|
|
function: manage.remove
|
|
- docstring: " \nCount the tasks in every status.Args:query (dict, optional):\
|
|
\ the query dict. Defaults to {}.Returns:dict"
|
|
function: manage.task_stat
|
|
- docstring: " \nReset all running task into waiting status. Can be used when\
|
|
\ some running task exit unexpected.Args:query (dict, optional): the query dict.\
|
|
\ Defaults to {}."
|
|
function: manage.reset_waiting
|
|
- docstring: null
|
|
function: manage.reset_status
|
|
- docstring: " \nSet priority for taskParameters----------task : dictThe task\
|
|
\ query from the databasepriority : intthe target priority"
|
|
function: manage.prioritize
|
|
- docstring: null
|
|
function: manage._get_undone_n
|
|
- docstring: null
|
|
function: manage._get_total
|
|
- docstring: " \nWhen multiprocessing, the main progress may fetch nothing\
|
|
\ from TaskManager because there are still some running tasks.So main progress\
|
|
\ should wait until all tasks are trained well by other progress or machines.Args:query\
|
|
\ (dict, optional): the query dict. Defaults to {}."
|
|
function: manage.wait
|
|
- docstring: null
|
|
function: manage.run_task
|
|
- docstring: " \nUse a list of TaskGen and a list of task templates to generate\
|
|
\ different tasks.For examples:There are 3 task templates a,b,c and 2 TaskGen\
|
|
\ A,B. A will generates 2 tasks from a template and B will generates 3 tasks from\
|
|
\ a template.task_generator([a, b, c], [A, B]) will finally generate 3*2*3 = 18\
|
|
\ tasks.Parameters----------tasks : List[dict] or dicta list of task templates\
|
|
\ or a single taskgenerators : List[TaskGen] or TaskGena list of TaskGen or a\
|
|
\ single TaskGenReturns-------lista list of tasks"
|
|
function: gen.task_generator
|
|
- docstring: " \nThe base class for generating different tasksExample 1:input:\
|
|
\ a specific task template and rolling stepsoutput: rolling version of the tasksExample\
|
|
\ 2:input: a specific task template and losses listoutput: a set of tasks with\
|
|
\ different losses"
|
|
function: gen.TaskGen
|
|
- docstring: " \nGenerate different tasks based on a task templateParameters----------task:\
|
|
\ dicta task templateReturns-------typing.List[dict]:A list of tasks"
|
|
function: gen.generate
|
|
- docstring: " \nHelp to modify the handler end time when using RollingGenIt try\
|
|
\ to handle the following case- Hander's data end_time is earlier than dataset's\
|
|
\ test_data's segments.- To handle this, handler's data's end_time is extended.If\
|
|
\ the handler's end_time is None, then it is not necessary to change it's end\
|
|
\ time.Args:task (dict): a task templaterg (RollingGen): an instance of RollingGen"
|
|
function: gen.handler_mod
|
|
- docstring: " \nTo avoid the leakage of future information, the segments should\
|
|
\ be truncated according to the test start_timeNOTE:This function will change\
|
|
\ segments **inplace**"
|
|
function: gen.trunc_segments
|
|
- docstring: " \nGenerate tasks for rollingParameters----------step : intstep\
|
|
\ to rollingrtype : strrolling type (expanding, sliding)ds_extra_mod_func: CallableA\
|
|
\ method like: handler_mod(task: dict, rg: RollingGen)Do some extra action after\
|
|
\ generating a task. For example, use ``handler_mod`` to modify the end time of\
|
|
\ the handler of a dataset.trunc_days: inttrunc some data to avoid future information\
|
|
\ leakagetask_copy_func: Callablethe function to copy entire task. This is very\
|
|
\ useful when user want to share something between tasks"
|
|
function: gen.RollingGen
|
|
- docstring: null
|
|
function: gen._update_task_segs
|
|
- docstring: " \ngenerating following rolling tasks for `task` until test_endParameters----------task\
|
|
\ : dictQlib task formattest_end : pd.Timestampthe latest rolling task includes\
|
|
\ `test_end`Returns-------List[dict]:the following tasks of `task`(`task` itself\
|
|
\ is excluded)"
|
|
function: gen.gen_following_tasks
|
|
- docstring: " \nConverting the task into a rolling task.Parameters----------task:\
|
|
\ dictA dict describing a task. For example... code-block:: pythonDEFAULT_TASK\
|
|
\ = {\"model\": {\"class\": \"LGBModel\",\"module_path\": \"qlib.contrib.model.gbdt\"\
|
|
,},\"dataset\": {\"class\": \"DatasetH\",\"module_path\": \"qlib.data.dataset\"\
|
|
,\"kwargs\": {\"handler\": {\"class\": \"Alpha158\",\"module_path\": \"qlib.contrib.data.handler\"\
|
|
,\"kwargs\": {\"start_time\": \"2008-01-01\",\"end_time\": \"2020-08-01\",\"fit_start_time\"\
|
|
: \"2008-01-01\",\"fit_end_time\": \"2014-12-31\",\"instruments\": \"csi100\"\
|
|
,},},\"segments\": {\"train\": (\"2008-01-01\", \"2014-12-31\"),\"valid\": (\"\
|
|
2015-01-01\", \"2016-12-20\"), # Please avoid leaking the future test data into\
|
|
\ validation\"test\": (\"2017-01-01\", \"2020-08-01\"),},},},\"record\": [{\"\
|
|
class\": \"SignalRecord\",\"module_path\": \"qlib.workflow.record_temp\",},]}Returns----------List[dict]:\
|
|
\ a list of tasks"
|
|
function: gen.generate
|
|
- docstring: " \nThis task generator tries to generate tasks for different\
|
|
\ horizons based on an existing taskParameters----------horizon : List[int]the\
|
|
\ possible horizons of the taskslabel_leak_n : intHow many future days it will\
|
|
\ take to get complete label after the day making predictionFor example:- User\
|
|
\ make prediction on day `T`(after getting the close price on `T`)- The label\
|
|
\ is the return of buying stock on `T + 1` and selling it on `T + 2`- the `label_leak_n`\
|
|
\ will be 2 (e.g. two days of information is leaked to leverage this sample)"
|
|
function: gen.MultiHorizonGenBase
|
|
- docstring: " \nThis method is designed to change the task **in place**Parameters----------task\
|
|
\ : dictQlib's taskhr : intthe horizon of task"
|
|
function: gen.set_horizon
|
|
- docstring: ' get_benchmark_weight
|
|
|
|
get the stock weight distribution of the benchmark:param bench::param start_date::param
|
|
end_date::param path::param freq::return: The weight distribution of the the benchmark
|
|
described by a pandas dataframeEvery row corresponds to a trading day.Every column
|
|
corresponds to a stock.Every cell represents the strategy.'
|
|
function: profit_attribution.get_benchmark_weight
|
|
- docstring: ' get_stock_weight_df
|
|
|
|
:param positions: Given a positions from backtest result.:return: A weight
|
|
distribution for the position'
|
|
function: profit_attribution.get_stock_weight_df
|
|
- docstring: ' decompose_portofolio_weight
|
|
|
|
'''''':param stock_weight_df: a pandas dataframe to describe the portofolio by
|
|
weight.every row corresponds to a dayevery column corresponds to a stock.Here
|
|
is an example below.code SH600004 SH600006 SH600017 SH600022 SH600026 SH600037 \date2016-01-05 0.001543 0.001570 0.002732 0.001320 0.003000 NaN2016-01-06 0.001538 0.001569 0.002770 0.001417 0.002945 NaN....:param
|
|
stock_group_df: a pandas dataframe to describe the stock group.every row corresponds
|
|
to a dayevery column corresponds to a stock.the value in the cell repreponds
|
|
the group id.Here is a example by for stock_group_df for industry. The value is
|
|
the industry codeinstrument SH600000 SH600004 SH600005 SH600006 SH600007 SH600008 \datetime2016-01-05 801780.0 801170.0 801040.0 801880.0 801180.0 801160.02016-01-06 801780.0 801170.0 801040.0 801880.0 801180.0 801160.0...:return: Two
|
|
dict will be returned. The group_weight and the stock_weight_in_group.The key
|
|
is the group. The value is a Series or Dataframe to describe the weight of group
|
|
or weight of stock'
|
|
function: profit_attribution.decompose_portofolio_weight
|
|
- docstring: " \n:param stock_weight_df: a pandas dataframe to describe the portofolio\
|
|
\ by weight.every row corresponds to a dayevery column corresponds to a stock.Here\
|
|
\ is an example below.code SH600004 SH600006 SH600017 SH600022 SH600026\
|
|
\ SH600037 \\date2016-01-05 0.001543 0.001570 0.002732 0.001320 0.003000\
|
|
\ NaN2016-01-06 0.001538 0.001569 0.002770 0.001417 0.002945 \
|
|
\ NaN2016-01-07 0.001555 0.001546 0.002772 0.001393 0.002904 NaN2016-01-08\
|
|
\ 0.001564 0.001527 0.002791 0.001506 0.002948 NaN2016-01-11 0.001597\
|
|
\ 0.001476 0.002738 0.001493 0.003043 NaN....:param stock_group_df:\
|
|
\ a pandas dataframe to describe the stock group.every row corresponds to a \
|
|
\ dayevery column corresponds to a stock.the value in the cell repreponds the\
|
|
\ group id.Here is a example by for stock_group_df for industry. The value is\
|
|
\ the industry codeinstrument SH600000 SH600004 SH600005 SH600006 SH600007\
|
|
\ SH600008 \\datetime2016-01-05 801780.0 801170.0 801040.0 801880.0 801180.0\
|
|
\ 801160.02016-01-06 801780.0 801170.0 801040.0 801880.0 801180.0 801160.02016-01-07\
|
|
\ 801780.0 801170.0 801040.0 801880.0 801180.0 801160.02016-01-08 801780.0\
|
|
\ 801170.0 801040.0 801880.0 801180.0 801160.02016-01-11 801780.0 801170.0\
|
|
\ 801040.0 801880.0 801180.0 801160.0...:param stock_ret_df: a pandas dataframe\
|
|
\ to describe the stock return.every row corresponds to a dayevery column corresponds\
|
|
\ to a stock.the value in the cell repreponds the return of the group.Here is\
|
|
\ a example by for stock_ret_df.instrument SH600000 SH600004 SH600005 SH600006\
|
|
\ SH600007 SH600008 \\datetime2016-01-05 0.007795 0.022070 0.099099 0.024707\
|
|
\ 0.009473 0.0162162016-01-06 -0.032597 -0.075205 -0.098361 -0.098985 -0.099707\
|
|
\ -0.0989362016-01-07 -0.001142 0.022544 0.100000 0.004225 0.000651 0.0472262016-01-08\
|
|
\ -0.025157 -0.047244 -0.038567 -0.098177 -0.099609 -0.0744082016-01-11 0.023460\
|
|
\ 0.004959 -0.034384 0.018663 0.014461 0.010962...:return: It will decompose\
|
|
\ the portofolio to the group weight and group return."
|
|
function: profit_attribution.decompose_portofolio
|
|
- docstring: ' get_daily_bin_group
|
|
|
|
Group the values of the stocks of benchmark into several bins in a day.Put the
|
|
stocks into these bins.:param bench_values: A series contains the value of stocks
|
|
in benchmark.The index is the stock code.:param stock_values: A series contains
|
|
the value of stocks of your portofolioThe index is the stock code.:param group_n: Bins
|
|
will be produced:return: A series with the same size and index as
|
|
the stock_value.The value in the series is the group id of the bins.The No.1 bin
|
|
contains the biggest values.'
|
|
function: profit_attribution.get_daily_bin_group
|
|
- docstring: null
|
|
function: profit_attribution.get_stock_group
|
|
- docstring: ' brinson profit attribution
|
|
|
|
:param positions: The position produced by the backtest class:param bench: The
|
|
benchmark for comparing. TODO: if no benchmark is set, the equal-weighted is used.:param
|
|
group_field: The field used to set the group for assets allocation.`industry`
|
|
and `market_value` is often used.:param group_method: ''category'' or ''bins''.
|
|
The method used to set the group for asstes allocation`bin` will split the value
|
|
into `group_n` bins and each bins represents a group:param group_n: . Only used
|
|
when group_method == ''bins''.:return:A dataframe with three columns: RAA(excess
|
|
Return of Assets Allocation), RSS(excess Return of Stock Selectino), RTotal(Total
|
|
excess Return)Every row corresponds to a trading day, the value corresponds to
|
|
the next return for this trading dayThe middle info of brinson profit attribution'
|
|
function: profit_attribution.brinson_pa
|
|
- docstring: null
|
|
function: decision.OrderDir
|
|
- docstring: " \nstock_id : stramount : floatstart_time : pd.Timestampclosed start\
|
|
\ time for order tradingend_time : pd.Timestampclosed end time for order tradingdirection\
|
|
\ : intOrder.SELL for sell; Order.BUY for buyfactor : floatpresents the weight\
|
|
\ factor assigned in Exchange()"
|
|
function: 'decision.Order:'
|
|
- docstring: " \nreturn the delta of amount.- Positive value indicates buying\
|
|
\ `amount` of share- Negative value indicates selling `amount` of share"
|
|
function: decision.amount_delta
|
|
- docstring: " \nreturn the delta of deal_amount.- Positive value indicates\
|
|
\ buying `deal_amount` of share- Negative value indicates selling `deal_amount`\
|
|
\ of share"
|
|
function: decision.deal_amount_delta
|
|
- docstring: " \nreturn the sign of trading- `+1` indicates buying- `-1` value\
|
|
\ indicates selling"
|
|
function: decision.sign
|
|
- docstring: null
|
|
function: decision.parse_dir
|
|
- docstring: ' A hashable & unique key to identify this order, under the granularity
|
|
in day.
|
|
|
|
return self.stock_id, self.date, self.direction@property'
|
|
function: decision.key_by_day
|
|
- docstring: ' A hashable & unique key to identify this order.
|
|
|
|
return self.stock_id, self.start_time, self.end_time, self.direction@property'
|
|
function: decision.key
|
|
- docstring: ' Date of the order.
|
|
|
|
return pd.Timestamp(self.start_time.replace(hour=0, minute=0, second=0))'
|
|
function: decision.date
|
|
- docstring: " \nMotivation- Make generating order easier- User may have no knowledge\
|
|
\ about the adjust-factor information about the system.- It involves too much\
|
|
\ interaction with the exchange when generating orders."
|
|
function: 'decision.OrderHelper:'
|
|
- docstring: " \nhelp to create a order# TODO: create order for unadjusted\
|
|
\ amount orderParameters----------code : strthe id of the instrumentamount : float**adjusted\
|
|
\ trading amount**direction : OrderDirtrading directionstart_time : Union[str,\
|
|
\ pd.Timestamp] (optional)The interval of the order which belongs toend_time :\
|
|
\ Union[str, pd.Timestamp] (optional)The interval of the order which belongs toReturns-------Order:The\
|
|
\ created order"
|
|
function: decision.create
|
|
- docstring: " \nThis method will be call with following wayThe outer strategy\
|
|
\ give a decision with with `TradeRange`The decision will be checked by the inner\
|
|
\ decision.inner decision will pass its trade_calendar as parameter when getting\
|
|
\ the trading range- The framework's step is integer-index based.Parameters----------trade_calendar\
|
|
\ : TradeCalendarManagerthe trade_calendar is from inner strategyReturns-------Tuple[int,\
|
|
\ int]:the start index and end index which are tradableRaises------NotImplementedError:Exceptions\
|
|
\ are raised when no range limitation"
|
|
function: 'decision.TradeRange:'
|
|
- docstring: " \nParameters----------start_time : pd.Timestampend_time : pd.TimestampBoth\
|
|
\ sides (start_time, end_time) are closedReturns-------Tuple[pd.Timestamp, pd.Timestamp]:The\
|
|
\ tradable time range.- It is intersection of [start_time, end_time] and the rule\
|
|
\ of TradeRange itself"
|
|
function: decision.clip_time_range
|
|
- docstring: null
|
|
function: decision.IdxTradeRange
|
|
- docstring: null
|
|
function: decision.clip_time_range
|
|
- docstring: ' This is a helper function for make decisions
|
|
|
|
'
|
|
function: decision.TradeRangeByTime
|
|
- docstring: null
|
|
function: decision.clip_time_range
|
|
- docstring: " \nTrade decisions are made by strategy and executed by executorMotivation:Here\
|
|
\ are several typical scenarios for `BaseTradeDecision`Case 1:1. Outer strategy\
|
|
\ makes a decision. The decision is not available at the start of current interval2.\
|
|
\ After a period of time, the decision are updated and become available3. The\
|
|
\ inner strategy try to get the decision and start to execute the decision according\
|
|
\ to `get_range_limit`Case 2:1. The outer strategy's decision is available at\
|
|
\ the start of the interval2. Same as `case 1.3`"
|
|
function: decision.BaseTradeDecision
|
|
- docstring: " \nget the **concrete decision** (e.g. execution orders)This\
|
|
\ will be called by the inner strategyReturns-------List[DecisionType:The decision\
|
|
\ result. Typically it is some ordersExample:[]:Decision not available[concrete_decision]:available"
|
|
function: decision.get_decision
|
|
- docstring: " \nBe called at the **start** of each step.This function is design\
|
|
\ for following purpose1) Leave a hook for the strategy who make `self` decision\
|
|
\ to update the decision itself2) Update some information from the inner executor\
|
|
\ calendarParameters----------trade_calendar : TradeCalendarManagerThe calendar\
|
|
\ of the **inner strategy**!!!!!Returns-------BaseTradeDecision:New update, use\
|
|
\ new decision. If no updates, return None (use previous decision (or unavailable))"
|
|
function: decision.update
|
|
- docstring: null
|
|
function: decision._get_range_limit
|
|
- docstring: " \nreturn the expected step range for limiting the decision execution\
|
|
\ timeBoth left and right are **closed**if no available trade_range, `default_value`\
|
|
\ will be returnedIt is only used in `NestedExecutor`- The outmost strategy will\
|
|
\ not follow any range limit (but it may give range_limit)- The inner most strategy's\
|
|
\ range_limit will be useless due to atomic executors don't have suchfeatures.**NOTE**:1)\
|
|
\ This function must be called after `self.update` in following cases(ensured\
|
|
\ by NestedExecutor):- user relies on the auto-clip feature of `self.update`2)\
|
|
\ This function will be called after _init_sub_trading in NestedExecutor.Parameters----------**kwargs:{\"\
|
|
default_value\": <default_value>, # using dict is for distinguish no value provided\
|
|
\ or None provided\"inner_calendar\": <trade calendar of inner strategy># because\
|
|
\ the range limit will control the step range of inner strategy, inner calendar\
|
|
\ will be a# important parameter when trade_range is callable}Returns-------Tuple[int,\
|
|
\ int]:Raises------NotImplementedError:If the following criteria meet1) the decision\
|
|
\ can't provide a unified start and end2) default_value is not provided"
|
|
function: decision.get_range_limit
|
|
- docstring: " \nget the range limit based on data calendarNOTE: it is **total**\
|
|
\ range limit instead of a single stepThe following assumptions are made1) The\
|
|
\ frequency of the exchange in common_infra is the same as the data calendar2)\
|
|
\ Users want the index mod by **day** (i.e. 240 min)Parameters----------rtype:\
|
|
\ str- \"full\": return the full limitation of the decision in the day- \"step\"\
|
|
: return the limitation of current stepraise_error: boolTrue: raise error if no\
|
|
\ trade_range is setFalse: return full trade calendar.It is useful in following\
|
|
\ cases- users want to follow the order specific trading time range when decision\
|
|
\ level trade range is notavailable. Raising NotImplementedError to indicates\
|
|
\ that range limit is not availableReturns-------Tuple[int, int]:the range limit\
|
|
\ in data calendarRaises------NotImplementedError:If the following criteria meet1)\
|
|
\ the decision can't provide a unified start and end2) raise_error is True"
|
|
function: decision.get_data_cal_range_limit
|
|
- docstring: null
|
|
function: decision.empty
|
|
- docstring: " \nThis method will be called on the inner_trade_decision after\
|
|
\ it is generated.`inner_trade_decision` will be changed **inplace**.Motivation\
|
|
\ of the `mod_inner_decision`- Leave a hook for outer decision to affect the decision\
|
|
\ generated by the inner strategy- e.g. the outmost strategy generate a time range\
|
|
\ for trading. But the upper layer can only affect thenearest layer in the original\
|
|
\ design. With `mod_inner_decision`, the decision can passed through multiplelayersParameters----------inner_trade_decision\
|
|
\ : BaseTradeDecision"
|
|
function: decision.mod_inner_decision
|
|
- docstring: null
|
|
function: decision.EmptyTradeDecision
|
|
- docstring: null
|
|
function: decision.get_decision
|
|
- docstring: null
|
|
function: decision.empty
|
|
- docstring: " \nTrade Decision (W)ith (O)rder.Besides, the time_range is also\
|
|
\ included."
|
|
function: decision.TradeDecisionWO
|
|
- docstring: null
|
|
function: decision.get_decision
|
|
- docstring: " \nDecision with detail information.Detail information is used to\
|
|
\ generate execution reports."
|
|
function: decision.TradeDecisionWithDetails
|
|
- docstring: " \nSome trading strategy make decisions based on other prediction\
|
|
\ signalsThe signals may comes from different sources(e.g. prepared data, online\
|
|
\ prediction from model and dataset)This interface is tries to provide unified\
|
|
\ interface for those different sources"
|
|
function: signal.Signal
|
|
- docstring: " \nget the signal at the end of the decision step(from `start_time`\
|
|
\ to `end_time`)Returns-------Union[pd.Series, pd.DataFrame, None]:returns None\
|
|
\ if no signal in the specific day"
|
|
function: signal.get_signal
|
|
- docstring: " \nSignal With pandas with based CacheSignalWCache will store the\
|
|
\ prepared signal as a attribute and give the according signal based on input\
|
|
\ query"
|
|
function: signal.SignalWCache
|
|
- docstring: null
|
|
function: signal.get_signal
|
|
- docstring: null
|
|
function: signal.ModelSignal
|
|
- docstring: " \nWhen using online data, update model in each bar as the following\
|
|
\ steps:- update dataset with online data, the dataset should support online update-\
|
|
\ make the latest prediction scores of the new bar- update the pred score into\
|
|
\ the latest prediction"
|
|
function: signal._update_model
|
|
- docstring: " \ncreate signal from diverse informationThis method will choose\
|
|
\ the right method to create a signal based on `obj`Please refer to the code below."
|
|
function: signal.create_signal_from
|
|
- docstring: ' Base executor for trading
|
|
|
|
self,time_per_step: str,start_time: Union[str, pd.Timestamp] = None,end_time:
|
|
Union[str, pd.Timestamp] = None,indicator_config: dict = {},generate_portfolio_metrics:
|
|
bool = False,verbose: bool = False,track_data: bool = False,trade_exchange: Exchange
|
|
| None = None,common_infra: CommonInfrastructure | None = None,settle_type: str
|
|
= BasePosition.ST_NO,**kwargs: Any,) -> None:'
|
|
function: 'executor.BaseExecutor:'
|
|
- docstring: " \nreset infrastructure for trading- reset trade_account"
|
|
function: executor.reset_common_infra
|
|
- docstring: ' get trade exchange in a prioritized order
|
|
|
|
return getattr(self, "_trade_exchange", None) or self.common_infra.get("trade_exchange")@property'
|
|
function: executor.trade_exchange
|
|
- docstring: " \nThough trade calendar can be accessed from multiple sources,\
|
|
\ but managing in a centralized way will make thecode easier"
|
|
function: executor.trade_calendar
|
|
- docstring: " \n- reset `start_time` and `end_time`, used in trade calendar-\
|
|
\ reset `common_infra`, used to reset `trade_account`, `trade_exchange`, .etc"
|
|
function: executor.reset
|
|
- docstring: null
|
|
function: executor.get_level_infra
|
|
- docstring: null
|
|
function: executor.finished
|
|
- docstring: ' execute the trade decision and return the executed result
|
|
|
|
NOTE: this function is never used directly in the framework. Should we delete
|
|
it?Parameters----------trade_decision : BaseTradeDecisionlevel : intthe level
|
|
of current executorReturns----------execute_result : List[object]the executed
|
|
result for trade decision'
|
|
function: executor.execute
|
|
- docstring: " \nPlease refer to the doc of collect_dataThe only difference\
|
|
\ between `_collect_data` and `collect_data` is that some common steps are moved\
|
|
\ intocollect_dataParameters----------Please refer to the doc of collect_dataReturns-------Tuple[List[object],\
|
|
\ dict]:(<the executed result for trade decision>, <the extra kwargs for `self.trade_account.update_bar_end`>)"
|
|
function: executor._collect_data
|
|
- docstring: ' Generator for collecting the trade decision data for rl training
|
|
|
|
his function will make a step forwardParameters----------trade_decision : BaseTradeDecisionlevel
|
|
: intthe level of current executor. 0 indicates the top levelreturn_value : dictthe
|
|
mem address to return the valuee.g. {"return_value": <the executed result>}Returns----------execute_result
|
|
: List[object]the executed result for trade decision.** NOTE!!!! **:1) This is
|
|
necessary, The return value of generator will be used in NestedExecutor2) Please
|
|
note the executed results are not merged.Yields-------objecttrade decision'
|
|
function: executor.collect_data
|
|
- docstring: ' get all executors
|
|
|
|
return [self]'
|
|
function: executor.get_all_executors
|
|
- docstring: " \nNested Executor with inner strategy and executor- At each time\
|
|
\ `execute` is called, it will call the inner strategy and executor to execute\
|
|
\ the `trade_decision`in a higher frequency env."
|
|
function: executor.NestedExecutor
|
|
- docstring: " \nreset infrastructure for trading- reset inner_strategy and\
|
|
\ inner_executor common infra"
|
|
function: executor.reset_common_infra
|
|
- docstring: null
|
|
function: executor._init_sub_trading
|
|
- docstring: null
|
|
function: executor._update_trade_decision
|
|
- docstring: null
|
|
function: executor._collect_data
|
|
- docstring: " \nA hook for doing sth after each step of inner strategyParameters----------inner_exe_res\
|
|
\ :the execution result of inner task"
|
|
function: executor.post_inner_exe_step
|
|
- docstring: ' get all executors, including self and inner_executor.get_all_executors()
|
|
|
|
return [self, *self.inner_executor.get_all_executors()]'
|
|
function: executor.get_all_executors
|
|
- docstring: " \nIDE-friendly helper function."
|
|
function: executor._retrieve_orders_from_decision
|
|
- docstring: ' Executor that simulate the true market
|
|
|
|
# TODO: TT_SERIAL & TT_PARAL will be replaced by feature fix_pos now.# Please
|
|
remove them in the future.# available trade_typesTT_SERIAL = "serial"# The orders
|
|
will be executed serially in a sequence# In each trading step, it is possible
|
|
that users sell instruments first and use the money to buy new instrumentsTT_PARAL
|
|
= "parallel"# The orders will be executed in parallel# In each trading step, if
|
|
users try to sell instruments first and buy new instruments with money, failure
|
|
will# occurself,time_per_step: str,start_time: Union[str, pd.Timestamp] = None,end_time:
|
|
Union[str, pd.Timestamp] = None,indicator_config: dict = {},generate_portfolio_metrics:
|
|
bool = False,verbose: bool = False,track_data: bool = False,common_infra: CommonInfrastructure
|
|
| None = None,trade_type: str = TT_SERIAL,**kwargs: Any,) -> None:'
|
|
function: executor.SimulatorExecutor
|
|
- docstring: " \nParameters----------trade_decision : BaseTradeDecisionthe\
|
|
\ trade decision given by the strategyReturns-------List[Order]:get a list orders\
|
|
\ according to `self.trade_type`"
|
|
function: executor._get_order_iterator
|
|
- docstring: " \nManager for trading calendar- BaseStrategy and BaseExecutor will\
|
|
\ use it"
|
|
function: 'utils.TradeCalendarManager:'
|
|
- docstring: " \nPlease refer to the docs of `__init__`Reset the trade calendar-\
|
|
\ self.trade_len : The total count for trading step- self.trade_step : The number\
|
|
\ of trading step finished, self.trade_step can be[0, 1, 2, ..., self.trade_len\
|
|
\ - 1]"
|
|
function: utils.reset
|
|
- docstring: " \nCheck if the trading finished- Should check before calling\
|
|
\ strategy.generate_decisions and executor.execute- If self.trade_step >= self.self.trade_len,\
|
|
\ it means the trading is finished- If self.trade_step < self.self.trade_len,\
|
|
\ it means the number of trading step finished is self.trade_step"
|
|
function: utils.finished
|
|
- docstring: null
|
|
function: utils.step
|
|
- docstring: null
|
|
function: utils.get_freq
|
|
- docstring: ' get the total step length
|
|
|
|
return self.trade_len'
|
|
function: utils.get_trade_len
|
|
- docstring: null
|
|
function: utils.get_trade_step
|
|
- docstring: " \nGet the left and right endpoints of the trade_step'th trading\
|
|
\ intervalAbout the endpoints:- Qlib uses the closed interval in time-series data\
|
|
\ selection, which has the same performance aspandas.Series.loc# - The returned\
|
|
\ right endpoints should minus 1 seconds because of the closed interval representation\
|
|
\ in# Qlib.# Note: Qlib supports up to minutely decision execution, so 1 seconds\
|
|
\ is less than any trading time# interval.Parameters----------trade_step : int,\
|
|
\ optionalthe number of trading step finished, by default None to indicate current\
|
|
\ stepshift : int, optionalshift bars , by default 0Returns-------Tuple[pd.Timestamp,\
|
|
\ pd.Timestamp]- If shift == 0, return the trading time range- If shift > 0, return\
|
|
\ the trading time range of the earlier shift bars- If shift < 0, return the trading\
|
|
\ time range of the later shift bar"
|
|
function: utils.get_step_time
|
|
- docstring: " \nget the calendar rangeThe following assumptions are made1)\
|
|
\ The frequency of the exchange in common_infra is the same as the data calendar2)\
|
|
\ Users want the **data index** mod by **day** (i.e. 240 min)Parameters----------rtype:\
|
|
\ str- \"full\": return the full limitation of the decision in the day- \"step\"\
|
|
: return the limitation of current stepReturns-------Tuple[int, int]:"
|
|
function: utils.get_data_cal_range
|
|
- docstring: ' Get the start_time and end_time for trading
|
|
|
|
return self.start_time, self.end_time# helper functions'
|
|
function: utils.get_all_time
|
|
- docstring: " \nget the range index which involve start_time~end_time (both\
|
|
\ sides are closed)Parameters----------start_time : pd.Timestampend_time : pd.TimestampReturns-------Tuple[int,\
|
|
\ int]:the index of the range. **the left and right are closed**"
|
|
function: utils.get_range_idx
|
|
- docstring: null
|
|
function: utils.clip
|
|
- docstring: null
|
|
function: 'utils.BaseInfrastructure:'
|
|
- docstring: null
|
|
function: utils.get_support_infra
|
|
- docstring: null
|
|
function: utils.reset_infra
|
|
- docstring: null
|
|
function: utils.get
|
|
- docstring: null
|
|
function: utils.has
|
|
- docstring: null
|
|
function: utils.update
|
|
- docstring: null
|
|
function: utils.CommonInfrastructure
|
|
- docstring: null
|
|
function: utils.get_support_infra
|
|
- docstring: ' level infrastructure is created by executor, and then shared to
|
|
strategies on the same level
|
|
|
|
'
|
|
function: utils.LevelInfrastructure
|
|
- docstring: " \nDescriptions about the infrastructuresub_level_infra:- **NOTE**:\
|
|
\ this will only work after _init_sub_trading !!!"
|
|
function: utils.get_support_infra
|
|
- docstring: ' reset trade calendar manager
|
|
|
|
if self.has("trade_calendar"):self.get("trade_calendar").reset(freq, start_time=start_time,
|
|
end_time=end_time)else:self.reset_infra(trade_calendar=TradeCalendarManager(freq,
|
|
start_time=start_time, end_time=end_time, level_infra=self),)'
|
|
function: utils.reset_cal
|
|
- docstring: ' this will make the calendar access easier when crossing multi-levels
|
|
|
|
self.reset_infra(sub_level_infra=sub_level_infra)'
|
|
function: utils.set_sub_level_infra
|
|
- docstring: " \nA helper function for getting the decision-level index range limitation\
|
|
\ for inner strategy- NOTE: this function is not applicable to order-levelParameters----------trade_calendar\
|
|
\ : TradeCalendarManagerouter_trade_decision : BaseTradeDecisionthe trade decision\
|
|
\ made by outer strategyReturns-------Union[int, int]:start index and end index"
|
|
function: utils.get_start_end_idx
|
|
- docstring: ' backtest function for the interaction of the outermost strategy
|
|
and executor in the nested decision execution
|
|
|
|
please refer to the docs of `collect_data_loop`Returns-------portfolio_dict: PORT_METRICit
|
|
records the trading portfolio_metrics informationindicator_dict: INDICATOR_METRICit
|
|
computes the trading indicator'
|
|
function: backtest.backtest_loop
|
|
- docstring: ' Generator for collecting the trade decision data for rl training
|
|
|
|
Parameters----------start_time : Union[pd.Timestamp, str]closed start time for
|
|
backtest**NOTE**: This will be applied to the outmost executor''s calendar.end_time
|
|
: Union[pd.Timestamp, str]closed end time for backtest**NOTE**: This will be applied
|
|
to the outmost executor''s calendar.E.g. Executor[day](Executor[1min]), setting
|
|
`end_time == 20XX0301` will include all the minutes on 20XX0301trade_strategy
|
|
: BaseStrategythe outermost portfolio strategytrade_executor : BaseExecutorthe
|
|
outermost executorreturn_value : dictused for backtest_loopYields-------objecttrade
|
|
decision'
|
|
function: backtest.collect_data_loop
|
|
- docstring: " \nThe Position wants to maintain the position like a dictionaryPlease\
|
|
\ refer to the `Position` class for the position"
|
|
function: 'position.BasePosition:'
|
|
- docstring: null
|
|
function: position.fill_stock_value
|
|
- docstring: " \nShould we skip updating operation for this positionFor example,\
|
|
\ updating is meaningless for InfPositionReturns-------bool:should we skip the\
|
|
\ updating operator"
|
|
function: position.skip_update
|
|
- docstring: " \ncheck if is the stock in the positionParameters----------stock_id\
|
|
\ : strthe id of the stockReturns-------bool:if is the stock in the position"
|
|
function: position.check_stock
|
|
- docstring: " \nParameters----------order : Orderthe order to update the positiontrade_val\
|
|
\ : floatthe trade value(money) of dealing resultscost : floatthe trade cost of\
|
|
\ the dealing resultstrade_price : floatthe trade price of the dealing results"
|
|
function: position.update_order
|
|
- docstring: " \nUpdating the latest price of the orderThe useful when clearing\
|
|
\ balance at each bar endParameters----------stock_id :the id of the stockprice\
|
|
\ : floatthe price to be updated"
|
|
function: position.update_stock_price
|
|
- docstring: " \ncalculate the value of the all assets except cash in the positionReturns-------float:the\
|
|
\ value(money) of all the stock"
|
|
function: position.calculate_stock_value
|
|
- docstring: null
|
|
function: position.calculate_value
|
|
- docstring: " \nGet the list of stocks in the position."
|
|
function: position.get_stock_list
|
|
- docstring: " \nget the latest price of the stockParameters----------code\
|
|
\ :the code of the stock"
|
|
function: position.get_stock_price
|
|
- docstring: " \nget the amount of the stockParameters----------code :the code\
|
|
\ of the stockReturns-------float:the amount of the stock"
|
|
function: position.get_stock_amount
|
|
- docstring: " \nParameters----------include_settle:will the unsettled(delayed)\
|
|
\ cash includedDefault: not include those unavailable cashReturns-------float:the\
|
|
\ available(tradable) cash in position"
|
|
function: position.get_cash
|
|
- docstring: " \ngenerate stock amount dict {stock_id : amount of stock}Returns-------Dict:{stock_id\
|
|
\ : amount of stock}"
|
|
function: position.get_stock_amount_dict
|
|
- docstring: " \ngenerate stock weight dict {stock_id : value weight of stock\
|
|
\ in the position}it is meaningful in the beginning or the end of each trade step-\
|
|
\ During execution of each trading step, the weight may be not consistent with\
|
|
\ the portfolio valueParameters----------only_stock : boolIf only_stock=True,\
|
|
\ the weight of each stock in total stock will be returnedIf only_stock=False,\
|
|
\ the weight of each stock in total assets(stock + cash) will be returnedReturns-------Dict:{stock_id\
|
|
\ : value weight of stock in the position}"
|
|
function: position.get_stock_weight_dict
|
|
- docstring: " \nWill be called at the end of each bar on each levelParameters----------bar\
|
|
\ :The level to be updated"
|
|
function: position.add_count_all
|
|
- docstring: " \nUpdating the position weight;# TODO: this function is a little\
|
|
\ weird. The weight data in the position is in a wrong state after dealing order#\
|
|
\ and before updating weight."
|
|
function: position.update_weight_all
|
|
- docstring: " \nsettlement startIt will act like start and commit a transactionParameters----------settle_type\
|
|
\ : strShould we make delay the settlement in each execution (each execution will\
|
|
\ make the executor a step forward)- \"cash\": make the cash settlement delayed.-\
|
|
\ The cash you get can't be used in current step (e.g. you can't sell a stock\
|
|
\ to get cash to buy anotherstock)- None: not settlement mechanism- TODO: other\
|
|
\ assets will be supported in the future."
|
|
function: position.settle_start
|
|
- docstring: " \nsettlement commit"
|
|
function: position.settle_commit
|
|
- docstring: ' Position
|
|
|
|
current state of positiona typical example is :{<instrument_id>: {''count'': <how
|
|
many days the security has been hold>,''amount'': <the amount of the security>,''price'':
|
|
<the close price of security in the last trading day>,''weight'': <the security
|
|
weight of total position value>,},}'
|
|
function: position.Position
|
|
- docstring: ' fill the stock value by the close price of latest last_days
|
|
from qlib.
|
|
|
|
Parameters----------start_time :the start time of backtest.freq : strFrequencylast_days
|
|
: int, optionalthe days to get the latest close price, by default 30.'
|
|
function: position.fill_stock_value
|
|
- docstring: " \ninitialization the stock in current positionParameters----------stock_id\
|
|
\ :the id of the stockamount : floatthe amount of the stockprice :the price when\
|
|
\ buying the init stock"
|
|
function: position._init_stock
|
|
- docstring: null
|
|
function: position._buy_stock
|
|
- docstring: null
|
|
function: position._sell_stock
|
|
- docstring: null
|
|
function: position._del_stock
|
|
- docstring: null
|
|
function: position.check_stock
|
|
- docstring: null
|
|
function: position.update_order
|
|
- docstring: null
|
|
function: position.update_stock_price
|
|
- docstring: null
|
|
function: position.update_stock_count
|
|
- docstring: null
|
|
function: position.update_stock_weight
|
|
- docstring: null
|
|
function: position.calculate_stock_value
|
|
- docstring: null
|
|
function: position.calculate_value
|
|
- docstring: null
|
|
function: position.get_stock_list
|
|
- docstring: null
|
|
function: position.get_stock_price
|
|
- docstring: null
|
|
function: position.get_stock_amount
|
|
- docstring: ' the days the account has been hold, it may be used in some special
|
|
strategies
|
|
|
|
if f"count_{bar}" in self.position[code]:return self.position[code][f"count_{bar}"]else:return
|
|
0'
|
|
function: position.get_stock_count
|
|
- docstring: null
|
|
function: position.get_stock_weight
|
|
- docstring: null
|
|
function: position.get_cash
|
|
- docstring: ' generate stock amount dict {stock_id : amount of stock}
|
|
|
|
d = {}stock_list = self.get_stock_list()for stock_code in stock_list:d[stock_code]
|
|
= self.get_stock_amount(code=stock_code)return d'
|
|
function: position.get_stock_amount_dict
|
|
- docstring: ' get_stock_weight_dict
|
|
|
|
generate stock weight dict {stock_id : value weight of stock in the position}it
|
|
is meaningful in the beginning or the end of each trade date:param only_stock:
|
|
If only_stock=True, the weight of each stock in total stock will be returnedIf
|
|
only_stock=False, the weight of each stock in total assets(stock + cash) will
|
|
be returned'
|
|
function: position.get_stock_weight_dict
|
|
- docstring: null
|
|
function: position.add_count_all
|
|
- docstring: null
|
|
function: position.update_weight_all
|
|
- docstring: null
|
|
function: position.settle_start
|
|
- docstring: null
|
|
function: position.settle_commit
|
|
- docstring: " \nPosition with infinite cash and amount.This is useful for generating\
|
|
\ random orders."
|
|
function: position.InfPosition
|
|
- docstring: ' Updating state is meaningless for InfPosition
|
|
|
|
return True'
|
|
function: position.skip_update
|
|
- docstring: null
|
|
function: position.check_stock
|
|
- docstring: null
|
|
function: position.update_order
|
|
- docstring: null
|
|
function: position.update_stock_price
|
|
- docstring: " \nReturns-------float:infinity stock value"
|
|
function: position.calculate_stock_value
|
|
- docstring: null
|
|
function: position.calculate_value
|
|
- docstring: null
|
|
function: position.get_stock_list
|
|
- docstring: ' the price of the inf position is meaningless
|
|
|
|
return np.nan'
|
|
function: position.get_stock_price
|
|
- docstring: null
|
|
function: position.get_stock_amount
|
|
- docstring: null
|
|
function: position.get_cash
|
|
- docstring: null
|
|
function: position.get_stock_amount_dict
|
|
- docstring: null
|
|
function: position.get_stock_weight_dict
|
|
- docstring: null
|
|
function: position.add_count_all
|
|
- docstring: null
|
|
function: position.update_weight_all
|
|
- docstring: null
|
|
function: position.settle_start
|
|
- docstring: " \naccumulated trading info, including accumulated return/cost/turnoverAccumulatedInfo\
|
|
\ should be shared across different levels"
|
|
function: 'account.AccumulatedInfo:'
|
|
- docstring: null
|
|
function: account.reset
|
|
- docstring: null
|
|
function: account.add_return_value
|
|
- docstring: null
|
|
function: account.add_cost
|
|
- docstring: null
|
|
function: account.add_turnover
|
|
- docstring: null
|
|
function: account.get_return
|
|
- docstring: null
|
|
function: account.get_cost
|
|
- docstring: null
|
|
function: account.get_turnover
|
|
- docstring: " \nThe correctness of the metrics of Account in nested execution\
|
|
\ depends on the shallow copy of `trade_account` inqlib/backtest/executor.py:NestedExecutorDifferent\
|
|
\ level of executor has different Account object when calculating metrics. But\
|
|
\ the position object isshared cross all the Account object."
|
|
function: 'account.Account:'
|
|
- docstring: null
|
|
function: account.init_vars
|
|
- docstring: " \nIs portfolio-based metrics enabled."
|
|
function: account.is_port_metr_enabled
|
|
- docstring: null
|
|
function: account.reset_report
|
|
- docstring: ' reset freq and report of account
|
|
|
|
Parameters----------freq : str, optionalfrequency of account & report, by default
|
|
Nonebenchmark_config : {}, optionalbenchmark config of report, by default Noneport_metr_enabled:
|
|
bool'
|
|
function: account.reset
|
|
- docstring: null
|
|
function: account.get_hist_positions
|
|
- docstring: null
|
|
function: account.get_cash
|
|
- docstring: null
|
|
function: account._update_state_from_order
|
|
- docstring: null
|
|
function: account.update_order
|
|
- docstring: " \nUpdate current to make rtn consistent with earning at the\
|
|
\ end of bar, and update holding bar count of stock"
|
|
function: account.update_current_position
|
|
- docstring: ' update portfolio_metrics
|
|
|
|
# calculate earning# account_value - last_account_value# for the first trade date,
|
|
account_value - init_cash# self.portfolio_metrics.is_empty() to judge is_first_trade_date#
|
|
get last_account_value, last_total_cost, last_total_turnoverassert self.portfolio_metrics
|
|
is not Noneif self.portfolio_metrics.is_empty():last_account_value = self.init_cashlast_total_cost
|
|
= 0last_total_turnover = 0else:last_account_value = self.portfolio_metrics.get_latest_account_value()last_total_cost
|
|
= self.portfolio_metrics.get_latest_total_cost()last_total_turnover = self.portfolio_metrics.get_latest_total_turnover()#
|
|
get now_account_value, now_stock_value, now_earning, now_cost, now_turnovernow_account_value
|
|
= self.current_position.calculate_value()now_stock_value = self.current_position.calculate_stock_value()now_earning
|
|
= now_account_value - last_account_valuenow_cost = self.accum_info.get_cost -
|
|
last_total_costnow_turnover = self.accum_info.get_turnover - last_total_turnover#
|
|
update portfolio_metrics for today# judge whether the trading is begin.# and don''t
|
|
add init account state into portfolio_metrics, due to we don''t have excess return
|
|
in those days.self.portfolio_metrics.update_portfolio_metrics_record(trade_start_time=trade_start_time,trade_end_time=trade_end_time,account_value=now_account_value,cash=self.current_position.position["cash"],return_rate=(now_earning
|
|
+ now_cost) / last_account_value,# here use earning to calculate return, position''s
|
|
view, earning consider cost, true return# in order to make same definition with
|
|
original backtest in evaluate.pytotal_turnover=self.accum_info.get_turnover,turnover_rate=now_turnover
|
|
/ last_account_value,total_cost=self.accum_info.get_cost,cost_rate=now_cost /
|
|
last_account_value,stock_value=now_stock_value,)'
|
|
function: account.update_portfolio_metrics
|
|
- docstring: ' update history position
|
|
|
|
now_account_value = self.current_position.calculate_value()# set now_account_value
|
|
to positionself.current_position.position["now_account_value"] = now_account_valueself.current_position.update_weight_all()#
|
|
update hist_positions# note use deepcopyself.hist_positions[trade_start_time]
|
|
= copy.deepcopy(self.current_position)'
|
|
function: account.update_hist_positions
|
|
- docstring: ' update trade indicators and order indicators in each bar end
|
|
|
|
# TODO: will skip empty decisions make it faster? `outer_trade_decision.empty():`#
|
|
indicator is trading (e.g. high-frequency order execution) related analysisself.indicator.reset()#
|
|
aggregate the information for each orderif atomic:self.indicator.update_order_indicators(trade_info)else:self.indicator.agg_order_indicators(inner_order_indicators,decision_list=decision_list,outer_trade_decision=outer_trade_decision,trade_exchange=trade_exchange,indicator_config=indicator_config,)#
|
|
aggregate all the order metrics a single stepself.indicator.cal_trade_indicators(trade_start_time,
|
|
self.freq, indicator_config)# record the metricsself.indicator.record(trade_start_time)'
|
|
function: account.update_indicator
|
|
- docstring: ' update account at each trading bar step
|
|
|
|
Parameters----------trade_start_time : pd.Timestampclosed start time of steptrade_end_time
|
|
: pd.Timestampclosed end time of steptrade_exchange : Exchangetrading exchange,
|
|
used to update currentatomic : boolwhether the trading executor is atomic, which
|
|
means there is no higher-frequency trading executor inside it- if atomic is True,
|
|
calculate the indicators with trade_info- else, aggregate indicators with inner
|
|
indicatorsouter_trade_decision: BaseTradeDecisionexternal trade decisiontrade_info
|
|
: List[(Order, float, float, float)], optionaltrading information, by default
|
|
None- necessary if atomic is True- list of tuple(order, trade_val, trade_cost,
|
|
trade_price)inner_order_indicators : Indicator, optionalindicators of inner executor,
|
|
by default None- necessary if atomic is False- used to aggregate outer indicatorsdecision_list:
|
|
List[Tuple[BaseTradeDecision, pd.Timestamp, pd.Timestamp]] = None,The decision
|
|
list of the inner level: List[Tuple[<decision>, <start_time>, <end_time>]]The
|
|
inner levelindicator_config : dict, optionalconfig of calculating indicators,
|
|
by default {}'
|
|
function: account.update_bar_end
|
|
- docstring: ' get the history portfolio_metrics and positions instance
|
|
|
|
if self.is_port_metr_enabled():assert self.portfolio_metrics is not None_portfolio_metrics
|
|
= self.portfolio_metrics.generate_portfolio_metrics_dataframe()_positions = self.get_hist_positions()return
|
|
_portfolio_metrics, _positionselse:raise ValueError("generate_portfolio_metrics
|
|
should be True if you want to generate portfolio_metrics")'
|
|
function: account.get_portfolio_metrics
|
|
- docstring: null
|
|
function: 'high_performance_ds.BaseQuote:'
|
|
- docstring: ' return all stock codes
|
|
|
|
Return------Iterableall stock codes'
|
|
function: high_performance_ds.get_all_stock
|
|
- docstring: ' get the specific field of stock data during start time and end_time,
|
|
|
|
and apply method to the data.Example:.. code-block::$close $volumeinstrument datetimeSH600000 2010-01-04 86.778313 16162960.02010-01-05 87.433578 28117442.02010-01-06 85.713585 23632884.02010-01-07 83.788803 20813402.02010-01-08 84.730675 16044853.0SH600655 2010-01-04 2699.567383 158193.3281252010-01-08 2612.359619 77501.4062502010-01-11 2712.982422 160852.3906252010-01-12 2788.688232 164587.9375002010-01-13 2790.604004 145460.453125this
|
|
function is used for three case:1. method is not None. It returns int/float/bool/None.-
|
|
It will return None in one case, the method return Noneprint(get_data(stock_id="SH600000",
|
|
start_time="2010-01-04", end_time="2010-01-06", field="$close", method="last"))85.7135852.
|
|
method is None. It returns IndexData.print(get_data(stock_id="SH600000", start_time="2010-01-04",
|
|
end_time="2010-01-06", field="$close", method=None))IndexData([86.778313, 87.433578,
|
|
85.713585], [2010-01-04, 2010-01-05, 2010-01-06])Parameters----------stock_id:
|
|
strstart_time : Union[pd.Timestamp, str]closed start time for backtestend_time
|
|
: Union[pd.Timestamp, str]closed end time for backtestfield : strthe columns of
|
|
data to fetchmethod : Union[str, None]the method apply to data.e.g [None, "last",
|
|
"all", "sum", "mean", "ts_data_last"]Return----------Union[None, int, float, bool,
|
|
IndexData]it will return None in following cases- There is no stock data which
|
|
meet the query criterion from data source.- The `method` returns None'
|
|
function: high_performance_ds.get_data
|
|
- docstring: null
|
|
function: high_performance_ds.PandasQuote
|
|
- docstring: null
|
|
function: high_performance_ds.get_all_stock
|
|
- docstring: null
|
|
function: high_performance_ds.get_data
|
|
- docstring: ' NumpyQuote
|
|
|
|
Parameters----------quote_df : pd.DataFramethe init dataframe from qlib.self.data
|
|
: Dict(stock_id, IndexData.DataFrame)'
|
|
function: high_performance_ds.NumpyQuote
|
|
- docstring: null
|
|
function: high_performance_ds.get_all_stock
|
|
- docstring: null
|
|
function: high_performance_ds.get_data
|
|
- docstring: ' Agg data by specific method.
|
|
|
|
# FIXME: why not call the method of data directly?if method == "sum":return np.nansum(data)elif
|
|
method == "mean":return np.nanmean(data)elif method == "last":# FIXME: I''ve never
|
|
seen that this method was called.# Please merge it with "ts_data_last"return data[-1]elif
|
|
method == "all":return data.all()elif method == "ts_data_last":valid_data = data.loc[~data.isna().data.astype(bool)]if
|
|
len(valid_data) == 0:return Noneelse:return valid_data.iloc[-1]else:raise ValueError(f"{method}
|
|
is not supported")'
|
|
function: high_performance_ds._agg_data
|
|
- docstring: " \nThe data structure of the single metric.The following methods\
|
|
\ are used for computing metrics in one indicator."
|
|
function: 'high_performance_ds.BaseSingleMetric:'
|
|
- docstring: null
|
|
function: high_performance_ds.sum
|
|
- docstring: null
|
|
function: high_performance_ds.mean
|
|
- docstring: ' Return the count of the single metric, NaN is not included.
|
|
|
|
raise NotImplementedError(f"Please implement the `count` method")'
|
|
function: high_performance_ds.count
|
|
- docstring: null
|
|
function: high_performance_ds.abs
|
|
- docstring: ' If metric is empty, return True.
|
|
|
|
raise NotImplementedError(f"Please implement the `empty` method")'
|
|
function: high_performance_ds.empty
|
|
- docstring: ' Replace np.NaN with fill_value in two metrics and add them.
|
|
|
|
raise NotImplementedError(f"Please implement the `add` method")'
|
|
function: high_performance_ds.add
|
|
- docstring: ' Replace the value of metric according to replace_dict.
|
|
|
|
raise NotImplementedError(f"Please implement the `replace` method")'
|
|
function: high_performance_ds.replace
|
|
- docstring: ' Replace the value of metric with func (metric).
|
|
|
|
Currently, the func is only qlib/backtest/order/Order.parse_dir.'
|
|
function: high_performance_ds.apply
|
|
- docstring: " \nThe data structure of order indicator.!!!NOTE: There are two ways\
|
|
\ to organize the data structure. Please choose a better way.1. One way is using\
|
|
\ BaseSingleMetric to represent each metric. For example, the datastructure of\
|
|
\ PandasOrderIndicator is Dict[str, PandasSingleMetric]. It usesPandasSingleMetric\
|
|
\ based on pd.Series to represent each metric.2. The another way doesn't use BaseSingleMetric\
|
|
\ to represent each metric. The datastructure of PandasOrderIndicator is a whole\
|
|
\ matrix. It means you are not necessaryto inherit the BaseSingleMetric."
|
|
function: 'high_performance_ds.BaseOrderIndicator:'
|
|
- docstring: ' assign one metric.
|
|
|
|
Parameters----------col : strthe metric name of one metric.metric : Union[dict,
|
|
pd.Series]one metric with stock_id index, such as deal_amount, ffr, etc.for example:SH600068 NaNSH600079 1.0SH600266 NaN...SZ300692 NaNSZ300719 NaN,'
|
|
function: high_performance_ds.assign
|
|
- docstring: ' compute new metric with existing metrics.
|
|
|
|
Parameters----------func : Callablethe func of computing new metric.the kwargs
|
|
of func will be replaced with metric data by name in this function.e.g.'
|
|
function: high_performance_ds.transfer
|
|
- docstring: " \nfunc_sig = inspect.signature(func).parameters.keys()func_kwargs\
|
|
\ = {sig: self.data[sig] for sig in func_sig}tmp_metric = func(**func_kwargs)if\
|
|
\ new_col is not None:self.data[new_col] = tmp_metricreturn Noneelse:return tmp_metric"
|
|
function: high_performance_ds.func
|
|
- docstring: ' return the single metric with pd.Series format.
|
|
|
|
Parameters----------metric : strthe metric name.Return----------pd.Seriesthe single
|
|
metric.If there is no metric name in the data, return pd.Series().'
|
|
function: high_performance_ds.get_metric_series
|
|
- docstring: ' get one metric with the format of SingleData
|
|
|
|
Parameters----------metric : strthe metric name.Return------IndexData.Seriesone
|
|
metric with the format of SingleData'
|
|
function: high_performance_ds.get_index_data
|
|
- docstring: ' sum indicators with the same metrics.
|
|
|
|
and assign to the order_indicator(BaseOrderIndicator).NOTE: indicators could be
|
|
a empty list when orders in lower level all fail.Parameters----------order_indicator
|
|
: BaseOrderIndicatorthe order indicator to assign.indicators : List[BaseOrderIndicator]the
|
|
list of all inner indicators.metrics : Union[str, List[str]]all metrics needs
|
|
to be sumed.fill_value : float, optionalfill np.NaN with value. By default None.'
|
|
function: high_performance_ds.sum_all_indicators
|
|
- docstring: ' return the metrics as pandas series
|
|
|
|
for example: { "ffr":SH600068 NaNSH600079 1.0SH600266 NaN...SZ300692 NaNSZ300719 NaN,...}'
|
|
function: high_performance_ds.to_series
|
|
- docstring: null
|
|
function: high_performance_ds.SingleMetric
|
|
- docstring: ' Each SingleMetric is based on pd.Series.
|
|
|
|
if isinstance(metric, dict):self.metric = pd.Series(metric)elif isinstance(metric,
|
|
pd.Series):self.metric = metricelse:raise ValueError(f"metric must be dict or
|
|
pd.Series")'
|
|
function: high_performance_ds.PandasSingleMetric
|
|
- docstring: null
|
|
function: high_performance_ds.sum
|
|
- docstring: null
|
|
function: high_performance_ds.mean
|
|
- docstring: null
|
|
function: high_performance_ds.count
|
|
- docstring: null
|
|
function: high_performance_ds.abs
|
|
- docstring: null
|
|
function: high_performance_ds.empty
|
|
- docstring: null
|
|
function: high_performance_ds.index
|
|
- docstring: null
|
|
function: high_performance_ds.add
|
|
- docstring: null
|
|
function: high_performance_ds.replace
|
|
- docstring: null
|
|
function: high_performance_ds.apply
|
|
- docstring: null
|
|
function: high_performance_ds.reindex
|
|
- docstring: " \nThe data structure is OrderedDict(str: PandasSingleMetric).Each\
|
|
\ PandasSingleMetric based on pd.Series is one metric.Str is the name of metric."
|
|
function: high_performance_ds.PandasOrderIndicator
|
|
- docstring: null
|
|
function: high_performance_ds.assign
|
|
- docstring: null
|
|
function: high_performance_ds.get_index_data
|
|
- docstring: null
|
|
function: high_performance_ds.get_metric_series
|
|
- docstring: null
|
|
function: high_performance_ds.to_series
|
|
- docstring: null
|
|
function: high_performance_ds.sum_all_indicators
|
|
- docstring: " \nThe data structure is OrderedDict(str: SingleData).Each idd.SingleData\
|
|
\ is one metric.Str is the name of metric."
|
|
function: high_performance_ds.NumpyOrderIndicator
|
|
- docstring: null
|
|
function: high_performance_ds.assign
|
|
- docstring: null
|
|
function: high_performance_ds.get_index_data
|
|
- docstring: null
|
|
function: high_performance_ds.get_metric_series
|
|
- docstring: null
|
|
function: high_performance_ds.to_series
|
|
- docstring: ' __init__
|
|
|
|
:param freq: frequency of data:param start_time: closed start
|
|
time for backtest:param end_time: closed end time for backtest:param codes: list
|
|
stock_id list or a string of instruments(i.e. all, csi500, sse50):param deal_price: Union[str,
|
|
Tuple[str, str], List[str]]The `deal_price` supports following two types of input-
|
|
<deal_price> : str- (<buy_price>, <sell_price>): Tuple[str] or List[str]<deal_price>,
|
|
<buy_price> or <sell_price> := <price><price> := str- for example ''$close'',
|
|
''$open'', ''$vwap'' ("close" is OK. `Exchange` will help to prepend"$" to the
|
|
expression):param subscribe_fields: list, subscribe fields. This expressions will
|
|
be added to the query and `self.quote`.It is useful when users want more fields
|
|
to be queried:param limit_threshold: Union[Tuple[str, str], float, None]1) `None`:
|
|
no limitation2) float, 0.1 for example, default None3) Tuple[str, str]: (<the
|
|
expression for buying stock limitation>,<the expression for sell stock limitation>)`False`
|
|
value indicates the stock is tradable`True` value indicates the stock is limited
|
|
and not tradable:param volume_threshold: Union[Dict["all": ("cum" or "current",
|
|
limit_str),"buy": ("cum" or "current", limit_str),"sell":("cum" or "current",
|
|
limit_str),],("cum" or "current", limit_str),]1) ("cum" or "current", limit_str)
|
|
denotes a single volume limit.- limit_str is qlib data expression which is allowed
|
|
to define your own Operator.Please refer to qlib/contrib/ops/high_freq.py, here
|
|
are any custom operator forhigh frequency, such as DayCumsum. !!!NOTE: if you
|
|
want you use the customoperator, you need to register it in qlib_init.- "cum"
|
|
means that this is a cumulative value over time, such as cumulative marketvolume.
|
|
So when it is used as a volume limit, it is necessary to subtract the dealtamount.-
|
|
"current" means that this is a real-time value and will not accumulate over time,so
|
|
it can be directly used as a capacity limit.e.g. ("cum", "0.2 * DayCumsum($volume,
|
|
''9:45'', ''14:45'')"), ("current", "$bidV1")2) "all" means the volume limits
|
|
are both buying and selling."buy" means the volume limits of buying. "sell" means
|
|
the volume limits of selling.Different volume limits will be aggregated with min().
|
|
If volume_threshold is only("cum" or "current", limit_str) instead of a dict,
|
|
the volume limits are forboth by default. In other words, it is same as {"all":
|
|
("cum" or "current", limit_str)}.3) e.g. "volume_threshold": {"all": ("cum", "0.2
|
|
* DayCumsum($volume, ''9:45'', ''14:45'')"),"buy": ("current", "$askV1"),"sell":
|
|
("current", "$bidV1"),}:param open_cost: cost rate for open, default 0.0015:param
|
|
close_cost: cost rate for close, default 0.0025:param trade_unit: trade
|
|
unit, 100 for China A market.None for disable trade unit.**NOTE**: `trade_unit`
|
|
is included in the `kwargs`. It is necessary because we mustdistinguish `not set`
|
|
and `disable trade_unit`:param min_cost: min cost, default 5:param impact_cost: market
|
|
impact cost rate (a.k.a. slippage). A recommended value is 0.1.:param extra_quote: pandas,
|
|
dataframe consists ofcolumns: like [''$vwap'', ''$close'', ''$volume'', ''$factor'',
|
|
''limit_sell'', ''limit_buy''].The limit indicates that the etf is tradable on
|
|
a specific day.Necessary fields:$close is for calculating the total value at end
|
|
of each day.Optional fields:$volume is only necessary when we limit the trade
|
|
amount or calculatePA(vwap) indicator$vwap is only necessary when we use the $vwap
|
|
price as the deal price$factor is for rounding to the trading unitlimit_sell will
|
|
be set to False by default (False indicates we can sellthis target on this day).limit_buy
|
|
will be set to False by default (False indicates we can buythis target on this
|
|
day).index: MultipleIndex(instrument, pd.Datetime)'
|
|
function: 'exchange.Exchange:'
|
|
- docstring: null
|
|
function: exchange.get_quote_from_qlib
|
|
- docstring: ' get limit type
|
|
|
|
if isinstance(limit_threshold, tuple):return self.LT_TP_EXPelif isinstance(limit_threshold,
|
|
float):return self.LT_FLTelif limit_threshold is None:return self.LT_NONEelse:raise
|
|
NotImplementedError(f"This type of `limit_threshold` is not supported")'
|
|
function: exchange._get_limit_type
|
|
- docstring: null
|
|
function: exchange._update_limit
|
|
- docstring: " \npreprocess the volume limit.get the fields need to get from\
|
|
\ qlib.get the volume limit list of buying and selling which is composed of all\
|
|
\ limits.Parameters----------volume_threshold :please refer to the doc of exchange.Returns-------fields:\
|
|
\ setthe fields need to get from qlib.buy_vol_limit: List[Tuple[str]]all volume\
|
|
\ limits of buying.sell_vol_limit: List[Tuple[str]]all volume limits of selling.Raises------ValueErrorthe\
|
|
\ format of volume_threshold is not supported."
|
|
function: exchange._get_vol_limit
|
|
- docstring: " \nParameters----------stock_id : strstart_time: pd.Timestampend_time:\
|
|
\ pd.Timestampdirection : int, optionaltrade direction, by default None- if direction\
|
|
\ is None, check if tradable for buying and selling.- if direction == Order.BUY,\
|
|
\ check the if tradable for buying- if direction == Order.SELL, check the sell\
|
|
\ limit for selling.Returns-------True: the trading of the stock is limited (maybe\
|
|
\ hit the highest/lowest price), hence the stock is not tradableFalse: the trading\
|
|
\ of the stock is not limited, hence the stock may be tradable"
|
|
function: exchange.check_stock_limit
|
|
- docstring: ' if stock is suspended(hence not tradable), True will be returned
|
|
|
|
# is suspendedif stock_id in self.quote.get_all_stock():# suspended stocks are
|
|
represented by None $close stock# The $close may contain NaN,close = self.quote.get_data(stock_id,
|
|
start_time, end_time, "$close")if close is None:# if no close record existsreturn
|
|
Trueelif isinstance(close, IndexData):# **any** non-NaN $close represents trading
|
|
opportunity may exist# if all returned is nan, then the stock is suspendedreturn
|
|
cast(bool, cast(IndexData, close).isna().all())else:# it is single value, make
|
|
sure is not Nonereturn np.isnan(close)else:# if the stock is not in the stock
|
|
list, then it is not tradable and regarded as suspendedreturn True'
|
|
function: exchange.check_stock_suspended
|
|
- docstring: null
|
|
function: exchange.is_stock_tradable
|
|
- docstring: null
|
|
function: exchange.check_order
|
|
- docstring: " \nDeal order when the actual transactionthe results section\
|
|
\ in `Order` will be changed.:param order: Deal the order.:param trade_account:\
|
|
\ Trade account to be updated after dealing the order.:param position: position\
|
|
\ to be updated after dealing the order.:param dealt_order_amount: the dealt order\
|
|
\ amount dict with the format of {stock_id: float}:return: trade_val, trade_cost,\
|
|
\ trade_price"
|
|
function: exchange.deal_order
|
|
- docstring: null
|
|
function: exchange.get_quote_info
|
|
- docstring: null
|
|
function: exchange.get_close
|
|
- docstring: ' get the total deal volume of stock with `stock_id` between the
|
|
time interval [start_time, end_time)
|
|
|
|
return self.quote.get_data(stock_id, start_time, end_time, field="$volume", method=method)'
|
|
function: exchange.get_volume
|
|
- docstring: null
|
|
function: exchange.get_deal_price
|
|
- docstring: " \nReturns-------Optional[float]:`None`: if the stock is suspended\
|
|
\ `None` may be returned`float`: return factor if the factor exists"
|
|
function: exchange.get_factor
|
|
- docstring: " \nGenerates the target position according to the weight and\
|
|
\ the cash.NOTE: All the cash will be assigned to the tradable stock.Parameter:weight_position\
|
|
\ : dict {stock_id : weight}; allocate cash by weight_positionamong then, weight\
|
|
\ must be in this range: 0 < weight < 1cash : cashstart_time : the start time\
|
|
\ point of the stepend_time : the end time point of the stepdirection : the direction\
|
|
\ of the deal price for estimating the amount# NOTE: this function is used for\
|
|
\ calculating target position. So the default direction is buy"
|
|
function: exchange.generate_amount_position_from_weight_position
|
|
- docstring: " \nCalculate the real adjust deal amount when considering the\
|
|
\ trading unit:param current_amount::param target_amount::param factor::return\
|
|
\ real_deal_amount; Positive deal_amount indicates buying more stock."
|
|
function: exchange.get_real_deal_amount
|
|
- docstring: " \nNote: some future information is used in this functionParameter:target_position\
|
|
\ : dict { stock_id : amount }current_position : dict { stock_id : amount}trade_unit\
|
|
\ : trade_unitdown sample : for amount 321 and trade_unit 100, deal_amount is\
|
|
\ 300deal order on trade_date"
|
|
function: exchange.generate_order_for_target_amount_position
|
|
- docstring: ' Parameter
|
|
|
|
position : Position()amount_dict : {stock_id : amount}direction : the direction
|
|
of the deal price for estimating the amount# NOTE:This function is used for calculating
|
|
current position value.So the default direction is sell.'
|
|
function: exchange.calculate_amount_position_value
|
|
- docstring: ' Please refer to the docs of get_amount_of_trade_unit
|
|
|
|
if factor is None:if stock_id is not None and start_time is not None and end_time
|
|
is not None:factor = self.get_factor(stock_id=stock_id, start_time=start_time,
|
|
end_time=end_time)else:raise ValueError(f"`factor` and (`stock_id`, `start_time`,
|
|
`end_time`) can''t both be None")assert factor is not Nonereturn factor'
|
|
function: exchange._get_factor_or_raise_error
|
|
- docstring: " \nget the trade unit of amount based on **factor**the factor\
|
|
\ can be given directly or calculated in given time range and stock id.`factor`\
|
|
\ has higher priority than `stock_id`, `start_time` and `end_time`Parameters----------factor\
|
|
\ : floatthe adjusted factorstock_id : strthe id of the stockstart_time :the start\
|
|
\ time of trading rangeend_time :the end time of trading range"
|
|
function: exchange.get_amount_of_trade_unit
|
|
- docstring: ' Parameter
|
|
|
|
Please refer to the docs of get_amount_of_trade_unitdeal_amount : float, adjusted
|
|
amountfactor : float, adjusted factorreturn : float, real amount'
|
|
function: exchange.round_amount_by_trade_unit
|
|
- docstring: ' parse the capacity limit string and return the actual amount
|
|
of orders that can be executed.
|
|
|
|
NOTE:this function will change the order.deal_amount **inplace**- This will make
|
|
the order info more accurateParameters----------order : Orderthe order to be executed.dealt_order_amount
|
|
: dict:param dealt_order_amount: the dealt order amount dict with the format of
|
|
{stock_id: float}'
|
|
function: exchange._clip_amount_by_volume
|
|
- docstring: ' return the real order amount after cash limit for buying.
|
|
|
|
Parameters----------trade_price : floatcash : floatcost_ratio : floatReturn----------floatthe
|
|
real order amount after cash limit for buying.'
|
|
function: exchange._get_buy_amount_by_cash_limit
|
|
- docstring: " \nCalculation of trade info**NOTE**: Order will be changed in\
|
|
\ this function:param order::param position: Position:param dealt_order_amount:\
|
|
\ the dealt order amount dict with the format of {stock_id: float}:return: trade_price,\
|
|
\ trade_val, trade_cost"
|
|
function: exchange._calc_trade_info_by_order
|
|
- docstring: " \nMotivation:PortfolioMetrics is for supporting portfolio related\
|
|
\ metrics.Implementation:daily portfolio metrics of the accountcontain those followings:\
|
|
\ return, cost, turnover, account, cash, bench, valueFor each step(bar/day/minute),\
|
|
\ each column represents- return: the return of the portfolio generated by strategy\
|
|
\ **without transaction fee**.- cost: the transaction fee and slippage.- account:\
|
|
\ the total value of assets(cash and securities are both included) in user account\
|
|
\ based on the close price of each step.- cash: the amount of cash in user's account.-\
|
|
\ bench: the return of the benchmark- value: the total value of securities/stocks/instruments\
|
|
\ (cash is excluded).update report"
|
|
function: 'report.PortfolioMetrics:'
|
|
- docstring: null
|
|
function: report.init_vars
|
|
- docstring: null
|
|
function: report.init_bench
|
|
- docstring: null
|
|
function: report._cal_benchmark
|
|
- docstring: null
|
|
function: report._sample_benchmark
|
|
- docstring: null
|
|
function: report.cal_change
|
|
- docstring: null
|
|
function: report.is_empty
|
|
- docstring: null
|
|
function: report.get_latest_date
|
|
- docstring: null
|
|
function: report.get_latest_account_value
|
|
- docstring: null
|
|
function: report.get_latest_total_cost
|
|
- docstring: null
|
|
function: report.get_latest_total_turnover
|
|
- docstring: null
|
|
function: report.update_portfolio_metrics_record
|
|
- docstring: null
|
|
function: report.generate_portfolio_metrics_dataframe
|
|
- docstring: null
|
|
function: report.save_portfolio_metrics
|
|
- docstring: ' load pm from a file
|
|
|
|
should have format likecolumns = [''account'', ''return'', ''total_turnover'',
|
|
''turnover'', ''cost'', ''total_cost'', ''value'', ''cash'', ''bench'']:parampath:
|
|
str/ pathlib.Path()'
|
|
function: report.load_portfolio_metrics
|
|
- docstring: " \n`Indicator` is implemented in a aggregate way.All the metrics\
|
|
\ are calculated aggregately.All the metrics are calculated for a separated stock\
|
|
\ and in a specific step on a specific level.| indicator | desc. \
|
|
\ ||--------------+--------------------------------------------------------------||\
|
|
\ amount | the *target* amount given by the outer strategy \
|
|
\ || deal_amount | the real deal amount \
|
|
\ || inner_amount | the total *target* amount of inner strategy \
|
|
\ || trade_price | the average deal price \
|
|
\ || trade_value | the total trade value \
|
|
\ || trade_cost | the total trade cost (base price need drection)\
|
|
\ || trade_dir | the trading direction \
|
|
\ || ffr | full fill rate \
|
|
\ || pa | price advantage \
|
|
\ || pos | win rate \
|
|
\ || base_price | the price of baseline \
|
|
\ || base_volume | the volume of baseline (for weighted\
|
|
\ aggregating base_price) |**NOTE**:The `base_price` and `base_volume` can't be\
|
|
\ NaN when there are not trading on that step. Otherwiseaggregating get wrong\
|
|
\ results.So `base_price` will not be calculated in a aggregate way!!"
|
|
function: 'report.Indicator:'
|
|
- docstring: null
|
|
function: report.reset
|
|
- docstring: null
|
|
function: report.record
|
|
- docstring: null
|
|
function: report._update_order_trade_info
|
|
- docstring: null
|
|
function: report._update_order_fulfill_rate
|
|
- docstring: null
|
|
function: report.func
|
|
- docstring: null
|
|
function: report.update_order_indicators
|
|
- docstring: null
|
|
function: report._agg_order_trade_info
|
|
- docstring: null
|
|
function: report.trade_amount_func
|
|
- docstring: null
|
|
function: report.func
|
|
- docstring: null
|
|
function: report.func_apply
|
|
- docstring: null
|
|
function: report._update_trade_amount
|
|
- docstring: " \nGet the base volume and price informationAll the base price\
|
|
\ values are rooted from this function"
|
|
function: report._get_base_vol_pri
|
|
- docstring: " \n# NOTE:!!!!# Strong assumption!!!!!!# the correctness of the\
|
|
\ base_price relies on that the **same** exchange is usedParameters----------inner_order_indicators\
|
|
\ : List[BaseOrderIndicator]the indicators of account of inner executordecision_list:\
|
|
\ List[Tuple[BaseTradeDecision, pd.Timestamp, pd.Timestamp]],a list of decisions\
|
|
\ according to inner_order_indicatorstrade_exchange : Exchangefor retrieving trading\
|
|
\ pricepa_config : dictFor example{\"agg\": \"twap\", # \"vwap\"\"price\": \"\
|
|
$close\", # TODO: this is not supported now!!!!!# default to use deal price of\
|
|
\ the exchange}"
|
|
function: report._agg_base_price
|
|
- docstring: null
|
|
function: report._agg_order_price_advantage
|
|
- docstring: null
|
|
function: report.if_empty_func
|
|
- docstring: null
|
|
function: report.func
|
|
- docstring: null
|
|
function: report.agg_order_indicators
|
|
- docstring: null
|
|
function: report._cal_trade_fulfill_rate
|
|
- docstring: null
|
|
function: report._cal_trade_price_advantage
|
|
- docstring: null
|
|
function: report._cal_trade_positive_rate
|
|
- docstring: null
|
|
function: report.func
|
|
- docstring: null
|
|
function: report._cal_deal_amount
|
|
- docstring: null
|
|
function: report.func
|
|
- docstring: null
|
|
function: report._cal_trade_value
|
|
- docstring: null
|
|
function: report.func
|
|
- docstring: null
|
|
function: report._cal_trade_order_count
|
|
- docstring: null
|
|
function: report.func
|
|
- docstring: null
|
|
function: report.cal_trade_indicators
|
|
- docstring: null
|
|
function: report.get_order_indicator
|
|
- docstring: null
|
|
function: report.get_trade_indicator
|
|
- docstring: ' get_exchange
|
|
|
|
Parameters----------# exchange related argumentsexchange: ExchangeIt could be
|
|
None or any types that are acceptable by `init_instance_by_config`.freq: strfrequency
|
|
of data.start_time: Union[pd.Timestamp, str]closed start time for backtest.end_time:
|
|
Union[pd.Timestamp, str]closed end time for backtest.codes: Union[list, str]list
|
|
stock_id list or a string of instruments (i.e. all, csi500, sse50)subscribe_fields:
|
|
listsubscribe fields.open_cost : floatopen transaction cost. It is a ratio. The
|
|
cost is proportional to your order''s deal amount.close_cost : floatclose transaction
|
|
cost. It is a ratio. The cost is proportional to your order''s deal amount.min_cost
|
|
: floatmin transaction cost. It is an absolute amount of cost instead of a ratio
|
|
of your order''s deal amount.e.g. You must pay at least 5 yuan of commission regardless
|
|
of your order''s deal amount.deal_price: Union[str, Tuple[str, str], List[str]]The
|
|
`deal_price` supports following two types of input- <deal_price> : str- (<buy_price>,
|
|
<sell_price>): Tuple[str, str] or List[str]<deal_price>, <buy_price> or <sell_price>
|
|
:= <price><price> := str- for example ''$close'', ''$open'', ''$vwap'' ("close"
|
|
is OK. `Exchange` will help to prepend"$" to the expression)limit_threshold :
|
|
floatlimit move 0.1 (10%) for example, long and short with same limit.Returns-------:class:
|
|
Exchangean initialized Exchange object'
|
|
function: __init__.get_exchange
|
|
- docstring: " \n# TODO: is very strange pass benchmark_config in the account (maybe\
|
|
\ for report)# There should be a post-step to process the report.Parameters----------start_timestart\
|
|
\ time of the benchmarkend_timeend time of the benchmarkbenchmark : strthe benchmark\
|
|
\ for reportingaccount : Union[float,{\"cash\": float,\"stock1\": Union[int,\
|
|
\ # it is equal to {\"amount\": int}{\"amount\": int, \"price\"(optional):\
|
|
\ float},]},]information for describing how to creating the accountFor `float`:Using\
|
|
\ Account with only initial cashFor `dict`:key \"cash\" means initial cash.key\
|
|
\ \"stock1\" means the information of first stock with amount and price(optional)....pos_type:\
|
|
\ strPostion type."
|
|
function: __init__.create_account_instance
|
|
- docstring: null
|
|
function: __init__.get_strategy_executor
|
|
- docstring: ' initialize the strategy and executor, then backtest function for
|
|
the interaction of the outermost strategy and
|
|
|
|
executor in the nested decision executionParameters----------start_time : Union[pd.Timestamp,
|
|
str]closed start time for backtest**NOTE**: This will be applied to the outmost
|
|
executor''s calendar.end_time : Union[pd.Timestamp, str]closed end time for backtest**NOTE**:
|
|
This will be applied to the outmost executor''s calendar.E.g. Executor[day](Executor[1min]), setting
|
|
`end_time == 20XX0301` will include all the minutes on 20XX0301strategy : Union[str,
|
|
dict, object, Path]for initializing outermost portfolio strategy. Please refer
|
|
to the docs of init_instance_by_config for moreinformation.executor : Union[str,
|
|
dict, object, Path]for initializing the outermost executor.benchmark: strthe benchmark
|
|
for reporting.account : Union[float, int, Position]information for describing
|
|
how to create the accountFor `float` or `int`:Using Account with only initial
|
|
cashFor `Position`:Using Account with a Positionexchange_kwargs : dictthe kwargs
|
|
for initializing Exchangepos_type : strthe type of Position.Returns-------portfolio_dict:
|
|
PORT_METRICit records the trading portfolio_metrics informationindicator_dict:
|
|
INDICATOR_METRICit computes the trading indicatorIt is organized in a dict format'
|
|
function: __init__.backtest
|
|
- docstring: ' initialize the strategy and executor, then collect the trade decision
|
|
data for rl training
|
|
|
|
please refer to the docs of the backtest for the explanation of the parametersYields-------objecttrade
|
|
decision'
|
|
function: __init__.collect_data
|
|
- docstring: " \nformat the decisions collected by `qlib.backtest.collect_data`The\
|
|
\ decisions will be organized into a tree-like structure.Parameters----------decisions\
|
|
\ : List[BaseTradeDecision]decisions collected by `qlib.backtest.collect_data`Returns-------Tuple[str,\
|
|
\ List[Tuple[BaseTradeDecision, Union[Tuple, None]]]]:reformat the list of decisions\
|
|
\ into a more user-friendly format<decisions> := Tuple[<freq>, List[Tuple[<decision>,\
|
|
\ <sub decisions>]]]- <sub decisions> := `<decisions> in lower level` | None-\
|
|
\ <freq> := \"day\" | \"30min\" | \"1min\" | ...- <decision> := <instance of BaseTradeDecision>"
|
|
function: __init__.format_decisions
|
|
- docstring: " \nThis config is for fast demo purpose.Please use BaseSettings insetead\
|
|
\ in the future"
|
|
function: conf.Config
|
|
- docstring: null
|
|
function: utils.SingletonMeta
|
|
- docstring: " \nBecause we try to support defining Singleton with `class A(SingletonBaseClass)`\
|
|
\ instead of `A(metaclass=SingletonMeta)`This class becomes necessary"
|
|
function: utils.SingletonBaseClass
|
|
- docstring: null
|
|
function: utils.parse_json
|
|
- docstring: null
|
|
function: utils.similarity
|
|
- docstring: null
|
|
function: utils.random_string
|
|
- docstring: null
|
|
function: utils.directory_tree
|
|
- docstring: null
|
|
function: prompt_template.PromptTemplate
|
|
- docstring: null
|
|
function: prompt_template.get
|
|
- docstring: null
|
|
function: prompt_template.update
|
|
- docstring: " \nThe user's intention, which was initially represented by a prompt,\
|
|
\ is achieved through a sequence of tasks.This class doesn't have to be abstract,\
|
|
\ but it is abstract in the sense that it is not supposed to be instantiated directly\
|
|
\ because it doesn't have any implementation.Some thoughts:- Do we have to split\
|
|
\ create a new concept of Action besides Task?- Most actions directly modify the\
|
|
\ disk, with their interfaces taking in and outputting text. The LLM's interface\
|
|
\ similarly takes in and outputs text.- Some actions will run some commands.Maybe\
|
|
\ we can just categorizing tasks by following?- Planning task (it is at a high\
|
|
\ level and difficult to execute directly; therefore, it should be further divided):-\
|
|
\ Action Task- CMD Task: it is expected to run a cmd- Edit Task: it is supposed\
|
|
\ to edit the code base directly."
|
|
function: 'task.Task:'
|
|
- docstring: ' After the execution of the task, it is supposed to generated
|
|
some context about the execution
|
|
|
|
This function might be converted to abstract method in the future'
|
|
function: task.summarize
|
|
- docstring: ' assign the workflow context manager to the task
|
|
|
|
then all tasks can use this context manager to share the same context'
|
|
function: task.assign_context_manager
|
|
- docstring: null
|
|
function: task.save_chat_history_to_context_manager
|
|
- docstring: ' The execution results of the task
|
|
|
|
All sub classes should implement the execute method to determine the next task'
|
|
function: task.execute
|
|
- docstring: " \nThe user can interact with the task. This method only handle\
|
|
\ business in current task. It will return Truewhile continuous is True. This\
|
|
\ method will return user input if input cannot be parsed as 'yes' or 'no'.@return\
|
|
\ True, False, str"
|
|
function: task.interact
|
|
- docstring: null
|
|
function: task.system
|
|
- docstring: null
|
|
function: task.user
|
|
- docstring: ' This task is supposed to be the first task of the workflow
|
|
|
|
super().__init__()'
|
|
function: task.WorkflowTask
|
|
- docstring: ' make the choice which main workflow (RL, SL) will be used
|
|
|
|
user_intention = self._context_manager.get_context("user_intention")prompt_workflow_selection
|
|
= self.user.render(user_intention=user_intention)response = APIBackend().build_messages_and_create_chat_completion(prompt_workflow_selection,
|
|
self.system.render())self.save_chat_history_to_context_manager(prompt_workflow_selection,
|
|
response, self.system.render())workflow = response.split(":")[1].strip().lower()self.executed
|
|
= Trueself._context_manager.set_context("workflow", workflow)confirm = self.interact(f"The
|
|
workflow has been determined to be: "f"{LogColors().render(workflow, color=LogColors.YELLOW,
|
|
style=LogColors.BOLD)}\n"f"Enter ''y'' to authorise command,''s'' to run self-feedback
|
|
commands, "f"''n'' to exit program, or enter feedback for WorkflowTask: ")if confirm
|
|
is False:return []if workflow == "supervised learning":return [HighLevelPlanTask(),
|
|
SLPlanTask()]elif workflow == "reinforcement learning":return [RLPlanTask()]else:raise
|
|
ValueError(f"The workflow: {workflow} is not supported")'
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.PlanTask
|
|
- docstring: null
|
|
function: task.IdeaTask
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.HighLevelPlanTask
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.SLPlanTask
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.RLPlanTask
|
|
- docstring: " \nreturn a list of interested tasksCopy the template project\
|
|
\ maybe a part of the task"
|
|
function: task.execute
|
|
- docstring: " \nThis train task is responsible for training model configure by\
|
|
\ yaml file."
|
|
function: task.TrainTask
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.summarize
|
|
- docstring: " \nThis Recorder task is responsible for analysing data such as index\
|
|
\ and distribution."
|
|
function: task.AnalysisTask
|
|
- docstring: null
|
|
function: task.assign_context_manager
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.ActionTask
|
|
- docstring: ' Find a template path that user can start with.
|
|
|
|
super().__init__()if conf_path is None:# If no path provided, find path from the
|
|
templates.import qlibconf_path = Path(os.path.abspath(inspect.getfile(qlib))).parent.parent
|
|
/ "examples" / "benchmarks"if isinstance(conf_path, str):conf_path = Path(conf_path)self.conf_path
|
|
= conf_path'
|
|
function: task.ConfigSearchTask
|
|
- docstring: null
|
|
function: task.crawl_the_folder
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: " \nThis CMD task is responsible for ensuring compatibility across\
|
|
\ different operating systems."
|
|
function: task.CMDTask
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.summarize
|
|
- docstring: null
|
|
function: task.HyperparameterFinetuneActionTask
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.HyperparameterActionTask
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.ConfigActionTask
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.remove_default
|
|
- docstring: null
|
|
function: task.ImplementActionTask
|
|
- docstring: " \nreturn a list of interested tasksCopy the template project\
|
|
\ maybe a part of the task"
|
|
function: task.execute
|
|
- docstring: ' This yaml edit task will replace a specific component directly
|
|
|
|
'
|
|
function: task.YamlEditTask
|
|
- docstring: null
|
|
function: task.replace_key_value_recursive
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.CodeDumpTask
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: null
|
|
function: task.SummarizeTask
|
|
- docstring: null
|
|
function: task.summarize_context_system
|
|
- docstring: null
|
|
function: task.summarize_context_user
|
|
- docstring: null
|
|
function: task.summarize_metrics_system
|
|
- docstring: null
|
|
function: task.summarize_metrics_user
|
|
- docstring: null
|
|
function: task.execute
|
|
- docstring: '
|
|
|
|
KnowledgeBase().practice_knowledge.add([experiment_practice_knowledge])prompt_workflow_selection
|
|
= self.user.render(experiment_1_info = KnowledgeBase().practice_knowledge.knowledge[-2],experiment_2_info
|
|
= KnowledgeBase().practice_knowledge.knowledge[-1],figure_path=figure_path,user_intention=user_intention,target=target,diffrence=diffrence,target_metrics=target_metrics)response
|
|
= APIBackend().build_messages_and_create_chat_completion(user_prompt=prompt_workflow_selection,
|
|
system_prompt=self.system.render())self._context_manager.set_context("summary",
|
|
response)self.save_markdown(content=response, path=workspace)self.logger.info(f"Report
|
|
has saved to {self.__DEFAULT_REPORT_NAME}", title="End")return []'
|
|
function: task._get_value_from_info
|
|
- docstring: null
|
|
function: task.summarize
|
|
- docstring: " \nread specific type of files under path"
|
|
function: task.get_info_from_file
|
|
- docstring: null
|
|
function: task.get_info_from_context
|
|
- docstring: null
|
|
function: task.get_info_from_recorder
|
|
- docstring: null
|
|
function: task.get_figure_path
|
|
- docstring: ' This manage the whole task automation workflow including tasks and
|
|
actions
|
|
|
|
self.logger = FinCoLog()if workspace is None:self._workspace = Path.cwd() / "finco_workspace"else:self._workspace
|
|
= Path(workspace)self.conf = Config()self._confirm_and_rm()self.prompt_template
|
|
= PromptTemplate()self.context = WorkflowContextManager(workspace=self._workspace)self.context.set_context("workspace",
|
|
self._workspace)self.default_user_prompt = "build an A-share stock market daily
|
|
portfolio in quantitative investment and minimize the maximum drawdown while maintaining
|
|
return."'
|
|
function: 'workflow.WorkflowManager:'
|
|
- docstring: null
|
|
function: workflow._confirm_and_rm
|
|
- docstring: ' Direct call set_context method of the context manager
|
|
|
|
self.context.set_context(key, value)'
|
|
function: workflow.set_context
|
|
- docstring: null
|
|
function: workflow.get_context
|
|
- docstring: " \nThe workflow manager is supposed to generate a codebase based\
|
|
\ on the promptParameters----------prompt: strthe prompt user givesReturns-------PathThe\
|
|
\ workflow manager is expected to produce output that includes a codebase containing\
|
|
\ generated code, results, and reports in a designated location.The path is returnedThe\
|
|
\ output path should follow a specific format:- TODO: designThere is a summarized\
|
|
\ report where user can start from."
|
|
function: workflow.run
|
|
- docstring: null
|
|
function: 'workflow.LearnManager:'
|
|
- docstring: null
|
|
function: workflow.run
|
|
- docstring: null
|
|
function: workflow.learn
|
|
- docstring: " \nThis class is responsible for storage and loading of Knowledge\
|
|
\ related data."
|
|
function: 'knowledge.Storage:'
|
|
- docstring: null
|
|
function: knowledge.add
|
|
- docstring: null
|
|
function: knowledge.load
|
|
- docstring: null
|
|
function: knowledge.save
|
|
- docstring: " \nThis class is responsible for storage and loading of Knowledge\
|
|
\ related data in pickle format."
|
|
function: knowledge.PickleStorage
|
|
- docstring: ' use pickle as the default load method
|
|
|
|
path = path if isinstance(path, Path) else Path(path)with open(path, "rb") as
|
|
f:return pickle.load(f)'
|
|
function: knowledge.load
|
|
- docstring: ' use pickle as the default save method
|
|
|
|
Path.mkdir(self.path.parent, exist_ok=True)with open(self.path, "wb") as f:pickle.dump(self,
|
|
f)'
|
|
function: knowledge.save
|
|
- docstring: " \nThis class is responsible for storage and loading of Knowledge\
|
|
\ related data in yaml format."
|
|
function: knowledge.YamlStorage
|
|
- docstring: ' load data from yaml format file
|
|
|
|
try:self.documents = yaml.safe_load(self.path.open())except FileNotFoundError:logger.warning(f"YamlStorage:
|
|
file {self.path} doesn''t exist.")'
|
|
function: knowledge.load
|
|
- docstring: ' use pickle as the default save method
|
|
|
|
Path.mkdir(self.path.parent, exist_ok=True, parents=True)with open(self.path,
|
|
''w'') as f:yaml.dump(self.documents, f)'
|
|
function: knowledge.save
|
|
- docstring: " \nThis class is responsible for storage and loading of mlflow related\
|
|
\ data."
|
|
function: knowledge.ExperimentStorage
|
|
- docstring: null
|
|
function: knowledge.load
|
|
- docstring: " \nUse to handle knowledge in finCo such as experiment and outside\
|
|
\ domain information"
|
|
function: 'knowledge.Knowledge:'
|
|
- docstring: " \nreturn first storage matched given name, else return None"
|
|
function: knowledge.get_storage
|
|
- docstring: " \nsummarize storage data to knowledge, default knowledge is\
|
|
\ storage.documentsParameters----------Return------"
|
|
function: knowledge.summarize
|
|
- docstring: " \nLoad knowledge in memoryuse pickle as the default file typeParameters----------Return------"
|
|
function: knowledge.load
|
|
- docstring: " \nReturn a brief summary of knowledgeParameters----------Return------"
|
|
function: knowledge.brief
|
|
- docstring: ' save knowledge persistently
|
|
|
|
# todo: storages save index onlyPath.mkdir(self.workdir.joinpath(self.name), exist_ok=True)with
|
|
open(self.workdir.joinpath(self.name).joinpath("knowledge.pkl"), "wb") as f:pickle.dump(self,
|
|
f)'
|
|
function: knowledge.save
|
|
- docstring: " \nHandle knowledge from experiments"
|
|
function: knowledge.ExperimentKnowledge
|
|
- docstring: null
|
|
function: knowledge.brief
|
|
- docstring: " \nsome template sentence for now"
|
|
function: knowledge.PracticeKnowledge
|
|
- docstring: null
|
|
function: knowledge.add
|
|
- docstring: " \nKnowledge from articles"
|
|
function: knowledge.FinanceKnowledge
|
|
- docstring: null
|
|
function: knowledge.add
|
|
- docstring: " \nread all .txt files under directory"
|
|
function: knowledge.read_files_in_directory
|
|
- docstring: " \nConfig and associate execution result(pass or error message).\
|
|
\ We can regard the example in prompt as pass execution"
|
|
function: knowledge.ExecuteKnowledge
|
|
- docstring: null
|
|
function: knowledge.add
|
|
- docstring: " \nKnowledge from sentences, docstring, and code"
|
|
function: knowledge.InfrastructureKnowledge
|
|
- docstring: null
|
|
function: knowledge.add
|
|
- docstring: " \nget all method and docstring in .py files under directory"
|
|
function: knowledge.get_functions_and_docstrings
|
|
- docstring: " \nExtract method name and docstring using string matching method"
|
|
function: knowledge.get_functions_with_docstrings
|
|
- docstring: null
|
|
function: 'knowledge.Topic:'
|
|
- docstring: null
|
|
function: knowledge.summarize
|
|
- docstring: " \nLoad knowledge, offer brief information of knowledge and common\
|
|
\ handle interfaces"
|
|
function: knowledge.KnowledgeBase
|
|
- docstring: null
|
|
function: knowledge.load_experiment_knowledge
|
|
- docstring: null
|
|
function: knowledge.load_practice_knowledge
|
|
- docstring: null
|
|
function: knowledge.load_execute_knowledge
|
|
- docstring: null
|
|
function: knowledge.load_finance_knowledge
|
|
- docstring: null
|
|
function: knowledge.load_infrastructure_knowledge
|
|
- docstring: null
|
|
function: knowledge.get_knowledge
|
|
- docstring: " \n@param knowledge_type: self.KT_EXECUTE, self.KT_PRACTICE or\
|
|
\ self.KT_FINANCE@param content: content to query KnowledgeBase@param n: top n\
|
|
\ knowledge to ask ChatGPT@return:"
|
|
function: knowledge.query
|
|
- docstring: " \nThis is a conversation manager of LLMIt is for convenience of\
|
|
\ exporting conversation for debugging."
|
|
function: 'llm.ConvManager:'
|
|
- docstring: null
|
|
function: llm._rotate_files
|
|
- docstring: null
|
|
function: llm.append
|
|
- docstring: null
|
|
function: llm.APIBackend
|
|
- docstring: ' build the messages to avoid implementing several redundant lines
|
|
of code
|
|
|
|
cfg = Config()# TODO: system prompt should always be provided. In development
|
|
stage we can use default valueif system_prompt is None:try:system_prompt = cfg.system_promptexcept
|
|
AttributeError:FinCoLog().warning("system_prompt is not set, using default value.")system_prompt
|
|
= "You are an AI assistant who helps to answer user''s questions about finance."messages
|
|
= [{"role": "system","content": system_prompt,}]messages.extend(former_messages[-1
|
|
* cfg.max_past_message_include :])messages.append({"role": "user","content": user_prompt,})fcl
|
|
= FinCoLog()response = self.try_create_chat_completion(messages=messages, **kwargs)fcl.log_message(messages)fcl.log_response(response)if
|
|
self.debug_mode:ConvManager().append((messages, response))return response'
|
|
function: llm.build_messages_and_create_chat_completion
|
|
- docstring: null
|
|
function: llm.try_create_chat_completion
|
|
- docstring: null
|
|
function: 'context.Design:'
|
|
- docstring: ' Experiment
|
|
|
|
# compomentsdataset: Optional[Design] = Nonedatahandler: Optional[Design] = Nonemodel:
|
|
Optional[Design] = Nonerecord: Optional[Design] = Nonestrategy: Optional[Design]
|
|
= Nonebacktest: Optional[Design] = None# basictemplate: Optional[Path] = None#
|
|
rolling strategy. None indicates no rollingrolling: Optional[Literal["base", "ddgda"]]
|
|
= None@dataclass'
|
|
function: 'context.Exp:'
|
|
- docstring: ' Part of the context have clear meaning and structure, so they will
|
|
be saved here and can be easily retrieved and understood
|
|
|
|
# TODO: move more content in WorkflowContextManager.context to hereworkspace:
|
|
Pathexp_list: List[Exp] = field(default_factory=list) # the planned experiments'
|
|
function: 'context.StructContext:'
|
|
- docstring: ' Context Manager stores the context of the workflow
|
|
|
|
All context are key value pairs which saves the input, output and status of the
|
|
whole workflow'
|
|
function: 'context.WorkflowContextManager:'
|
|
- docstring: null
|
|
function: context.set_context
|
|
- docstring: null
|
|
function: context.get_context
|
|
- docstring: null
|
|
function: context.update_context
|
|
- docstring: ' return a deep copy of the context
|
|
|
|
TODO: do we need to return a deep copy?'
|
|
function: context.get_all_context
|
|
- docstring: null
|
|
function: context.retrieve
|
|
- docstring: " \nreturn the template pathBecause the template path is located in\
|
|
\ the folder. We don't know where it is located. So __file__ for this module will\
|
|
\ be used."
|
|
function: __init__.get_finco_path
|
|
- docstring: " \nANSI color codes for use in console output."
|
|
function: 'log.LogColors:'
|
|
- docstring: null
|
|
function: log.get_all_colors
|
|
- docstring: " \nrender text by input color and style. It's not recommend that\
|
|
\ input text is already rendered."
|
|
function: log.render
|
|
- docstring: " \na context manager, print liens before and after a function"
|
|
function: log.formatting_log
|
|
- docstring: null
|
|
function: log.FinCoLog
|
|
- docstring: " \nmessages is some info like this [{\"role\": \"system\",\"\
|
|
content\": system_prompt,},{\"role\": \"user\",\"content\": user_prompt,},]"
|
|
function: log.log_message
|
|
- docstring: null
|
|
function: log.log_response
|
|
- docstring: null
|
|
function: log.info
|
|
- docstring: null
|
|
function: log.plain_info
|
|
- docstring: null
|
|
function: log.warning
|
|
- docstring: " \nreturn the template pathBecause the template path is located in\
|
|
\ the folder. We don't know where it is located. So __file__ for this module will\
|
|
\ be used."
|
|
function: __init__.get_tpl_path
|
|
- docstring: null
|
|
function: utils.ConcatDataset
|
|
- docstring: ' Modeling things
|
|
|
|
@abc.abstractmethod'
|
|
function: base.BaseModel
|
|
- docstring: ' Make predictions after modeling things
|
|
|
|
leverage Python syntactic sugar to make the models'' behaviors like functions'
|
|
function: base.predict
|
|
- docstring: ' Learnable Models
|
|
|
|
'
|
|
function: base.Model
|
|
- docstring: " \nLearn model from the base model.. note::The attribute names\
|
|
\ of learned model should `not` start with '_'. So that the model could bedumped\
|
|
\ to disk.The following code example shows how to retrieve `x_train`, `y_train`\
|
|
\ and `w_train` from the `dataset`:.. code-block:: Python# get features and labelsdf_train,\
|
|
\ df_valid = dataset.prepare([\"train\", \"valid\"], col_set=[\"feature\", \"\
|
|
label\"], data_key=DataHandlerLP.DK_L)x_train, y_train = df_train[\"feature\"\
|
|
], df_train[\"label\"]x_valid, y_valid = df_valid[\"feature\"], df_valid[\"label\"\
|
|
]# get weightstry:wdf_train, wdf_valid = dataset.prepare([\"train\", \"valid\"\
|
|
], col_set=[\"weight\"],data_key=DataHandlerLP.DK_L)w_train, w_valid = wdf_train[\"\
|
|
weight\"], wdf_valid[\"weight\"]except KeyError as e:w_train = pd.DataFrame(np.ones_like(y_train.values),\
|
|
\ index=y_train.index)w_valid = pd.DataFrame(np.ones_like(y_valid.values), index=y_valid.index)Parameters----------dataset\
|
|
\ : Datasetdataset will generate the processed data from model training."
|
|
function: base.fit
|
|
- docstring: ' give prediction given Dataset
|
|
|
|
Parameters----------dataset : Datasetdataset will generate the processed dataset
|
|
from model training.segment : Text or slicedataset will use this segment to prepare
|
|
data. (default=test)Returns-------Prediction results with certain type such as
|
|
`pandas.Series`.'
|
|
function: base.predict
|
|
- docstring: ' Model (F)ine(t)unable
|
|
|
|
@abc.abstractmethod'
|
|
function: base.ModelFT
|
|
- docstring: ' finetune model based given dataset
|
|
|
|
A typical use case of finetuning model with qlib.workflow.R.. code-block:: python#
|
|
start exp to train init modelwith R.start(experiment_name="init models"):model.fit(dataset)R.save_objects(init_model=model)rid
|
|
= R.get_recorder().id# Finetune model based on previous trained modelwith R.start(experiment_name="finetune
|
|
model"):recorder = R.get_recorder(recorder_id=rid, experiment_name="init models")model
|
|
= recorder.load_object("init_model")model.finetune(dataset, num_boost_round=10)Parameters----------dataset
|
|
: Datasetdataset will generate the processed dataset from model training.'
|
|
function: base.finetune
|
|
- docstring: null
|
|
function: trainer._log_task_info
|
|
- docstring: null
|
|
function: trainer._exe_task
|
|
- docstring: " \nBegin task training to start a recorder and save the task config.Args:task_config\
|
|
\ (dict): the config of a taskexperiment_name (str): the name of experimentrecorder_name\
|
|
\ (str): the given name will be the recorder name. None for using rid.Returns:Recorder:\
|
|
\ the model recorder"
|
|
function: trainer.begin_task_train
|
|
- docstring: " \nFinish task training with real model fitting and saving.Args:rec\
|
|
\ (Recorder): the recorder will be resumedexperiment_name (str): the name of experimentReturns:Recorder:\
|
|
\ the model recorder"
|
|
function: trainer.end_task_train
|
|
- docstring: " \nTask based training, will be divided into two steps.Parameters----------task_config\
|
|
\ : dictThe config of a task.experiment_name: strThe name of experimentrecorder_name:\
|
|
\ strThe name of recorderReturns----------Recorder: The instance of the recorder"
|
|
function: trainer.task_train
|
|
- docstring: " \nThe trainer can train a list of models.There are Trainer and DelayTrainer,\
|
|
\ which can be distinguished by when it will finish real training."
|
|
function: 'trainer.Trainer:'
|
|
- docstring: " \nGiven a list of task definitions, begin training, and return\
|
|
\ the models.For Trainer, it finishes real training in this method.For DelayTrainer,\
|
|
\ it only does some preparation in this method.Args:tasks: a list of tasksReturns:list:\
|
|
\ a list of models"
|
|
function: trainer.train
|
|
- docstring: " \nGiven a list of models, finished something at the end of training\
|
|
\ if you need.The models may be Recorder, txt file, database, and so on.For Trainer,\
|
|
\ it does some finishing touches in this method.For DelayTrainer, it finishes\
|
|
\ real training in this method.Args:models: a list of modelsReturns:list: a list\
|
|
\ of models"
|
|
function: trainer.end_train
|
|
- docstring: " \nIf Trainer will delay finishing `end_train`.Returns:bool:\
|
|
\ if DelayTrainer"
|
|
function: trainer.is_delay
|
|
- docstring: " \nSome trainer has backend worker to support parallel trainingThis\
|
|
\ method can tell if the worker is enabled.Returns-------bool:if the worker is\
|
|
\ enabled"
|
|
function: trainer.has_worker
|
|
- docstring: " \nstart the workerRaises------NotImplementedError:If the worker\
|
|
\ is not supported"
|
|
function: trainer.worker
|
|
- docstring: " \nTrainer based on (R)ecorder.It will train a list of tasks and\
|
|
\ return a list of model recorders in a linear way.Assumption: models were defined\
|
|
\ by `task` and the results will be saved to `Recorder`."
|
|
function: trainer.TrainerR
|
|
- docstring: " \nGiven a list of `tasks` and return a list of trained Recorder.\
|
|
\ The order can be guaranteed.Args:tasks (list): a list of definitions based on\
|
|
\ `task` dicttrain_func (Callable): the training method which needs at least `tasks`\
|
|
\ and `experiment_name`. None for the default training method.experiment_name\
|
|
\ (str): the experiment name, None for use default name.kwargs: the params for\
|
|
\ train_func.Returns:List[Recorder]: a list of Recorders"
|
|
function: trainer.train
|
|
- docstring: " \nSet STATUS_END tag to the recorders.Args:models (list): a\
|
|
\ list of trained recorders.Returns:List[Recorder]: the same list as the param."
|
|
function: trainer.end_train
|
|
- docstring: " \nA delayed implementation based on TrainerR, which means `train`\
|
|
\ method may only do some preparation and `end_train` method can do the real model\
|
|
\ fitting."
|
|
function: trainer.DelayTrainerR
|
|
- docstring: " \nGiven a list of Recorder and return a list of trained Recorder.This\
|
|
\ class will finish real data loading and model fitting.Args:models (list): a\
|
|
\ list of Recorder, the tasks have been saved to themend_train_func (Callable,\
|
|
\ optional): the end_train method which needs at least `recorders` and `experiment_name`.\
|
|
\ Defaults to None for using self.end_train_func.experiment_name (str): the experiment\
|
|
\ name, None for use default name.kwargs: the params for end_train_func.Returns:List[Recorder]:\
|
|
\ a list of Recorders"
|
|
function: trainer.end_train
|
|
- docstring: " \nTrainer based on (R)ecorder and Task(M)anager.It can train a list\
|
|
\ of tasks and return a list of model recorders in a multiprocessing way.Assumption:\
|
|
\ `task` will be saved to TaskManager and `task` will be fetched and trained from\
|
|
\ TaskManager"
|
|
function: trainer.TrainerRM
|
|
- docstring: " \nGiven a list of `tasks` and return a list of trained Recorder.\
|
|
\ The order can be guaranteed.This method defaults to a single process, but TaskManager\
|
|
\ offered a great way to parallel training.Users can customize their train_func\
|
|
\ to realize multiple processes or even multiple machines.Args:tasks (list): a\
|
|
\ list of definitions based on `task` dicttrain_func (Callable): the training\
|
|
\ method which needs at least `tasks` and `experiment_name`. None for the default\
|
|
\ training method.experiment_name (str): the experiment name, None for use default\
|
|
\ name.before_status (str): the tasks in before_status will be fetched and trained.\
|
|
\ Can be STATUS_WAITING, STATUS_PART_DONE.after_status (str): the tasks after\
|
|
\ trained will become after_status. Can be STATUS_WAITING, STATUS_PART_DONE.kwargs:\
|
|
\ the params for train_func.Returns:List[Recorder]: a list of Recorders"
|
|
function: trainer.train
|
|
- docstring: " \nSet STATUS_END tag to the recorders.Args:recs (list): a list\
|
|
\ of trained recorders.Returns:List[Recorder]: the same list as the param."
|
|
function: trainer.end_train
|
|
- docstring: " \nThe multiprocessing method for `train`. It can share a same\
|
|
\ task_pool with `train` and can run in other progress or other machines.Args:train_func\
|
|
\ (Callable): the training method which needs at least `tasks` and `experiment_name`.\
|
|
\ None for the default training method.experiment_name (str): the experiment name,\
|
|
\ None for use default name."
|
|
function: trainer.worker
|
|
- docstring: null
|
|
function: trainer.has_worker
|
|
- docstring: " \nA delayed implementation based on TrainerRM, which means `train`\
|
|
\ method may only do some preparation and `end_train` method can do the real model\
|
|
\ fitting."
|
|
function: trainer.DelayTrainerRM
|
|
- docstring: " \nSame as `train` of TrainerRM, after_status will be STATUS_PART_DONE.Args:tasks\
|
|
\ (list): a list of definition based on `task` dicttrain_func (Callable): the\
|
|
\ train method which need at least `tasks` and `experiment_name`. Defaults to\
|
|
\ None for using self.train_func.experiment_name (str): the experiment name, None\
|
|
\ for use default name.Returns:List[Recorder]: a list of Recorders"
|
|
function: trainer.train
|
|
- docstring: " \nGiven a list of Recorder and return a list of trained Recorder.This\
|
|
\ class will finish real data loading and model fitting.Args:recs (list): a list\
|
|
\ of Recorder, the tasks have been saved to them.end_train_func (Callable, optional):\
|
|
\ the end_train method which need at least `recorders` and `experiment_name`.\
|
|
\ Defaults to None for using self.end_train_func.experiment_name (str): the experiment\
|
|
\ name, None for use default name.kwargs: the params for end_train_func.Returns:List[Recorder]:\
|
|
\ a list of Recorders"
|
|
function: trainer.end_train
|
|
- docstring: " \nThe multiprocessing method for `end_train`. It can share a\
|
|
\ same task_pool with `end_train` and can run in other progress or other machines.Args:end_train_func\
|
|
\ (Callable, optional): the end_train method which need at least `recorders` and\
|
|
\ `experiment_name`. Defaults to None for using self.end_train_func.experiment_name\
|
|
\ (str): the experiment name, None for use default name."
|
|
function: trainer.worker
|
|
- docstring: ' Feature (Int)erpreter
|
|
|
|
@abstractmethod'
|
|
function: 'base.FeatureInt:'
|
|
- docstring: ' get feature importance
|
|
|
|
Returns-------The index is the feature name.The greater the value, the higher
|
|
importance.'
|
|
function: base.get_feature_importance
|
|
- docstring: ' LightGBM (F)eature (Int)erpreter
|
|
|
|
self.model = None'
|
|
function: base.LightGBMFInt
|
|
- docstring: ' get feature importance
|
|
|
|
Notes-----parameters reference:https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Booster.html?highlight=feature_importance#lightgbm.Booster.feature_importance'
|
|
function: base.get_feature_importance
|
|
- docstring: ' Risk Model
|
|
|
|
A risk model is used to estimate the covariance matrix of stock returns.'
|
|
function: base.RiskModel
|
|
- docstring: " \nArgs:X (pd.Series, pd.DataFrame or np.ndarray): data from\
|
|
\ which to estimate the covariance,with variables as columns and observations\
|
|
\ as rows.return_corr (bool): whether return the correlation matrix.is_price (bool):\
|
|
\ whether `X` contains price (if not assume stock returns).return_decomposed_components\
|
|
\ (bool): whether return decomposed components of the covariance matrix.Returns:pd.DataFrame\
|
|
\ or np.ndarray: estimated covariance (or correlation)."
|
|
function: base.predict
|
|
- docstring: ' covariance estimation implementation
|
|
|
|
This method should be overridden by child classes.By default, this method implements
|
|
the empirical covariance estimation.Args:X (np.ndarray): data matrix containing
|
|
multiple variables (columns) and observations (rows).Returns:np.ndarray: covariance
|
|
matrix.'
|
|
function: base._predict
|
|
- docstring: ' handle nan and centerize data
|
|
|
|
Note:if `nan_option=''mask''` then the returned array will be `np.ma.MaskedArray`.'
|
|
function: base._preprocess
|
|
- docstring: " Principal Orthogonal Complement Thresholding Estimator (POET)\n\
|
|
Reference:[1] Fan, J., Liao, Y., & Mincheva, M. (2013). Large covariance estimation\
|
|
\ by thresholding principal orthogonal complements.Journal of the Royal Statistical\
|
|
\ Society. Series B: Statistical Methodology, 75(4), 603\u2013680. https://doi.org/10.1111/rssb.12016[2]\
|
|
\ http://econweb.rutgers.edu/yl1114/papers/poet/POET.m"
|
|
function: poet.POETCovEstimator
|
|
- docstring: " Structured Covariance Estimator\nThis estimator assumes observations\
|
|
\ can be predicted by multiple factorsX = B @ F.T + Uwhere `X` contains observations\
|
|
\ (row) of multiple variables (column),`F` contains factor exposures (column)\
|
|
\ for all variables (row),`B` is the regression coefficients matrix for all observations\
|
|
\ (row) onall factors (columns), and `U` is the residual matrix with shape like\
|
|
\ `X`.Therefore, the structured covariance can be estimated bycov(X.T) = F @ cov(B.T)\
|
|
\ @ F.T + diag(var(U))In finance domain, there are mainly three methods to design\
|
|
\ `F` [1][2]:- Statistical Risk Model (SRM): latent factor models major components-\
|
|
\ Fundamental Risk Model (FRM): human designed factors- Deep Risk Model (DRM):\
|
|
\ neural network designed factors (like a blend of SRM & DRM)In this implementation\
|
|
\ we use latent factor models to specify `F`.Specifically, the following two latent\
|
|
\ factor models are supported:- `pca`: Principal Component Analysis- `fa`: Factor\
|
|
\ AnalysisReference:[1] Fan, J., Liao, Y., & Liu, H. (2016). An overview of the\
|
|
\ estimation of large covariance andprecision matrices. Econometrics Journal,\
|
|
\ 19(1), C1\u2013C32. https://doi.org/10.1111/ectj.12061[2] Lin, H., Zhou, D.,\
|
|
\ Liu, W., & Bian, J. (2021). Deep Risk Model: A Deep Learning Solution forMining\
|
|
\ Latent Risk Factors to Improve Covariance Matrix Estimation. arXiv preprint\
|
|
\ arXiv:2107.05201."
|
|
function: structured.StructuredCovEstimator
|
|
- docstring: " \ncovariance estimation implementationArgs:X (np.ndarray): data\
|
|
\ matrix containing multiple variables (columns) and observations (rows).return_decomposed_components\
|
|
\ (bool): whether return decomposed components of the covariance matrix.Returns:tuple\
|
|
\ or np.ndarray: decomposed covariance matrix or covariance matrix."
|
|
function: structured._predict
|
|
- docstring: " Shrinkage Covariance Estimator\nThis estimator will shrink the sample\
|
|
\ covariance matrix towardsan identify matrix:S_hat = (1 - alpha) * S + alpha\
|
|
\ * Fwhere `alpha` is the shrink parameter and `F` is the shrinking target.The\
|
|
\ following shrinking parameters (`alpha`) are supported:- `lw` [1][2][3]: use\
|
|
\ Ledoit-Wolf shrinking parameter.- `oas` [4]: use Oracle Approximating Shrinkage\
|
|
\ shrinking parameter.- float: directly specify the shrink parameter, should be\
|
|
\ between [0, 1].The following shrinking targets (`F`) are supported:- `const_var`\
|
|
\ [1][4][5]: assume stocks have the same constant variance and zero correlation.-\
|
|
\ `const_corr` [2][6]: assume stocks have different variance but equal correlation.-\
|
|
\ `single_factor` [3][7]: assume single factor model as the shrinking target.-\
|
|
\ np.ndarray: provide the shrinking targets directly.Note:- The optimal shrinking\
|
|
\ parameter depends on the selection of the shrinking target.Currently, `oas`\
|
|
\ is not supported for `const_corr` and `single_factor`.- Remember to set `nan_option`\
|
|
\ to `fill` or `mask` if your data has missing values.References:[1] Ledoit, O.,\
|
|
\ & Wolf, M. (2004). A well-conditioned estimator for large-dimensional covariance\
|
|
\ matrices.Journal of Multivariate Analysis, 88(2), 365\u2013411. https://doi.org/10.1016/S0047-259X(03)00096-4[2]\
|
|
\ Ledoit, O., & Wolf, M. (2004). Honey, I shrunk the sample covariance matrix.Journal\
|
|
\ of Portfolio Management, 30(4), 1\u201322. https://doi.org/10.3905/jpm.2004.110[3]\
|
|
\ Ledoit, O., & Wolf, M. (2003). Improved estimation of the covariance matrix\
|
|
\ of stock returnswith an application to portfolio selection.Journal of Empirical\
|
|
\ Finance, 10(5), 603\u2013621. https://doi.org/10.1016/S0927-5398(03)00007-0[4]\
|
|
\ Chen, Y., Wiesel, A., Eldar, Y. C., & Hero, A. O. (2010). Shrinkage algorithms\
|
|
\ for MMSE covarianceestimation. IEEE Transactions on Signal Processing, 58(10),\
|
|
\ 5016\u20135029.https://doi.org/10.1109/TSP.2010.2053029[5] https://www.econ.uzh.ch/dam/jcr:ffffffff-935a-b0d6-0000-00007f64e5b9/cov1para.m.zip[6]\
|
|
\ https://www.econ.uzh.ch/dam/jcr:ffffffff-935a-b0d6-ffff-ffffde5e2d4e/covCor.m.zip[7]\
|
|
\ https://www.econ.uzh.ch/dam/jcr:ffffffff-935a-b0d6-0000-0000648dfc98/covMarket.m.zip"
|
|
function: shrink.ShrinkCovEstimator
|
|
- docstring: null
|
|
function: shrink._predict
|
|
- docstring: ' get shrinking target `F`
|
|
|
|
if self.target == self.TGT_CONST_VAR:return self._get_shrink_target_const_var(X,
|
|
S)if self.target == self.TGT_CONST_CORR:return self._get_shrink_target_const_corr(X,
|
|
S)if self.target == self.TGT_SINGLE_FACTOR:return self._get_shrink_target_single_factor(X,
|
|
S)return self.target'
|
|
function: shrink._get_shrink_target
|
|
- docstring: ' get shrinking target with constant variance
|
|
|
|
This target assumes zero pair-wise correlation and constant variance.The constant
|
|
variance is estimated by averaging all sample''s variances.'
|
|
function: shrink._get_shrink_target_const_var
|
|
- docstring: ' get shrinking target with constant correlation
|
|
|
|
This target assumes constant pair-wise correlation but keep the sample variance.The
|
|
constant correlation is estimated by averaging all pairwise correlations.'
|
|
function: shrink._get_shrink_target_const_corr
|
|
- docstring: ' get shrinking target with single factor model
|
|
|
|
X_mkt = np.nanmean(X, axis=1)cov_mkt = np.asarray(X.T.dot(X_mkt) / len(X))var_mkt
|
|
= np.asarray(X_mkt.dot(X_mkt) / len(X))F = np.outer(cov_mkt, cov_mkt) / var_mktnp.fill_diagonal(F,
|
|
np.diag(S))return F'
|
|
function: shrink._get_shrink_target_single_factor
|
|
- docstring: ' get shrinking parameter `alpha`
|
|
|
|
Note:The Ledoit-Wolf shrinking parameter estimator consists of three different
|
|
methods.'
|
|
function: shrink._get_shrink_param
|
|
- docstring: ' Oracle Approximating Shrinkage Estimator
|
|
|
|
This method uses the following formula to estimate the `alpha`parameter for the
|
|
shrink covariance estimator:A = (1 - 2 / p) * trace(S^2) + trace^2(S)B = (n +
|
|
1 - 2 / p) * (trace(S^2) - trace^2(S) / p)alpha = A / Bwhere `n`, `p` are the
|
|
dim of observations and variables respectively.'
|
|
function: shrink._get_shrink_param_oas
|
|
- docstring: ' Ledoit-Wolf Shrinkage Estimator (Constant Variance)
|
|
|
|
This method shrinks the covariance matrix towards the constand variance target.'
|
|
function: shrink._get_shrink_param_lw_const_var
|
|
- docstring: ' Ledoit-Wolf Shrinkage Estimator (Constant Correlation)
|
|
|
|
This method shrinks the covariance matrix towards the constand correlation target.'
|
|
function: shrink._get_shrink_param_lw_const_corr
|
|
- docstring: ' Ledoit-Wolf Shrinkage Estimator (Single Factor Model)
|
|
|
|
This method shrinks the covariance matrix towards the single factor model target.'
|
|
function: shrink._get_shrink_param_lw_single_factor
|
|
- docstring: " \nA dataset fetching the data in a meta-level.A Meta Dataset is\
|
|
\ responsible for- input tasks(e.g. Qlib tasks) and prepare meta tasks- meta task\
|
|
\ contains more information than normal tasks (e.g. input data for meta model)The\
|
|
\ learnt pattern could transfer to other meta dataset. The following cases should\
|
|
\ be supported- A meta-model trained on meta-dataset A and then applied to meta-dataset\
|
|
\ B- Some pattern are shared between meta-dataset A and B, so meta-input on meta-dataset\
|
|
\ A are used when meta model are applied on meta-dataset-B"
|
|
function: dataset.MetaTaskDataset
|
|
- docstring: " \nPrepare the data in each meta-task and ready for training.The\
|
|
\ following code example shows how to retrieve a list of meta-tasks from the `meta_dataset`:..\
|
|
\ code-block:: Python# get the train segment and the test segment, both of them\
|
|
\ are liststrain_meta_tasks, test_meta_tasks = meta_dataset.prepare_tasks([\"\
|
|
train\", \"test\"])Parameters----------segments: Union[List[Text], Tuple[Text],\
|
|
\ Text]the info to select dataReturns-------list:A list of the prepared data of\
|
|
\ each meta-task for training the meta-model. For multiple segments [seg1, seg2,\
|
|
\ ... , segN], the returned list will be [[tasks in seg1], [tasks in seg2], ...\
|
|
\ , [tasks in segN]].Each task is a meta task"
|
|
function: dataset.prepare_tasks
|
|
- docstring: " \nprepare a single segment of data for training dataParameters----------seg\
|
|
\ : Textthe name of the segment"
|
|
function: dataset._prepare_seg
|
|
- docstring: " \nA single meta-task, a meta-dataset contains a list of them.It\
|
|
\ serves as a component as in MetaDatasetDSThe data processing is different- the\
|
|
\ processed input may be different between training and testing- When training,\
|
|
\ the X, y, X_test, y_test in training tasks are necessary (# PROC_MODE_FULL #)but\
|
|
\ not necessary in test tasks. (# PROC_MODE_TEST #)- When the meta model can be\
|
|
\ transferred into other dataset, only meta_info is necessary (# PROC_MODE_TRANSFER\
|
|
\ #)"
|
|
function: 'task.MetaTask:'
|
|
- docstring: null
|
|
function: task.get_dataset
|
|
- docstring: " \nReturn the **processed** meta_info"
|
|
function: task.get_meta_input
|
|
- docstring: " \nThe meta-model guiding the model learning.The word `Guiding` can\
|
|
\ be categorized into two types based on the stage of model learning- The definition\
|
|
\ of learning tasks: Please refer to docs of `MetaTaskModel`- Controlling the\
|
|
\ learning process of models: Please refer to the docs of `MetaGuideModel`"
|
|
function: model.MetaModel
|
|
- docstring: " \nThe training process of the meta-model."
|
|
function: model.fit
|
|
- docstring: " \nThe inference process of the meta-model.Returns-------object:Some\
|
|
\ information to guide the model learning"
|
|
function: model.inference
|
|
- docstring: " \nThis type of meta-model deals with base task definitions. The\
|
|
\ meta-model creates tasks for training new base forecasting models after it is\
|
|
\ trained. `prepare_tasks` directly modifies the task definitions."
|
|
function: model.MetaTaskModel
|
|
- docstring: " \nThe MetaTaskModel is expected to get prepared MetaTask from\
|
|
\ meta_dataset.And then it will learn knowledge from the meta tasks"
|
|
function: model.fit
|
|
- docstring: " \nMetaTaskModel will make inference on the meta_datasetThe MetaTaskModel\
|
|
\ is expected to get prepared MetaTask from meta_dataset.Then it will create modified\
|
|
\ task with Qlib format which can be executed by Qlib trainer.Returns-------List[dict]:A\
|
|
\ list of modified task definitions."
|
|
function: model.inference
|
|
- docstring: " \nThis type of meta-model aims to guide the training process of\
|
|
\ the base model. The meta-model interacts with the base forecasting models during\
|
|
\ their training process."
|
|
function: model.MetaGuideModel
|
|
- docstring: null
|
|
function: model.fit
|
|
- docstring: ' Group the objects based on dict
|
|
|
|
'
|
|
function: 'group.Group:'
|
|
- docstring: " \nGroup a set of objects and change them to a dict.For example:\
|
|
\ {(A,B,C1): object, (A,B,C2): object} -> {(A,B): {C1: object, C2: object}}Returns:dict:\
|
|
\ grouped dict"
|
|
function: group.group
|
|
- docstring: " \nReduce grouped dict.For example: {(A,B): {C1: object, C2:\
|
|
\ object}} -> {(A,B): object}Returns:dict: reduced dict"
|
|
function: group.reduce
|
|
- docstring: ' Group the rolling dict
|
|
|
|
'
|
|
function: group.RollingGroup
|
|
- docstring: ' Given an rolling dict likes {(A,B,R): things}, return the grouped
|
|
dict likes {(A,B): {R:things}}
|
|
|
|
NOTE: There is an assumption which is the rolling key is at the end of the key
|
|
tuple, because the rolling results always need to be ensemble firstly.Args:rolling_dict
|
|
(dict): an rolling dict. If the key is not a tuple, then do nothing.Returns:dict:
|
|
grouped dict'
|
|
function: group.group
|
|
- docstring: ' Merge the ensemble_dict into an ensemble object.
|
|
|
|
For example: {Rollinga_b: object, Rollingb_c: object} -> objectWhen calling this
|
|
class:Args:ensemble_dict (dict): the ensemble dict like {name: things} waiting
|
|
for mergingReturns:object: the ensemble object'
|
|
function: 'ensemble.Ensemble:'
|
|
- docstring: " \nExtract the object if there is only one key and value in the dict.\
|
|
\ Make the result more readable.{Only key: Only value} -> Only valueIf there is\
|
|
\ more than 1 key or less than 1 key, then do nothing.Even you can run this recursively\
|
|
\ to make dict more readable.NOTE: Default runs recursively.When calling this\
|
|
\ class:Args:ensemble_dict (dict): the dict. The key of the dict will be ignored.Returns:dict:\
|
|
\ the readable dict."
|
|
function: ensemble.SingleKeyEnsemble
|
|
- docstring: ' Merge a dict of rolling dataframe like `prediction` or `IC` into
|
|
an ensemble.
|
|
|
|
NOTE: The values of dict must be pd.DataFrame, and have the index "datetime".When
|
|
calling this class:Args:ensemble_dict (dict): a dict like {"A": pd.DataFrame,
|
|
"B": pd.DataFrame}.The key of the dict will be ignored.Returns:pd.DataFrame: the
|
|
complete result of rolling.'
|
|
function: ensemble.RollingEnsemble
|
|
- docstring: " \nAverage and standardize a dict of same shape dataframe like `prediction`\
|
|
\ or `IC` into an ensemble.NOTE: The values of dict must be pd.DataFrame, and\
|
|
\ have the index \"datetime\". If it is a nested dict, then flat it.When calling\
|
|
\ this class:Args:ensemble_dict (dict): a dict like {\"A\": pd.DataFrame, \"B\"\
|
|
: pd.DataFrame}.The key of the dict will be ignored.Returns:pd.DataFrame: the\
|
|
\ complete result of averaging and standardizing."
|
|
function: ensemble.AverageEnsemble
|
|
- docstring: 'All the models can be import from `qlib.contrib.models` # Keywords:
|
|
supervised learning'
|
|
- docstring: "The API to run rolling models can be found in \u2026 #Keywords: control"
|
|
- docstring: "Here are a list of Qlib\u2019s available analyzers. #KEYWORDS: analysis"
|