1
0
mirror of https://github.com/microsoft/qlib.git synced 2026-06-06 05:51:17 +08:00
Commit Graph

149 Commits

Author SHA1 Message Date
cyncyw
c9ed050ef0 Ptnn4both datatypes and alignment tests (#1827)
* Init model for both dataset

* Remove some deprecated code

* Add model template;

* We must align with previous results

* We choose another mode as the initial version

* Almost success to run GRU

* Successfully run training

* Passed general_nn test

* gru test

* Alignment test passed

* comment

* fix readme & minor errors

* general nn updates & benchmarks

* Update examples/benchmarks/GeneralPtNN/workflow_config_gru2mlp.yaml

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2024-07-11 17:59:18 +08:00
Linlang
2c33332dd6 More dataloader example (#1823)
* More dataloader example

* optimize code

* optimeze code

* optimeze code

* optimeze code

* optimeze code

* optimeze code

* fix pylint error

* fix CI error

* fix CI error

* Comments

* fix error type

---------

Co-authored-by: Young <afe.young@gmail.com>
2024-07-10 14:48:44 +08:00
you-n-g
a7d5a9b500 Nested data loader (#1822)
* nested data loader

* Amend

* add data loder test

* fix pylint error

* fix pytest error

* fix pytest error

* delete comments

* Update qlib/contrib/data/handler.py

---------

Co-authored-by: Linlang <Lv.Linlang@hotmail.com>
2024-07-05 15:44:16 +08:00
cyncyw
cde80206e4 Update index_data.py for datatype conversion and alignment (#1813)
* Update index_data.py for data convertion and alignment

* Update qlib/utils/index_data.py

* Update qlib/utils/index_data.py

* fix linting

---------

Co-authored-by: taozhiwang <taozhiwa@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2024-06-24 15:34:48 +08:00
陈屹华
ebc0ca893e Fix TSDataSampler Slicing Bug #1716 (#1803)
* Fix TSDataSampler Slicing Bug #1716

* Fix TSDataSampler Slicing Bug #1716

* Fix TSDataSampler Slicing Bug #1716

* Fix TSDataSampler Slicing Bug with simplyer implmentation#1716
 with Simplified Implementation

* Refactor: Fix CI errors by addressing pylint formatting issues

* Refactor: Remove extraneous whitespace for improved code formatting with Black
2024-06-21 09:25:23 +08:00
Lee Yuntong
3a348aec9f Fix typo (#1811)
Co-authored-by: LeeYuntong <nukuihayu@outlook.com>
2024-06-20 18:12:07 +08:00
Linlang
39f88daaa7 download orderbook data (#1754)
* download orderbook data

* fix CI error

* fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* test fix CI error

* optimize get_data code

* optimize get_data code

* optimize get_data code

* optimize README

---------

Co-authored-by: Linlang <v-linlanglv@microsoft.com>
2024-03-07 14:41:21 +08:00
Fivele-Li
4c30e5827b Troubleshooting pip version issues in CI (#1504)
* CI failed to run on 23.1 and 23.1.1

* add pyproject.toml

* upgrade pip in slow.yml

* upgrade build-system requires

* troubleshooting pytest problem

* troubleshooting pytest problem

* troubleshooting pytest problem

* troubleshooting pytest problem

* add qlib root path to python sys.path

* add qlib root path to $PYTHONPATH

* add qlib root path to $PYTHONPATH

* add qlib root path to $PYTHONPATH

* modify pytest root;

* remove set env

* change_pytest_command_CI

* change_pytest_command_CI

* fix_ci

* fix_ci

* fix_ci

* fix_ci

* fix_ci

* fix_ci

* fix_ci

* remove_toml

* recover_toml

---------

Co-authored-by: lijinhui <362237642@qq.com>
Co-authored-by: linlang <Lv.Linlang@hotmail.com>
2023-08-24 21:24:50 +08:00
you-n-g
be4646b4b7 Adjust rolling api (#1594)
* Intermediate version

* Fix yaml template & Successfully run rolling

* Be compatible with benchmark

* Get same results with previous linear model

* Black formatting

* Update black

* Update the placeholder mechanism

* Update CI

* Update CI

* Upgrade Black

* Fix CI and simplify code

* Fix CI

* Move the data processing caching mechanism into utils.

* Adjusting DDG-DA

* Organize import
2023-07-14 12:16:12 +08:00
Linlang
21f0b394e7 change get_data url (#1558)
* change_url

* fix_CI

* fix_CI_2

* fix_CI_3

* fix_CI_4

* fix_CI_5

* fix_CI_6

* fix_CI_7

* fix_CI_8

* fix_CI_9

* fix_CI_10

* fix_CI_11

* fix_CI_12

* fix_CI_13

* fix_CI_13

* fix_CI_14

* fix_CI_15

* fix_CI_16

* fix_CI_17

* fix_CI_18

* fix_CI_19

* fix_CI_20

* fix_CI_21

* fix_CI_22

* fix_CI_23

* fix_CI_24

* fix_CI_25

* fix_CI_26

* fix_CI_27

* fix_get_data_error

* fix_get_data_error2

* modify_get_data

* modify_get_data2

* modify_get_data3

* modify_get_data4

* fix_CI_28

* fix_CI_29

* fix_CI_30

---------

Co-authored-by: Linlang <v-linlanglv@microsoft.com>
2023-06-25 23:39:11 +08:00
Chaoying
37a59f28d3 Fix deprecated syntax in numpy (#1507)
* Fix deprecated syntax in numpy

* Replace np.bool with bool
2023-05-08 16:17:02 +08:00
Huoran Li
7f1e8c5206 Refine Qlib RL data format (#1480)
* wip

* wip

* wip

* Fix naming errors

* Backtest test passed

* Why training stuck?

* Minor

* Refine train configs

* Use dummy in training

* Remove pickle_dataframe

* CI

* CI

* Add more strict condition to filter orders

* Pass test

* Add TODO in example

---------

Co-authored-by: Young <afe.young@gmail.com>
2023-04-26 21:14:30 +08:00
you-n-g
32c3070b73 Refine DDG-DA (#1472)
* Run ddg-da successfully

* Support include valid; More parameters

* Support L2 reg & visualization

* Blackformat

* Enable fill_method

* Support specify handler & optim dataset

* Fix Pylint
2023-04-07 15:00:21 +08:00
YQ Tsui
d8764660dc [BUGFIX] allow sell in limit-up case and allow buy in limit-down case in topk strategy (#1407)
* 1) check limit_up/down should consider direction; 2) fix some typo, typehint etc

* fix error

* Update test_all_pipeline.py

Believe it's just some arbitrary number.
The excess return is expected to change when trading logic changes.

* add flag forbid_all_trade_at_limit to keep previous behivour for backward compatibility
2023-01-10 09:46:18 +08:00
Linlang
756bd0f65b Fix ZScoreNorm processor bug (#1398)
* fix_ZScoreNorm_bug

* fix_CI_error

* fix_CI_error

* add_test_processor

* fix_pylint_error

* fix_some_error_and_optimize_code

* modify_terrible_code

* optimize_code

* optimize_code
2022-12-30 20:42:37 +08:00
Chia-hung Tai
0c4db8b0f8 Set _artifact_uri when mlflow_run is not None. (#1367)
* Set _artifact_uri when mlflow_run is not None.

* Fix black.
2022-11-19 11:56:30 +08:00
Chia-hung Tai
e47b0f1c50 Fix typo. (#1365) 2022-11-19 11:52:34 +08:00
you-n-g
994f89319d Optimize the implementation of uri & Fix async log bug (#1364)
* Optimize the implementation of uri

* remove redundant func

* Set the right order of _set_client_uri

* Update qlib/workflow/expm.py

* Simplify client & add test.Add docs; Fix async bug

* Fix comments & pylint

* Improve README
2022-11-18 13:11:31 +08:00
Huoran Li
35794846ff Refine RL todos (#1332)
* Refine several todos

* CI issues

* Remove Dropna limitation of `quote_df` in Exchange  (#1334)

* Remove Dropna limitation of `quote_df` of Exchange

* Impreove docstring

* Fix type error when expression is specified (#1335)

* Refine fill_missing_data()

* Remove several TODO comments

* Add back env for interpreters

* Change Literal import

* Resolve PR comments

* Move  to SAOEState

* Add Trainer.get_policy_state_dict()

* Mypy issue

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2022-11-10 21:10:11 +08:00
Chia-hung Tai
fb5888be9e Use mock data for element operator tests. (#1330) 2022-10-30 16:27:59 +08:00
Linlang
1861c8edaf optimize_CI (#1314) 2022-10-20 08:38:05 +08:00
Chia-hung Tai
84e9df2603 Add REG_US and REG_TW into test case: test_utils.py. (#1310)
* Add REG_US and REG_TW into test case: test_utils.py.

* Fix black.

* Trigger checks.

* Add REG_US and REG_TW into test case: test_utils.py.

* Fix black.

* Trigger checks.
2022-10-14 11:49:30 +08:00
Huoran Li
216a8ec2de RL backtest with simulator (#1299)
* RL backtest with simulator

* Minor modification in init_qlib

* Cherry pick PR 1302

* Resolve PR comments

* Fix missing data processing

* Minor bugfix

* Add TODOs and docs

* Add a comment
2022-10-12 16:44:28 +08:00
Huoran Li
bee05f56ef Migrate backtest logic from NT (#1263)
* Backtest migration

* Minor bug fix in test

* Reorganize file to avoid loop import

* Fix test SAOE bug

* Remove unnecessary names

* Resolve PR comments; remove private classes;

* Fix CI error

* Resolve PR comments

* Refactor data interfaces

* Remove convert_instance_config and change config

* Pylint issue

* Pylint issue

* Fix tempfile warning

* Resolve PR comments

* Add more comments
2022-09-19 14:54:26 +08:00
Huoran Li
1d65d28b28 Qlib simulator refinement (redo of PR 1244) (#1262)
* Use dict-like configuration

* Rename from_neutrader to integration

* SAOE strategy

* Optimize file structure

* Optimize code

* Format code

* create_state_maintainer_recursive

* Remove explicit time_per_step

* CI test passed

* Resolve PR comments

* Pass all CI

* Minor test issue

* Refine SAOE adapter logic

* Minor bugfix

* Cherry pick updates

* Resolve PR comments

* CI issues

* Refine adapter & saoe_data logic

* Resolve PR comments

* Resolve PR comments

* Rename ONE_SEC to EPS_T; complete backtest loop

* CI issue

* Resolve Yuge's PR comments
2022-08-24 14:09:45 +08:00
Huoran Li
2752bdc92c Migrate NeuTrader to Qlib RL (#1169)
* Refine previous version RL codes

* Polish utils/__init__.py

* Draft

* Use | instead of Union

* Simulator & action interpreter

* Test passed

* Migrate to SAOEState & new qlib interpreter

* Black format

* . Revert file_storage change

* Refactor file structure & renaming functions

* Enrich test cases

* Add QlibIntradayBacktestData

* Test interpreter

* Black format

* .

.

.

* Rename receive_execute_result()

* Use indicator to simplify state update

* Format code

* Modify data path

* Adjust file structure

* Minor change

* Add copyright message

* Format code

* Rename util functions

* Add CI

* Pylint issue

* Remove useless code to pass pylint

* Pass mypy

* Mypy issue

* mypy issue

* mypy issue

* Revert "mypy issue"

This reverts commit 8eb1b0174e.

* mypy issue

* mypy issue

* Fix the numpy version incompatible bug

* Fix a minor typing issue

* Try to skip python 3.7 test for qlib simulator

* Resolve PR comments by Yuge; solve several CI issues.

* Black issue

* Fix a low-level type error

* Change data name

* Resolve PR comments. Leave TODOs in the code base.

Co-authored-by: Young <afe.young@gmail.com>
2022-08-01 09:56:07 +08:00
you-n-g
a6700d81ff Update test_qlib_from_source_slow.yml's timeout setting. (#1178)
* Update test_qlib_from_source_slow.yml

* Update test_qlib_from_source.yml

* Update test_pit.py

* Update test_pit.py

* Update test_pit.py

* Update test_pit.py
2022-07-06 20:44:10 +08:00
Chao Wang
3db22452fb Adding ChangeInstrument op (#1005)
* add ChangeInstrument to ops

Adding Change instrument OP. This op allows one to use  features of a different instrument.

* Update __init__.py

update parse_field to accommodate ChangeInstrument

* Propose test

* Add test case and fix bug

* Update ops.py

* Update ops.py

* simplify the operator further

* implement abstract method

* fix arg bug

* clean test

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2022-07-04 08:45:26 +08:00
Yuge Zhang
25ecb1135f Qlib RL framework (stage 2) - trainer (#1125)
* checkpoint

(cherry picked from commit 1a8e0bd4671ee6d624a7d09bb198a273282cd050)

* Not a workable version

(cherry picked from commit 3498e185684cd5590d3ab97e0ab69eab8c1e0e3a)

* vessel

* ckpt

* .

* vessel

* .

* .

* checkpoint callback

* .

* cleanup

* logger

* .

* test

* .

* add test

* .

* .

* .

* .

* New reward

* Add train API

* fix mypy

* fix lint

* More comment

* 3.7 compat

* fix test

* fix test

* .

* Resolve comments

* fix typehint
2022-06-28 19:53:05 +08:00
Linlang
2ca0d88d2d change_pitdata_source (#1171)
* change_pitdata_source

* retain_normalize

* add_comment
2022-06-28 16:29:59 +08:00
Linlang
50d74b5560 split_CI (#1141) 2022-06-28 10:17:29 +08:00
Huoran Li
89972f6c6f Refine backtest codes (#1120)
* Refine backtest code

* Keep working

* Minor

* Resolve PR comments

* Fix import error

* Fix import error
2022-06-10 12:14:48 +08:00
you-n-g
cc94c32db6 init_instance_by_config enhancement (#1103)
* fix SepDataFrame when we del it to empty

* init_instance_by_config enhancement

* Update test_sepdf.py
2022-05-21 20:16:22 +08:00
Yuge Zhang
9a40fd3cdc Qlib RL framework (stage 1) - single-asset order execution (#1076)
* rl init

* aux info

* Reward config

* update

* simple

* update saoe init

* update simulator and seed

* minor

* minor

* update sim

* checkpoint

* obs

* Update interpreter

* init qlib simulator

* checkpoint

* Refine codebase

* checkpoint

* checkpoint

* Add one test

* More tests

* Simulator checkpoint

* checkpoint

* First-step tested

* Checkpoint

* Update data_queue API

* Checkpoint

* Update test

* Move files

* Checkpoint

* Single-quote -> double-quote

* Fix finite env tests

* Tested with mypy

* pep-574

* No call for env done

* Update finite env docs

* Fix csv writer

* Refine tester

* Update logger

* Add another logger test

* Checkpoint

* Add network sanity test

* steps per episode is not correct

* Cleanup code, ready for PR

* Reformat with black

* Fix pylint for py37

* Fix lint

* Fix lint

* Fix flake

* update mypy command

* mypy

* Update exclude pattern

* Use pyproject.toml

* test

* .

* .

* Refactor pipeline

* .

* defaults run bash

* .

* Revert and skip follow_imports

* Fix toml issue

* fix mypy

* .

* .

* .

* Fix install

* Minor fix

* Fix test

* Fix test

* Remove requirements

* Revert

* fix tests

* Fix lint

* .

* .

* .

* .

* .

* update install from source command

* .

* Fix data download

* .

* .

* .

* .

* .

* .

* Fix py37

* Ignore tests on non-linux

* resolve comments

* fix tests

* resolve comments

* some typo

* style updates

* More comments

* fix dummy

* add warning

* Align precision in some system

* Added some impl notes

Co-authored-by: Young <afe.young@gmail.com>
2022-05-21 18:19:24 +08:00
Linlang
2cf842bcfe add_test_pit (#1089)
* add_test_pit

* add_test_pit_to_tests

* add_baostock_to_setup

* add_pip_to_CI

Co-authored-by: Linlang Lv (iSoftStone) <v-linlanglv@microsoft.com>
2022-05-06 16:47:20 +08:00
you-n-g
c5cf8fb9cc fix est_sepdf.py with black 2022-04-29 17:21:20 +08:00
you-n-g
3c9c76b384 fix SepDataFrame when we del it to empty (#1082) 2022-04-29 14:29:17 +08:00
you-n-g
9d0a8f61d1 Make sepdf more like DataFrame (#1080) 2022-04-28 19:13:45 +08:00
Chia-hung Tai
07f0d4f599 [930] Fix typo HasingStockStorage to HashingStockStorage. (#947) 2022-03-04 12:40:04 +08:00
you-n-g
60d45ad770 Enhance pytorch nn (#917)
* enhance pytorch_nn

* fix dim bug

* Black format

* Fix pylint error
2022-02-15 19:22:48 +08:00
luocy16
2bb8a4ce0e Supporting Arctic Backend Provider & Orderbook, Tick Data Example (#744)
* change weight_decay & batchsize

* del weight_decay

* big weight_decay

* mid weight_decay

* small layer

* 2 layer

* full layer

* no weight decay

* divide into two data source

* change parse field

* delete some debug

* add Toperator

* new format of arctic

* fix cache bug to arctic read

* fix connection problem

* add some operator

* final version for arcitc

* clear HZ cache

* remove not used function

* add topswrappers

* successfully import data and run first test

* A simpler version to support arctic

* Successfully run all high-freq expressions

* Black format and fix add docs

* Add docs for download and test data

* update scripts and docs

* Add docs

* fix bug

* Refine docs

* fix test bug

* fix CI error

* clean code

Co-authored-by: bxdd <bxddream@gmail.com>
Co-authored-by: wangwenxi.handsome <wangwenxi.handsome@gmail.com>
Co-authored-by: Young <afe.young@gmail.com>
2022-01-18 09:13:11 +08:00
Chia-hung Tai
edd8badeaf [840] - Test case for operators. (#841)
* [840] - Test case for operators.

* Move import to the head of file and add test_setting.
2022-01-11 18:44:15 +08:00
Chia-hung Tai
184ce34a34 [807] Move the REG_CONSTANT/EPS to constant.py. (#811)
* [807] Move the REG_CONSTANT to constant.py.

* import REG_US.

* Move EPS to constant.py.
2022-01-09 21:39:46 +08:00
Pengrong Zhu
c276de4040 Fix backtest (#719)
* modify FileStorage to support multiple freqs

* modify backtest's sample documentation

* change the logging level of read data exception from error to debug

* fix the backtest exception when volume is 0 or np.nan

* fix test_storage.py

* add backtest_daily

* modify backtest_daily's docstring

* add __repr__/__str__ to Position

* fix the bug of nested_decision_execution example

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2021-12-07 19:04:23 +08:00
you-n-g
45ebb1d0e0 support adding from date when updating pred (#703)
* support adding from date when updating pred

* fix updating data error
2021-11-22 19:52:52 +08:00
you-n-g
cae4c9c924 An example to get index from TSDataSampler (#679) 2021-11-10 14:35:27 +08:00
Young
3fa48d7017 simplify record tmp 2021-11-05 12:57:14 +00:00
Young
64130d9407 Fix the aggregation function of IndexData 2021-10-22 15:20:45 +08:00
Young
052aad7982 simplify signal parameter 2021-10-15 14:48:31 +00:00
Young
ac08468330 Make static prediction easier 2021-10-15 11:21:03 +00:00