liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-06 04:20:57 +08:00
Code Activity

Update docs and delete estimator

Browse Source
This commit is contained in:
Jactus
2020-11-26 19:40:41 +08:00
parent 0f8f9453bd
commit 2fd982a98f
14 changed files with 245 additions and 1878 deletions
Download Patch File Download Diff File Expand all files Collapse all files

119
docs/component/data.rst
View File

@@ -159,6 +159,9 @@ Data Loader
``Data Loader`` in ``Qlib`` is designed to load raw data from the original data source. It will be loaded and used in the ``Data Handler`` module.
QlibDataLoader
---------------
The ``QlibDataLoader`` class in ``Qlib`` is such an interface that allows users to load raw data from the data source.
Interface
@@ -166,33 +169,8 @@ Interface
Here are some interfaces of the ``QlibDataLoader`` class:
- `load(instruments, start_time=None, end_time=None)`
- This method loads the data as pd.DataFrame
- Parameters:
- `instruments` \: str or dict
it can either be the market name or the config file of instruments generated by InstrumentProvider.
- `start_time` \: str
start of the time range.
- `end_time` \: str
end of the time range.
- Returns:
- The data being loaded with type `pd.DataFrame`
- `load_group_df(instruments, exprs: list, names: list, start_time=None, end_time=None)`
- This method loads the dataframe for specific group.
- Parameters:
- `instruments` \: str or dict
it can either be the market name or the config file of instruments generated by InstrumentProvider.
- `exprs` \: list
the expressions to describe the content of the data.
- `names` \: list
the name of the data.
- `start_time` \: str
start of the time range.
- `end_time` \: str
end of the time range.
- Returns:
- The queried data in type `pd.DataFrame`.
.. autoclass:: qlib.data.dataset.loader.QlibDataLoader
:members: load, load_group_df
API
-----------
@@ -207,74 +185,24 @@ The ``Data Handler`` module in ``Qlib`` is designed to handler those common data
Users can use ``Data Handler`` in an automatic workflow by ``qrun``, refer to `Workflow: Workflow Management <workflow.html>`_ for more details.
Base Class & Interface
----------------------
DataHandlerLP
--------------
In addition to use ``Data Handler`` in an automatic workflow with ``qrun``, ``Data Handler`` can be used as an independent module, by which users can easily preprocess data (standardization, remove NaN, etc.) and build datasets.
In order to achieve so, ``Qlib`` provides a base class `qlib.data.dataset.DataHandlerLP <../reference/api.html#qlib.data.dataset.handler.DataHandlerLP>`_. The core idea of this class is that: we will have some leanable ``Processors`` which can learn the parameters of data processing. When new data comes in, these `trained` ``Processors`` can then infer on the new data and thus processing real-time data in an efficient way. More information about ``Processors`` will be listed in the next subsection.
Interface
----------------------
Here are some important interfaces that ``DataHandlerLP`` provides:
- `__init__(instruments=None, start_time=None, end_time=None, data_loader: Tuple[dict, str, DataLoader] = None, infer_processors=[], learn_processors=[], process_type=PTYPE_A, **kwargs)`
- Initialization of the class.
- Parameters:
- `infer_processors` \: list
- list of <description info> of processors to generate data for inference
- example of <description info>:
.. code-block::
1) classname & kwargs:
{
"class": "MinMaxNorm",
"kwargs": {
"fit_start_time": "20080101",
"fit_end_time": "20121231"
}
}
2) Only classname:
"DropnaFeature"
3) object instance of Processor
- `learn_processors` \: list
similar to infer_processors, but for generating data for learning models
- `process_type`: str
- PTYPE_I = 'independent'
- self._infer will processed by infer_processors
- self._learn will be processed by learn_processors
- PTYPE_A = 'append'
- self._infer will processed by infer_processors
- self._learn will be processed by infer_processors + learn_processors
- (e.g. self._infer processed by learn_processors )
- `fetch(selector: Union[pd.Timestamp, slice, str] = slice(None, None), level: Union[str, int] = "datetime", col_set=DataHandler.CS_ALL, data_key: str = DK_I)`
- This method fetches data from underlying data source
- Parameters:
- `selector` \: Union[pd.Timestamp, slice, str]
describe how to select data by index.
- `level` \: Union[str, int]
which index level to select the data.
- `col_set` \: str
select a set of meaningful columns.(e.g. features, columns).
- `data_key` \: str
The data to fetch: DK_*.
- Returns:
- The retrieved results in the type: `pd.DataFrame`.
- `get_cols(col_set=DataHandler.CS_ALL, data_key: str = DK_I)`
- This method gets the column names.
- Parameters:
- `col_set` \: str
select a set of meaningful columns.(e.g. features, columns).
- `data_key` \: str
the data to fetch: DK_*.
- Returns:
- A list of column names.
.. autoclass:: qlib.data.dataset.handler.DataHandlerLP
:members: __init__, fetch, get_cols
If users want to load features and labels by config, users can inherit ``qlib.data.dataset.handler.ConfigDataHandler``, ``Qlib`` also provides some preprocess method in this subclass.
If users want to use qlib data, `QLibDataHandler` is recommended. Users can inherit their custom class from `QLibDataHandler`, which is also a subclass of `ConfigDataHandler`.
@@ -353,23 +281,8 @@ The motivation of this module is that we want to maximize the flexibility of of
The ``DatasetH`` class is the `dataset` with `Data Handler`. Here is the most important interface of the class:
- `prepare(segments: Union[List[str], Tuple[str], str, slice], col_set=DataHandler.CS_ALL, data_key=DataHandlerLP.DK_I, **kwargs)`
- This method prepares the data for learning and inference.
- Parameters:
- `segments` \: Union[List[str], Tuple[str], str, slice]
Describe the scope of the data to be prepared
Here are some examples:
- 'train'
- ['train', 'valid']
- `col_set` \: str
The col_set will be passed to self._handler when fetching data.
- `data_key` \: str
The data to fetch: DK_*
Default is DK_I, which indicate fetching data for **inference**.
.. autoclass:: qlib.data.dataset.__init__.DatasetH
:members:
API
---------