liuyu/qlib
1
0
Fork 0
mirror of https://github.com/microsoft/qlib.git synced 2026-07-07 13:00:58 +08:00
Code Activity

update docstring and document

Browse Source
This commit is contained in:
lzh222333
2021-03-15 03:50:43 +00:00
parent 9d84d389ab
commit 646d899f8d
7 changed files with 106 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
docs/advanced/task_management.rst
View File

@@ -9,50 +9,52 @@ Task Management
Introduction
=============
The `Workflow <../component/introduction.html>`_ part introduce how to run research workflow in a loosely-coupled way. But it can only execute one ``task`` when you use ``qrun``. To automatically generate and execute different tasks, Task Management module provide a whole process including `Task Generating`_, `Task Storing`_, `Task Running`_ and `Task Collecting`_.
With this module, users can run their ``task`` automatically at different periods, in different losses or even by different models.
The `Workflow <../component/introduction.html>`_ part introduces how to run research workflow in a loosely-coupled way. But it can only execute one ``task`` when you use ``qrun``.
To automatically generate and execute different tasks, ``Task Management`` provides a whole process including `Task Generating`_, `Task Storing`_, `Task Running`_ and `Task Collecting`_.
With this module, users can run their ``task`` automatically at different periods, in different losses, or even by different models.
An example of the entire process is shown `here <>`_.
An example of the entire process is shown `here <https://github.com/microsoft/qlib/tree/main/examples/taskmanager/task_manager_rolling.py>`_.
Task Generating
===============
A ``task`` consists of `Model`, `Dataset`, `Record` or anything added by users.
The specific task template can be viewed in
The specific task template(/definition/config) can be viewed in
`Task Section <../component/workflow.html#task-section>`_.
Even though the task template is fixed, Users can use ``TaskGen`` to generate different ``task`` by task template.
Even though the task template is fixed, users can customize their ``TaskGen`` to generate different ``task`` by task template.
Here is the base class of TaskGen:
Here is the base class of ``TaskGen``:
.. autoclass:: qlib.workflow.task.gen.TaskGen
:members:
``Qlib`` provider a class `RollingGen<https://github.com/microsoft/qlib/tree/main/qlib/workflow/task/gen.py>`_ to generate a list of ``task`` of dataset in different date segments.
This allows users to verify the effect of data from different periods on the model in one experiment.
``Qlib`` provider a class `RollingGen <https://github.com/microsoft/qlib/tree/main/qlib/workflow/task/gen.py>`_ to generate a list of ``task`` of the dataset in different date segments.
This class allows users to verify the effect of data from different periods on the model in one experiment.
Task Storing
===============
In order to achieve higher efficiency and the possibility of cluster operation, ``Task Manager`` will store all tasks in `MongoDB <https://www.mongodb.com/>`_.
To achieve higher efficiency and the possibility of cluster operation, ``Task Manager`` will store all tasks in `MongoDB <https://www.mongodb.com/>`_.
Users **MUST** finished the configuration of `MongoDB <https://www.mongodb.com/>`_ when using this module.
Users need to provide the url and database of ``task`` storing like this.
Users need to provide the URL and database name of ``task`` storing like this.
.. code-block:: python
from qlib.config import C
C["mongo"] = {
"task_url" : "mongodb://localhost:27017/", # maybe you need to change it to your url
"task_db_name" : "rolling_db" # you can custom database name
"task_url" : "mongodb://localhost:27017/", # your MongoDB url
"task_db_name" : "rolling_db" # database name
}
The CRUD methods of ``task`` can be found in TaskManager. More methods can be seen in the `Github<https://github.com/microsoft/qlib/tree/main/qlib/workflow/task/manage.py>`_.
The CRUD methods of ``task`` can be found in TaskManager.
More methods can be seen in the `Github <https://github.com/microsoft/qlib/tree/main/qlib/workflow/task/manage.py>`_.
.. autoclass:: qlib.workflow.task.manage.TaskManager
:members:
Task Running
===============
After generating and storing those ``task``, it's time to run the ``task`` in the *WAITING* status.
``qlib`` provide a method to run those ``task`` in task pool, however users can also customize how tasks are executed.
After generating and storing those ``task``, it's time to run the ``task`` which are in the *WAITING* status.
``Qlib`` provides a method called ``run_task`` to run those ``task`` in task pool, however, users can also customize how tasks are executed.
An easy way to get the ``task_func`` is using ``qlib.model.trainer.task_train`` directly.
It will run the whole workflow defined by ``task``, which includes *Model*, *Dataset*, *Record*.
@@ -60,8 +62,12 @@ It will run the whole workflow defined by ``task``, which includes *Model*, *Dat
Task Collecting
===============
To see the results of ``task`` after running, ``Qlib`` provide a task collector to collect the tasks by filter condition (optional).
The collector will return a dict of filtered key (users defined by task config) and value (predict scores from ``pred.pkl``).
To see the results of ``task`` after running or to update something, ``Qlib`` provides a ``TaskCollector`` to collect the tasks by filter condition (optional).
Here are some methods in this class.
.. autoclass:: qlib.workflow.task.collect.TaskCollector
:members:
:members:
``Qlib`` provides a concrete `example <https://github.com/microsoft/qlib/tree/main/examples/taskmanager/task_manager_rolling_with_updating.py>`_, including a whole process of `Task Generating`_ (using `RollingGen <https://github.com/microsoft/qlib/tree/main/qlib/workflow/task/gen.py>`_), `Task Storing`_, `Task Running`_ and `Task Collecting`_.
Besides, the `example <https://github.com/microsoft/qlib/tree/main/examples/taskmanager/task_manager_rolling_with_updating.py>`_ uses a ``ModelUpdater`` inherited from ``TaskCollector``, which can update the inferences and retrain the model if it is out of date.
Actually, the model updating can be viewed as a subset of ``Online Serving``.

29
docs/reference/api.rst
View File

@@ -155,6 +155,35 @@ Record Template
:members:
Task Management
====================
RollingGen
--------------------
.. autoclass:: qlib.workflow.task.gen.RollingGen
:members:
TaskManager
--------------------
.. autoclass:: qlib.workflow.task.manage.TaskManager
:members:
TaskCollector
--------------------
.. autoclass:: qlib.workflow.task.collect.TaskCollector
:members:
ModelUpdater
--------------------
.. autoclass:: qlib.workflow.task.update.ModelUpdater
:members:
TimeAdjuster
--------------------
.. autoclass:: qlib.workflow.task.utils.TimeAdjuster
:members:
Utils
====================