Future Developments
Soon to be released¶
A couple of features will be released soon:
- handle problems when calling
ExecNodes
wrongly. - (for example when using args as parameters but only *kwargs are provided).
- Calling
ExecNodes
must be similar to calling the original function (must imitate the same signature otherwise raise the correct exception). - support mixing ExecNodes and non
ExecNodes
functions. - test the case where
ExecNodes
are stored in a list and then passed via * operator. - an example of running a set of calculations with the
DAG
and without theDAG
. - we can show that using tawazi is transparent and one just has to remove the decorators. Then everything is back to normal.
- decide whether to identify the
ExecNode
by a Hashable ID or by its own Python ID. This is breaking change and must change to 0.2.1. - support constants by resolving the error in the tests.
- Remove the global object and maybe replace it with an attribute to the creating function.
- improve the graph dependency rendering on the console (using graphviz).
- automatically generate release on new tag https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes#configuring-automatically-generated-release-notes
- use opnssf service to evaluate code best practices https://bestpractices.coreinfrastructure.org/fr/projects/1486
Features to be discussed¶
- support multiprocessing.
- simulation of the execution using a
DAG
stored ledger. - Disallow execution in parallel of some threads in parallel with some other threads.
- maybe by making a group of threads that are CPU bound and a group of threads that are IO bound ?
- Remove dependency on networkx to make
tawazi
a standalone package. - save the results of the calculation in pickled format in case an error is encountered ? or just at the end of the run
- re-run the same calculations of the graph but take the input from the presaved pickle files instead
- put documentation about different cases where it is advantageous to use it
- in methods not only in functions
- in a gunicorn application
- for getting information from multiple resources
- pretty-print the graph deps on the console:
- remove the argument_name behavior using a more intelligent way
- or instead of doing this, you can make this optional and then prefer to infer the argument_name more intelligently using the place in the arguments where the ExecNode was passed!