luigi_tools.util¶

This module provides some functions to work with luigi tasks.

Functions

`apply_over_inputs`(task, func)	Apply the given function to all inputs of a `luigi.Task`.
`apply_over_luigi_iterable`(luigi_iterable, func)	Apply the given function to a luigi iterable (task.input() or task.output()).
`apply_over_outputs`(task, func)	Apply the given function to all outputs of a `luigi.Task`.
`apply_over_required`(task, func)	Apply the given function to all required tasks of a `luigi.WrapperTask`.
`configparser_to_dict`(luigi_config)	Transform a ConfigParser object to a dict.
`export_dependency_graph`(task[, filepath, ...])	Build and export a dependency graph.
`get_dependency_graph`(task[, allow_orphans])	Compute dependency graph of a given task.
`graphviz_dependency_graph`(g[, graph_attrs, ...])	Create a GraphViz dependency graph.
`luigi_config_to_dict`([filename])	Load a luigi config file to a dict.
`recursive_check`(task[, attr])	Check if a task or any of its recursive dependencies has a given attribute set to True.
`register_templates`([directory, name, ...])	Add INI templates to the config file list processed by luigi.
`render_dependency_graph`(graph, filepath, ...)	Render a dependency graph using GraphViz.
`target_remove`(target, args, *kwargs)	Remove a given target by calling its 'exists()' and 'remove()' methods.

Classes

set_luigi_config([params, configfile])

Context manager to set current luigi config.

Exceptions

WorkflowError

Exception raised when the workflow is not consistent.

exception luigi_tools.util.WorkflowError¶

Bases: Exception

Exception raised when the workflow is not consistent.

luigi_tools.util.apply_over_inputs(task, func)¶

Apply the given function to all inputs of a luigi.Task.

The given function should accept the following arguments:

task_output: the output(s) of the required task(s) of the given task
key=None: the key when the iterable is a dictionary

luigi_tools.util.apply_over_luigi_iterable(luigi_iterable, func)¶: Apply the given function to a luigi iterable (task.input() or task.output()).

luigi_tools.util.apply_over_outputs(task, func)¶

Apply the given function to all outputs of a luigi.Task.

The given function should accept the following arguments:

target: the output target(s) of the given task
key=None: the key when the iterable is a dictionary

luigi_tools.util.apply_over_required(task, func)¶

Apply the given function to all required tasks of a luigi.WrapperTask.

The given function should accept the following arguments:

target: the output target(s) of the given task
key=None: the key when the iterable is a dictionary

luigi_tools.util.configparser_to_dict(luigi_config)¶: Transform a ConfigParser object to a dict.

luigi_tools.util.export_dependency_graph(task, filepath='dependency_graph.png', allow_orphans=False, graph_attrs=None, node_attrs=None, edge_attrs=None, root_attrs=None, task_names=None, node_kwargs=None, edge_kwargs=None, graphviz_class=None, **render_kwargs)¶

Build and export a dependency graph.

Parameters:

task (luigi.Task) – the task from which the dependency graph is computed.
filepath (str) – The path to the rendered file. If the format kwarg is not given, this path should contain an extension. Otherwise, the extension is guessed from the format.
allow_orphans (bool) – If set to True, orphan nodes are returned with a None child.
graph_attrs (dict) – The graph attributes that will override the defaults.
node_attrs (dict) – The node attributes that will override the defaults.
edge_attrs (dict) – The edge attributes that will override the defaults.
root_attrs (dict) – The specific node attributes only used for the root that will override the defaults.
task_names (dict) – A dictionary with luigi.Task objects as keys and custom names as values.
node_kwargs (dict) – A dictionary with luigi.Task objects as keys and the kwargs given to graphviz.Digraph.node() as values.
edge_kwargs (dict) – A dictionary with luigi.Task objects as keys and the kwargs given to graphviz.Digraph.edge() as values.
graphviz_class (graphviz.Graph or graphviz.Digraph) – The class used to store the graph.

Keyword Arguments:

render_kwargs – The keyword arguments are passed to the graphviz.Digraph.render() function.

luigi_tools.util.get_dependency_graph(task, allow_orphans=False)¶

Compute dependency graph of a given task.

Parameters:

task (luigi.Task) – the task from which the dependency graph is computed.
allow_orphans (bool) – If set to True, orphan nodes are returned with a None child.

Returns:

A list of (parent, child) tuples.

Return type:

list(luigi.Task)

luigi_tools.util.graphviz_dependency_graph(g, graph_attrs=None, node_attrs=None, edge_attrs=None, root_attrs=None, task_names=None, node_kwargs=None, edge_kwargs=None, graphviz_class=None)¶

Create a GraphViz dependency graph.

Parameters:

g (list(tuple(luigi.Task))) – A list of tuples of luigi.Task objects, usually created with luigi_tools.util.get_dependency_graph().
graph_attrs (dict) – The graph attributes that will override the defaults.
node_attrs (dict) – The node attributes that will override the defaults.
edge_attrs (dict) – The edge attributes that will override the defaults.
root_attrs (dict) – The specific node attributes only used for the root that will override the defaults.
task_names (dict) – A dictionary with luigi.Task objects as keys and custom names as values.
node_kwargs (dict) – A dictionary with luigi.Task objects as keys and the kwargs given to graphviz.Digraph.node() as values.
edge_kwargs (dict) – A dictionary with luigi.Task objects as keys and the kwargs given to graphviz.Digraph.edge() as values.
graphviz_class (graphviz.Graph or graphviz.Digraph) – The class used to store the graph.

Returns:

The Graph or Digraph created.

luigi_tools.util.luigi_config_to_dict(filename='luigi.cfg')¶: Load a luigi config file to a dict.

luigi_tools.util.recursive_check(task, attr='rerun')¶: Check if a task or any of its recursive dependencies has a given attribute set to True.

luigi_tools.util.register_templates(directory=None, name=None, hierarchy_end=True)¶

Add INI templates to the config file list processed by luigi.

This function should be used before the luigi.Task are processed, so it should usually be used in the __init__.py of your package.

In order to use a template, a Template section entry must be added to the luigi.cfg of the project. This section should contain at least one entry with the name of the template to use, and, optionally, another entry with the path to the directory containing the templates, as the following example:

[Template]
name = template_name
directory = /path/to/the/template/directory

If the directory or the name entries are given in the luigi.cfg file, they override the arguments given to this function.

A template file should be similar to a luigi.cfg file with only the entries for which specific default values should be defined. The file should named according to the template name. For example, the template named template_1 should be a INI file located in the given directory and named template_1.cfg.

Parameters:

directory (str) – Path to the directory containing the template files.
name (str) – The name of the template to use.
hierarchy_end (bool) – If set to False, the luigi.cfg entry is not added after the template entry, so it is possible to combine several templates before processing the luigi.cfg file.

luigi_tools.util.render_dependency_graph(graph, filepath, **kwargs)¶

Render a dependency graph using GraphViz.

Parameters:

graph (graphviz.Graph or graphviz.Digraph) – The graph to render, usually create with luigi_tools.util.graphviz_dependency_graph().
filepath (str) – The path to the rendered file. If the format kwarg is not given, this path should contain an extension. Otherwise, the extension is guessed from the format.
kwargs – The kwargs are passed to the graphviz.Digraph.render() function.

class luigi_tools.util.set_luigi_config(params=None, configfile=None)¶

Bases: object

Context manager to set current luigi config.

Parameters:

params (dict) – The parameters to load into the luigi configuration.
configfile (str) – Path to the temporary config file (luigi.cfg by default).

export_config(filepath)¶: Export the configuration to the configuration file.

get_config()¶: Convert the parameter dict to a configparser.ConfigParser object.

luigi_tools.util.target_remove(target, *args, **kwargs)¶: Remove a given target by calling its ‘exists()’ and ‘remove()’ methods.