luigi_tools.target

This module provides some specific luigi targets.

Classes

OutputLocalTarget(*args[, prefix, create_parent])

A target that adds a prefix before the given path.

class luigi_tools.target.OutputLocalTarget(*args, prefix=None, create_parent=True, **kwargs)

Bases: LocalTarget

A target that adds a prefix before the given path.

Parameters:

• prefix (str) – The prefix to use. If not given, the current working directory is taken.

• create_parent (bool) – If set to True, the parent directory is automatically created.

This class can be subclassed to easily create an output directory tree.

Note

If an absolute path is given to the target, the prefix is ignored.

Usage:

class PathConfig(luigi.Config):
    '''Paths config.'''
    result_path = luigi.Parameter()

class Sub1OutputLocalTarget(OutputLocalTarget):
    '''Specific target for first category outputs.'''
    __prefix = "sub_path_1"

class Sub2OutputLocalTarget(Sub1OutputLocalTarget):
    '''Specific target for second category outputs.'''
    __prefix = "sub_path_2"

OutputLocalTarget.set_default_prefix(PathConfig().result_path)

class TaskA(luigi.Task):
    def run(self):
        # do something
        # and write output
        write_output(self.output().path)

    def output(self):
        return Sub1OutputLocalTarget("file1.dat")

class TaskB(luigi.Task):
    def run(self):
        # do something
        # and write outputs
        f1, f2, f3 = self.output()
        write_output1(f1.path)
        write_output2(f2.path)
        write_output3(f3.path)

    def output(self):
        return [
            Sub1OutputLocalTarget("file2.dat"),
            Sub2OutputLocalTarget("file1.dat"),
            Sub2OutputLocalTarget("file2.dat"),
        ]

Running this luigi workflow creates the following output directory tree:

└── result_path
    └── sub_path_1
        ├── file1.dat
        └── file2.dat
        └── sub_path_2
            ├── file1.dat
            └── file2.dat

classmethod get_default_prefix()

Return the default prefix.

get_prefix()

Return the default prefix.

mkdir(is_dir=False, mode=511, parents=True, exist_ok=True)

Create the directory of this path.

Parameters:

• is_dir (bool) – if set to True, the current path is create, otherwise only the parent directory is create.

• mode (int) – numeric mode used to create the directory with given permissions (unix only).

• parents (bool) – if set to True, the parents are also created if they are missing.

• exist_ok (bool) – if set to True, do not raise an exception if the directory already exists.

property path

The path stored in this target.

property pathlib_path

The path stored in this target returned as a pathlib.Path object.

classmethod set_default_prefix(prefix)

Set the default prefix to the class.

Warning

This method is not thread-safe and should not be used inside a luigi.Task.

set_prefix(prefix)

Set the prefix of the current instance.

classmethod super_prefix()

Build a prefix from the default prefixes of the parent classes.