reliure.pipeline

inheritance diagrams

Class

class reliure.pipeline.Composable(func=None, name=None)

Bases: object

Basic composable element

Composable is abstract, you need to implemented the __call__() method

>>> e1 = Composable(lambda element: element**2, name="e1")
>>> e2 = Composable(lambda element: element + 10, name="e2")

Then Composable can be pipelined this way :

>>> chain = e1 | e2
>>> # so yo got:
>>> chain(2)
14
>>> # which is equivalent to :
>>> e2(e1(2))
14
>>> # not that by defaut the pipeline agregate the components name
>>> chain.name
'e1|e2'
>>> # however you can override it
>>> chain.name = "chain"
>>> chain.name
'chain'

It also possible to ‘map’ the composables >>> cmap = e1 & e2 >>> # so you got: >>> cmap(2) [4, 12]

__call__(*args, **kwargs)

__init__(func=None, name=None)

You can create a Composable from a simple function:

>>> def square(val, pow=2):
...     return val ** pow
>>> cfct = Composable(square)
>>> cfct.name
'square'
>>> cfct(2)
4
>>> cfct(3, 3)
27

name

Name of the optionable component

class reliure.pipeline.MapReduce(reduce, *composables)

Bases: reliure.pipeline.MapSeq

MapReduce implentation for components

One can pass a simple function:

>>> mapseq = MapReduce(sum, lambda x: x+1, lambda x: x+2, lambda x: x+3)
>>> mapseq(10)
36

Or implements sub class of MapReduce:

>>> class MyReduce(MapReduce):
...     def __init__(self, *composables):
...         super(MyReduce, self).__init__(None, *composables)
...     def reduce(self, array, *args, **kwargs):
...         return  list(args) + [sum(array)]
>>> mapreduce = MyReduce(lambda x: x+1, lambda x: x+2, lambda x: x+3)
>>> mapreduce(10)
[10, 36]

__call__(*args, **kwargs)

__init__(reduce, *composables)

reduce(array, *args, **kwargs)

class reliure.pipeline.MapSeq(*composants)

Bases: reliure.pipeline.OptionableSequence

Map implentation for components

>>> mapseq = MapSeq(lambda x: x+1, lambda x: x+2, lambda x: x+3)
>>> mapseq(10)
[11, 12, 13]
>>> sum(mapseq(10))
36

__call__(*args, **kwargs)

map(*args, **kwargs)

class reliure.pipeline.Optionable(name=None)

Bases: reliure.pipeline.Composable

Abstract class for an optionable component

__init__(name=None)

Parameters:name (str) – name of the component

add_option(opt_name, otype, hidden=False)

Add an option to the object

Parameters:

• opt_name (str) – option name
• otype (subclass of GenericType) – option type
• hidden (bool) – if True the option will be hidden

change_option_default(opt_name, default_val)

Change the default value of an option

Parameters:

• opt_name (str) – option name
• value – new default option value

static check(call_fct)

Decorator for optionable __call__ method It check the given option values

clear_option_value(opt_name)

Clear the stored option value (so the default will be used)

Parameters:opt_name (str) – option name

clear_options_values()

Clear all stored option values (so the defaults will be used)

force_option_value(opt_name, value)

force the (default) value of an option. The option is then no more listed by get_options().

Parameters:

• opt_name (str) – option name
• value – option value

get_option_default(opt_name)

Return the default value of a given option

Parameters:opt_name (str) – option name Returns:the default value of the option

get_option_value(opt_name)

Return the value of a given option

Parameters:opt_name (str) – option name Returns:the value of the option

get_options(hidden=False)

Parameters:hidden (bool) – whether to return hidden options Returns:dictionary of all options (with option’s information) Return type:dict

get_options_values(hidden=False)

return a dictionary of options values

Parameters:hidden (bool) – whether to return hidden options Returns:dictionary of all option values Return type:dict

get_ordered_options(hidden=False)

Parameters:hidden (bool) – whether to return hidden option Returns:ordered list of options pre-serialised (as_dict) Return type:list [opt_dict, ...]

has_option(opt_name)

Whether the component have a given option

option_is_hidden(opt_name)

Whether the given option is hidden

options

parse_options(option_values)

Set the options (with parsing) and returns a dict of all options values

print_options()

print description of the component options

set_option_value(opt_name, value, parse=False)

Set the value of one option.

# TODO/FIXME

• add force/hide argument
• add default argument
• remove methods force option value
• remove change_option_default

Parameters:

• opt_name (str) – option name
• value – the new value
• parse (bool) – if True the value is converted from string to the correct type

set_options_values(options, parse=False, strict=False)

Set the options from a dict of values (in string).

Parameters:

• option_values (dict) – the values of options (in format {“opt_name”: “new_value”})
• parse (bool) – whether to parse the given value
• strict (bool) – if True the given option_values dict should only contains existing options (no other key)

class reliure.pipeline.OptionableSequence(*composants)

Bases: reliure.pipeline.Optionable

Abstract class to manage a composant made of a sequence of callable (Optionable or not).

This object is an Optionable witch as all the options of it composants

__call__(*args, **kwargs)

__init__(*composants)

add_option(opt_name, otype, hidden=False)

call_item(item, *args, **kwargs)

change_option_default(opt_name, default_val)

clear_option_value(opt_name)

clear_options_values()

close()

Close all the neested components

force_option_value(opt_name, value)

get_option_default(opt_name)

get_option_value(opt_name)

get_options_values(hidden=False)

get_ordered_options(hidden=False)

has_option(opt_name)

option_is_hidden(opt_name)

options

set_option_value(opt_name, value, parse=False)

set_options_values(option_values, parse=True, strict=False)

class reliure.pipeline.Pipeline(*composables)

Bases: reliure.pipeline.OptionableSequence

A Pipeline is a sequence of function called sequentially.

It may be created explicitely:

>>> step1 = lambda x: x**2
>>> step2 = lambda x: x-1
>>> step3 = lambda x: min(x, 22)
>>> processing = Pipeline(step1, step2, step3)
>>> processing(4)
15
>>> processing(40)
22

Or it can be created implicitely with the pipe operator (__or__) if the first function is Composable:

>>> step1 = Composable(step1)
>>> processing = step1 | step2 | step3
>>> processing(3)
8
>>> processing(0)
-1

__call__(*args, **kwargs)

__init__(*composables)