`reliure.engine`¶

class reliure.engine.BasicPlayMeta(component)¶

Bases: object

Object to store and manage meta data for one component exec

Here is a typical usage :

>>> import time
>>> comp = Composable(name="TheComp", func=lambda x: x)
>>> # create the meta result before to use the component
>>> meta = BasicPlayMeta(comp)
>>> # imagine some input and options for the component
>>> args, kwargs = [12], {}
>>> # store these data:
>>> meta.run_with(args, kwargs)
>>> # run the component
>>> start = time.time()     # starting time
>>> try:
...     output = comp(*args, **kwargs)
... except Exception as error:
...     # store the exception if any
...     meta.add_error(error)
...     # one can raise a custom error (or not)
...     #raise RuntimeError()
... finally:
...     # this will always be executed (even if the exception is not catched)
...     meta.time = time.time() - start
...     # for testing purpose we put a fixed time
...     meta.time = 9.2e-5
>>> # one can get a pre-serialization of the collected meta data
>>> from pprint import pprint
>>> pprint(meta.as_dict())
{'errors': [], 'name': 'TheComp', 'time': 9.2e-05, 'warnings': []}

__init__(component)¶

add_error(error)¶

Register an error that occurs during component running

>>> comp = Composable(name="TheComp", func=lambda x: x)
>>> meta = BasicPlayMeta(comp)
>>> try:
...     output = 1/0
... except Exception as error:
...     # store the exception if any
...     meta.add_error(error)
>>> from pprint import pprint
>>> pprint(meta.as_dict())                           
{'errors': ['...division ...by zero'],
 'name': 'TheComp',
 'time': 0.0,
 'warnings': []}

as_dict()¶: Pre-serialisation of the meta data

errors¶

has_error¶: wether any error happened

has_warning¶: wether there where a warning during play

name¶: Name of the component

run_with(inputs, options)¶: Store the run parameters (inputs and options)

time¶

Execution time (walltime)

>>> comp = Composable(name="TheComp", func=lambda x: x)
>>> meta = BasicPlayMeta(comp)
>>> meta.time = 453.6
>>> meta.time
453.6

warnings¶

class reliure.engine.Block(name)¶

Bases: object

A block is a processing step realised by one component.

A component is a callable object that has a name attribute, often it is also a reliure.Optionable object or a pipeline beeing a reliure.Composable.

Block object provides methods to discover and parse components options (if any).

Warning

You should not have to use a Block directly but always throught a Engine.

__init__(name)¶

Intialise a block. This should be done only from the Engine.

Parameters:	name (str) – name of the Block

all_outputs()¶: Returns a set of outputs name

append(component, default=False)¶

Add one component to the block

Parameters:	default (bool) – if true this component will be use by default

as_dict()¶: returns a dictionary representation of the block and of all component options

clear_selections()¶: Reset the current selections and reset option values to default for all components

Warning

This method also reset the components options values to the defaults values.

component_names()¶

returns the list of component names.

Component names will have the same order than components

configure(config)¶

Configure the block from an (horible) configuration dictionary (or list) this data are coming from a json client request and has to be parsed. It takes the default value if missing (for component selection and options).

Parameters:	config (dict or list) – component to use and the associated options

config format

{
    'name': name_of_the_comp_to_use,
    'options': {
        name: value,
        name: va...
    }
}

or for multiple selection

[
    {
        'name': name_of_the_comp_to_use,
        'options': {
            name: value,
            name: va...
        }
    },
    {...}
]

Warning

values of options in this dictionnary are strings

defaults¶: component default component

Note

default components is just an indication for user and the views, except if the Block is required. If required then default is selected if nothing explisitely selected.

name¶: Name of the optionable component

needed_inputs()¶: Return a list of (needed) inputs names

play(*inputs, **named_inputs)¶

Run the selected components of the block. The selected components are run with the already setted options.

Warning

Defaut ‘multiple’ behavior is a pipeline !

Parameters:	*inputs – arguments (i.e. inputs) to give to the components

reset()¶: Removes all the components of the block

select(comp_name, options=None)¶

Select the components that will by played (with given options).

options will be passed to Optionable.parse_options() if the component is a subclass of Optionable.

Warning

this function also setup the options (if given) of the selected component. Use clear_selections() to restore both selection and component’s options.

This method may be call at play ‘time’, before to call play() to run all selected components.

Parameters:	name – name of the component to select options (dict) – options to set to the components

selected()¶

returns the list of selected component names.

if no component selected return the one marked as default. If the block is required and no component where indicated as default, then the first component is selected.

set(*components)¶

Set the possible components of the block

Parameters:	components – components to append Optionables or Composables

setup(in_name=None, out_name=None, required=None, hidden=None, multiple=None, defaults=None)¶

Set the options of the block. Only the not None given options are set

Note

a block may have multiple inputs but have only one output

Parameters:

in_name (str or list of str) – name(s) of the block input data
out_name (str) – name of the block output data
required (bool) – whether the block will be required or not
hidden (bool) – whether the block will be hidden to the user or not
multiple (bool) – if True more than one component may be selected/ run)
defaults (list of str, or str) – names of the selected components

validate()¶: check that the block can be run

class reliure.engine.Engine(*names)¶

Bases: object

The Reliure engine.

DEFAULT_IN_NAME = 'input'¶

__init__(*names)¶

Create the engine

Parameters:	names – names of the engine blocks

all_outputs()¶

Returns a list of all engine possible outputs (note that inputs are also possible inputs)

>>> engine = Engine("op1", "op2")
>>> engine.op1.setup(in_name="in", out_name="middle", required=False)
>>> engine.op2.setup(in_name="middle", out_name="out")
>>> sorted(list(engine.all_outputs()))
['in', 'middle', 'out']

More complex example:

>>> engine = Engine("op1", "op2")
>>> engine.op1.setup(in_name="in", out_name="middle")
>>> engine.op2.setup(in_name=["middle", "in2"], out_name="out")
>>> sorted(list(engine.all_outputs()))
['in', 'in2', 'middle', 'out']

Note that by default the needed input is ‘input’:

>>> engine = Engine("op1", "op2")
>>> engine.op1.append(lambda x:x+2)
>>> engine.op2.append(lambda x:x*2)
>>> sorted(list(engine.all_outputs()))
['input', 'op1', 'op2']

as_dict()¶: dict repr of the components

configure(config)¶

Configure all the blocks from an (horible) configuration dictionary this data are coming from a json client request and has to be parsed. It takes the default value if missing (for component selection and options).

Parameters:	config (dict) – dictionary that give the component to use for each step and the associated options

config format

 {
     block_name: [{
         'name': name_of_the_comp_to_use,
         'options': {
                 name: value,
                 name: va...
             }
         },
         {...}
     ]
}

Warning

values of options in this dictionnary are strings

in_name¶

Give the input name of the first block.

If this first block is not required or if other block need some inputs then you beter have to look at needed_inputs().

names()¶: Returns the list of block names

needed_inputs()¶

List all the needed inputs of a configured engine

>>> engine = Engine("op1", "op2")
>>> engine.op1.setup(in_name="in", out_name="middle", required=False)
>>> engine.op2.setup(in_name="middle", out_name="out")
>>> engine.op1.append(lambda x:x+2)
>>> engine.op2.append(lambda x:x*2)
>>> engine.op1.select('<lambda>')
>>> list(engine.needed_inputs())
['in']

But now if we unactivate the first component:

>>> engine.op1.clear_selections()
>>> list(engine.needed_inputs())
['middle']

More complex example:

>>> engine = Engine("op1", "op2")
>>> engine.op1.setup(in_name="in", out_name="middle")
>>> engine.op2.setup(in_name=["middle", "in2"], out_name="out")
>>> engine.op1.append(lambda x:x+2)
>>> engine.op2.append(lambda x, y:x*y)
>>> engine.needed_inputs() == {'in', 'in2'}
True

Note that by default the needed input is ‘input’:

>>> engine = Engine("op1", "op2")
>>> engine.op1.append(lambda x:x+2)
>>> engine.op2.append(lambda x:x*2)
>>> list(engine.needed_inputs())
['input']

play(*inputs, **named_inputs)¶

Run the engine (that should have been configured first)

if the inputs are given without name it should be the inputs of the first block, ig named_inputs are used it may be the inputs of any block.

Note

Either inputs or named_inputs should be provided, not both

Parameters:	inputs – the data to give as input to the first block named_inputs – named input data should match with `needed_inputs()` result.

requires(*names)¶

Declare what block will be used in this engine.

It should be call before adding or setting any component. Blocks order will be preserved for runnning task.

set(name, *components, **parameters)¶

Set available components and the options of one block.

Parameters:	name – block name components – the components (see `Block.set()`) parameters – block configuration (see `Block.setup()`)

for example :

>>> engine = Engine("op1")
>>> engine.set("op1", Composable(), required=True, in_name="query", out_name="holygrail")

validate(inputs=None)¶

Check that the blocks configuration is ok

Parameters:	inputs (list of str) – the names of the play inputs

class reliure.engine.PlayMeta(name)¶

Bases: reliure.engine.BasicPlayMeta

Object to store and manage meta data for a set of component or block play

>>> gres = PlayMeta("operation")
>>> res_plus = BasicPlayMeta(Composable(name="plus"))
>>> res_plus.time = 1.6
>>> res_moins = BasicPlayMeta(Composable(name="moins"))
>>> res_moins.time = 5.88
>>> gres.append(res_plus)
>>> gres.append(res_moins)
>>> from pprint import pprint
>>> pprint(gres.as_dict())
{'details': [{'errors': [], 'name': 'plus', 'time': 1.6, 'warnings': []},
             {'errors': [], 'name': 'moins', 'time': 5.88, 'warnings': []}],
 'errors': [],
 'name': 'operation:[plus, moins]',
 'time': 7.48,
 'warnings': []}

__init__(name)¶

add_error(error)¶: It is not possible to add an error here, you sould add it on a BasicPlayMeta

append(meta)¶: Add a BasicPlayMeta

as_dict()¶: Pre-serialisation of the meta data

errors¶

get all the errors

>>> gres = PlayMeta("operation")
>>> res_plus = BasicPlayMeta(Composable(name="plus"))
>>> gres.append(res_plus)
>>> res_plus.add_error(ValueError("invalid data"))
>>> res_moins = BasicPlayMeta(Composable(name="moins"))
>>> gres.append(res_moins)
>>> res_plus.add_error(RuntimeError("server not anwsering"))
>>> gres.errors
[ValueError('invalid data',), RuntimeError('server not anwsering',)]

name¶

Compute a name according to sub meta results names

>>> gres = PlayMeta("operation")
>>> res_plus = BasicPlayMeta(Composable(name="plus"))
>>> res_moins = BasicPlayMeta(Composable(name="moins"))
>>> gres.append(res_plus)
>>> gres.append(res_moins)
>>> gres.name
'operation:[plus, moins]'

time¶

Compute the total time (walltime)

>>> gres = PlayMeta("operation")
>>> res_plus = BasicPlayMeta(Composable(name="plus"))
>>> res_plus.time = 1.6
>>> res_moins = BasicPlayMeta(Composable(name="moins"))
>>> res_moins.time = 5.88
>>> gres.append(res_plus)
>>> gres.append(res_moins)
>>> gres.time
7.48

warnings¶

reliure.engine¶

`reliure.engine`¶