eumap.parallel.utils.ProcessGeneratorLazy

ProcessGeneratorLazy(worker, args, max_workers=2, chunk=4, fixed_args=())[source]

Execute a function in parallel using a ProcessPoolExecutor [1].

Parameters

  • worker (Callable) – Function to execute in parallel.

  • args (Iterator[tuple]) – to separate job of the pool.

  • max_workers (int) – Number of CPU cores to use in the parallelization. By default all cores are used.

  • chunk (int) – Number of chunks to split the parallelization jobs.

  • fixed_args (tuple) – Constant arguments added in args in each execution of the worker function.

Returns

A generator with the return of all workers

Return type

Generator

References

[1] Python ProcessPoolExecutor class

Examples

>>> from eumap.parallel import ProcessGeneratorLazy
>>>
>>> def worker(i, msg):
>>>   print(f'{i}: {msg}')
>>>   return f'Worker {i} finished'
>>>
>>> args = iter([ (i,) for i in range(0,5)])
>>> fixed_args = ("I'm running in parallel", )
>>>
>>> for result in ProcessGeneratorLazy(worker, args, fixed_args=fixed_args):
>>>   print(result)