eumap.parallel.utils.job

job(worker, worker_args, n_jobs=- 1, joblib_args={})

Execute a function in parallel using joblib [1].

Parameters

• worker (Callable) – Function to execute in parallel.

• worker_args (Iterator[tuple]) – Argument iterator where each element is send to separate job.

• joblib_args (set) – Number of CPU cores to use in the parallelization. By default all cores are used.

• joblib_args – Joblib argumets to send to Parallel class [1].

Returns

A generator with the return of all workers

Return type

Generator

References

[1] joblib.Parallel class

Examples

>>> from eumap import parallel
>>>
>>> def worker(i, msg):
>>>   print(f'{i}: {msg}')
>>>   return f'Worker {i} finished'
>>>
>>> msg = ("I'm running in parallel", )
>>> args = iter([ (i,msg) for i in range(0,5)])
>>>
>>> for result in parallel.job(worker, args, n_jobs=-1, joblib_args={'backend': 'threading'}):
>>>   print(result)