eumap.parallel.utils.TilingProcessing

class TilingProcessing(tiling_system_fn='http://s3.eu-central-1.wasabisys.com/eumap/tiling_system_30km.gpkg', base_raster_fn='http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_ndvi_landsat.glad.ard_p50_30m_0..0cm_201903_eumap_epsg3035_v1.0.tif', verbose=False, epsg_checking=True)[source]

Bases: object

Execute a processing function in parallel considering a tiling system and a base raster. It creates a rasterio window object for each tile according to the pixel size of the specified base.

Parameters

  • tiling_system_fn – Vector file path with the tiles to read.

  • base_raster_fn – Raster file path used the retrieve the affine transformation for windows.

  • verbose (bool) – Use True to print information about read tiles and the base raster.

Methods

generate_tiles

Generate a custom tiling system based on a regular grid.

process_all

Process in parallel all of tile using the specified function args.

process_multiple

Process in parallel a list of tile using the specified function args.

process_one

Process a single tile using the specified function args.
static generate_tiles(tile_size, extent, crs, raster_layer_fn=None)[source]

Generate a custom tiling system based on a regular grid.

Parameters

  • tile_size (int) – Single value used to define the width and height of a individual tile. It assumes the same unit of crs (degree for geographic coordinate systems and meter for projected coordinate systems). Tiles outside of the image are clipped to fit in the informed extent.

  • extent (tuple) – Extent definition considering minx, miny, maxx, maxy according to the crs argument.

  • crs (str) – Coordinate reference system for the tile geometries. Can be anything accepted by pyproj.CRS.from_user_input(), such as an authority string (EPSG:4326) or a WKT/proj4 string.

  • raster_layer_fn (Optional[str]) – If provided, for each tile the min, max and mode values are calculated considering the raster pixels inside the tile. It assumes the same crs for the raster layer and tiles.

Returns

Tiling system with follow columns: tile_id, minx, miny, maxx, maxy and geometry. The additional columns raster_min, raster_mode_value, raster_mode_count and raster_max are returned when a raster layer is provided.

Return type

geopandas.GeoDataFrame

Examples

>>> eumap_extent = (900000, 930010, 6540000, 5460010)
>>> tiling_system = TilingProcessing.generate_tiles(30000, eumap_extent, 'epsg:3035')
>>> tiling_system.to_file(tiling_system_fn,  driver="GPKG")
process_all(func, *args, max_workers=2, use_threads=True, progress_bar=False)[source]

Process in parallel all of tile using the specified function args.

Parameters

  • func (Callable) – A function with at least the arguments idx, tile, window.

  • args (any) – Additional arguments to send to the function.

  • max_workers (int) – Number of CPU cores to use in the parallelization. By default all cores are used.

  • use_threads (bool) – If True the parallel processing uses ThreadGeneratorLazy, otherwise it uses ProcessGeneratorLazy.

  • progress_bar (bool) – If True the parallel processing uses pqdm [1] presenting a progress bar, ignoring the use_threads.

References

[1] Parallel TQDM

Examples

>>> from eumap.parallel import TilingProcessing
>>> from eumap.raster import read_rasters
>>>
>>> def run(idx, tile, window, msg):
>>>     print(f'Tile {idx} => {msg}')
>>>
>>> tiling= TilingProcessing(verbose=True)
>>> msg = "Let's crunch some data."
>>> result = tiling.process_all(run)
process_multiple(idx_list, func, *args, max_workers=2, use_threads=True, progress_bar=False)[source]

Process in parallel a list of tile using the specified function args.

Parameters

  • idx – The tile ids to process. This idx is generated for all the tiles in a sequence starting from 0.

  • func (Callable) – A function with at least the arguments idx, tile, window.

  • args (any) – Additional arguments to send to the function.

  • max_workers (int) – Number of CPU cores to use in the parallelization. By default all cores are used.

  • use_threads (bool) – If True the parallel processing uses ThreadGeneratorLazy, otherwise it uses ProcessGeneratorLazy.

  • progress_bar (bool) – If True the parallel processing uses pqdm [1] presenting a progress bar and ignoring the use_threads.

References

[1] Parallel TQDM

Examples

>>> from eumap.parallel import TilingProcessing
>>> from eumap.raster import read_rasters
>>>
>>> def run(idx, tile, window, raster_files):
>>>     data, _ = read_rasters(raster_files=raster_files, spatial_win=window, verbose=True)
>>>     print(f'Tile {idx}: data read {data.shape}')
>>>
>>> raster_files = [
>>>     'http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_ndvi_landsat.glad.ard_p50_30m_0..0cm_201903_eumap_epsg3035_v1.0.tif', # winter
>>>     'http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_ndvi_landsat.glad.ard_p50_30m_0..0cm_201906_eumap_epsg3035_v1.0.tif', # spring
>>>     'http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_ndvi_landsat.glad.ard_p50_30m_0..0cm_201909_eumap_epsg3035_v1.0.tif', # summer
>>>     'http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_ndvi_landsat.glad.ard_p50_30m_0..0cm_201912_eumap_epsg3035_v1.0.tif'  # fall
>>> ]
>>>
>>> tiling= TilingProcessing(verbose=True)
>>> idx_list = [0,10,100]
>>> result = tiling.process_multiple(idx_list, run, raster_files)
process_one(idx, func, *args)[source]

Process a single tile using the specified function args.

Parameters

  • idx (int) – The tile id to process. This idx is generated for all the tiles in a sequence starting from 0.

  • func (Callable) – A function with at least the arguments idx, tile, window.

  • args (any) – Additional arguments to send to the function.

Examples

>>> from eumap.parallel import TilingProcessing
>>> from eumap.raster import read_rasters
>>>
>>> def run(idx, tile, window, raster_files):
>>>     data, _ = read_rasters(raster_files=raster_files, spatial_win=window, verbose=True)
>>>     print(f'Tile {idx}: data read {data.shape}')
>>>
>>> raster_files = [
>>>     'http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_ndvi_landsat.glad.ard_p50_30m_0..0cm_201903_eumap_epsg3035_v1.0.tif', # winter
>>>     'http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_ndvi_landsat.glad.ard_p50_30m_0..0cm_201906_eumap_epsg3035_v1.0.tif', # spring
>>>     'http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_ndvi_landsat.glad.ard_p50_30m_0..0cm_201909_eumap_epsg3035_v1.0.tif', # summer
>>>     'http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_ndvi_landsat.glad.ard_p50_30m_0..0cm_201912_eumap_epsg3035_v1.0.tif'  # fall
>>> ]
>>>
>>> tiling= TilingProcessing(verbose=True)
>>> tiling.process_one(0, run, raster_files)