Dividing a large image into smaller overlapping blocks for parallel processing
Hi guys I would like to use scikit-image to process large images, for example (5696, 13500). In the interest of speed I need to divide the image into smaller sub-images with the possibility of processing these in parallel. If I define the sub-images so that neighbouring sub-images overlap then edge effects should not be a problem for the algorithm operating on each sub-image. This is probably a specific case of the more general border/edge-effect handling issue as addressed by the mode parameter here: http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co... My questions: 1. Is there already a image-division function/strategy implemented in scikit-image? 2. Is this something that might be included in future if an implementation is available? 3. Please share any references to articles or code that deals with this. Riaan
Hi Riaan, Unfortunately we do not have (at least I do not know of) a function similar to Matlab's `blockproc`. Such feature would be a great addition to skimage! Regards, Johannes Am 31.08.2013 um 16:04 schrieb Riaan van den Dool <riaanvddool@gmail.com>:
Hi guys
I would like to use scikit-image to process large images, for example (5696, 13500).
In the interest of speed I need to divide the image into smaller sub-images with the possibility of processing these in parallel.
If I define the sub-images so that neighbouring sub-images overlap then edge effects should not be a problem for the algorithm operating on each sub-image.
This is probably a specific case of the more general border/edge-effect handling issue as addressed by the mode parameter here: http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a image-division function/strategy implemented in scikit-image? • Is this something that might be included in future if an implementation is available? • Please share any references to articles or code that deals with this. Riaan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
The blockproc function's signature provides a useful starting point, thanks. http://www.mathworks.com/help/images/ref/blockproc.html I will have to think about how to do the parallel execution from the function. Blockproc provides two 'padding' methods: replicate and symmetric. I guess what I need could be called margin, or overlap perhaps. For the margin case it might make sense that such a function merely returns an array of block definitions, rather than blocks of pixel data. But this would not be so applicable for the replicate and symmetric cases I think. R On Saturday, August 31, 2013 6:49:31 PM UTC+2, Johannes Schönberger wrote:
Hi Riaan,
Unfortunately we do not have (at least I do not know of) a function similar to Matlab's `blockproc`. Such feature would be a great addition to skimage!
Regards, Johannes
Am 31.08.2013 um 16:04 schrieb Riaan van den Dool <riaan...@gmail.com<javascript:>>:
Hi guys
I would like to use scikit-image to process large images, for example (5696, 13500).
In the interest of speed I need to divide the image into smaller sub-images with the possibility of processing these in parallel.
If I define the sub-images so that neighbouring sub-images overlap then edge effects should not be a problem for the algorithm operating on each sub-image.
This is probably a specific case of the more general border/edge-effect handling issue as addressed by the mode parameter here:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a image-division function/strategy
implemented in scikit-image?
• Is this something that might be included in future if an
implementation is available?
• Please share any references to articles or code that deals
with this.
Riaan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com <javascript:>. For more options, visit https://groups.google.com/groups/opt_out.
Some hints: - pad image with skimage.util.pad, which allows a large number of padding methods - spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue Am 31.08.2013 um 20:05 schrieb Riaan van den Dool <riaanvddool@gmail.com>:
The blockproc function's signature provides a useful starting point, thanks. http://www.mathworks.com/help/images/ref/blockproc.html
I will have to think about how to do the parallel execution from the function.
Blockproc provides two 'padding' methods: replicate and symmetric. I guess what I need could be called margin, or overlap perhaps.
For the margin case it might make sense that such a function merely returns an array of block definitions, rather than blocks of pixel data. But this would not be so applicable for the replicate and symmetric cases I think.
R
On Saturday, August 31, 2013 6:49:31 PM UTC+2, Johannes Schönberger wrote: Hi Riaan,
Unfortunately we do not have (at least I do not know of) a function similar to Matlab's `blockproc`. Such feature would be a great addition to skimage!
Regards, Johannes
Am 31.08.2013 um 16:04 schrieb Riaan van den Dool <riaan...@gmail.com>:
Hi guys
I would like to use scikit-image to process large images, for example (5696, 13500).
In the interest of speed I need to divide the image into smaller sub-images with the possibility of processing these in parallel.
If I define the sub-images so that neighbouring sub-images overlap then edge effects should not be a problem for the algorithm operating on each sub-image.
This is probably a specific case of the more general border/edge-effect handling issue as addressed by the mode parameter here: http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a image-division function/strategy implemented in scikit-image? • Is this something that might be included in future if an implementation is available? • Please share any references to articles or code that deals with this. Riaan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Thanks On Saturday, August 31, 2013 8:17:25 PM UTC+2, Johannes Schönberger wrote:
Some hints:
- pad image with skimage.util.pad, which allows a large number of padding methods - spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue
Am 31.08.2013 um 20:05 schrieb Riaan van den Dool <riaan...@gmail.com<javascript:>>:
The blockproc function's signature provides a useful starting point, thanks. http://www.mathworks.com/help/images/ref/blockproc.html
I will have to think about how to do the parallel execution from the function.
Blockproc provides two 'padding' methods: replicate and symmetric. I guess what I need could be called margin, or overlap perhaps.
For the margin case it might make sense that such a function merely returns an array of block definitions, rather than blocks of pixel data. But this would not be so applicable for the replicate and symmetric cases I think.
R
On Saturday, August 31, 2013 6:49:31 PM UTC+2, Johannes Schönberger wrote: Hi Riaan,
Unfortunately we do not have (at least I do not know of) a function similar to Matlab's `blockproc`. Such feature would be a great addition to skimage!
Regards, Johannes
Am 31.08.2013 um 16:04 schrieb Riaan van den Dool <riaan...@gmail.com>:
Hi guys
I would like to use scikit-image to process large images, for example (5696, 13500).
In the interest of speed I need to divide the image into smaller sub-images with the possibility of processing these in parallel.
If I define the sub-images so that neighbouring sub-images overlap then edge effects should not be a problem for the algorithm operating on each sub-image.
This is probably a specific case of the more general border/edge-effect handling issue as addressed by the mode parameter here:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a image-division function/strategy
implemented in scikit-image?
• Is this something that might be included in future if an
implementation is available?
• Please share any references to articles or code that deals
with this.
Riaan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com <javascript:>. For more options, visit https://groups.google.com/groups/opt_out.
You also might want to look into joblib which makes it very easy to do parallel computations. This is used frequently in sklearn to speedup code. http://pythonhosted.org/joblib/ On Saturday, August 31, 2013 3:26:51 PM UTC-4, Riaan van den Dool wrote:
Thanks
On Saturday, August 31, 2013 8:17:25 PM UTC+2, Johannes Schönberger wrote:
Some hints:
- pad image with skimage.util.pad, which allows a large number of padding methods - spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue
Am 31.08.2013 um 20:05 schrieb Riaan van den Dool <riaan...@gmail.com>:
The blockproc function's signature provides a useful starting point, thanks. http://www.mathworks.com/help/images/ref/blockproc.html
I will have to think about how to do the parallel execution from the function.
Blockproc provides two 'padding' methods: replicate and symmetric. I guess what I need could be called margin, or overlap perhaps.
For the margin case it might make sense that such a function merely returns an array of block definitions, rather than blocks of pixel data. But this would not be so applicable for the replicate and symmetric cases I think.
R
On Saturday, August 31, 2013 6:49:31 PM UTC+2, Johannes Schönberger wrote: Hi Riaan,
Unfortunately we do not have (at least I do not know of) a function similar to Matlab's `blockproc`. Such feature would be a great addition to skimage!
Regards, Johannes
Am 31.08.2013 um 16:04 schrieb Riaan van den Dool <riaan...@gmail.com>:
Hi guys
I would like to use scikit-image to process large images, for example
In the interest of speed I need to divide the image into smaller
sub-images with the possibility of processing these in parallel.
If I define the sub-images so that neighbouring sub-images overlap
(5696, 13500). then edge effects should not be a problem for the algorithm operating on each sub-image.
This is probably a specific case of the more general
border/edge-effect handling issue as addressed by the mode parameter here:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a image-division function/strategy
implemented in scikit-image?
• Is this something that might be included in future if an
implementation is available?
• Please share any references to articles or code that deals
with this.
Riaan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <jsch@demuc.de> wrote:
- pad image with skimage.util.pad, which allows a large number of padding methods - spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue
How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem. Stéfan
participants (4)
-
Colin Lea -
Johannes Schönberger -
Riaan van den Dool -
Stéfan van der Walt