Dividing a large image into smaller overlapping blocks for parallel processing
Hi guys
I would like to use scikitimage to process large images, for example (5696, 13500).
In the interest of speed I need to divide the image into smaller subimages with the possibility of processing these in parallel.
If I define the subimages so that neighbouring subimages overlap then edge effects should not be a problem for the algorithm operating on each subimage.
This is probably a specific case of the more general border/edgeeffect handling issue as addressed by the mode parameter here: http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions:
1. Is there already a imagedivision function/strategy implemented in scikitimage? 2. Is this something that might be included in future if an implementation is available? 3. Please share any references to articles or code that deals with this.
Riaan
Hi Riaan,
Unfortunately we do not have (at least I do not know of) a function similar to Matlab's `blockproc`. Such feature would be a great addition to skimage!
Regards, Johannes
Am 31.08.2013 um 16:04 schrieb Riaan van den Dool riaanvddool@gmail.com:
Hi guys
I would like to use scikitimage to process large images, for example (5696, 13500).
In the interest of speed I need to divide the image into smaller subimages with the possibility of processing these in parallel.
If I define the subimages so that neighbouring subimages overlap then edge effects should not be a problem for the algorithm operating on each subimage.
This is probably a specific case of the more general border/edgeeffect handling issue as addressed by the mode parameter here: http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a imagedivision function/strategy implemented in scikitimage? • Is this something that might be included in future if an implementation is available? • Please share any references to articles or code that deals with this. Riaan
 You received this message because you are subscribed to the Google Groups "scikitimage" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikitimage+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
The blockproc function's signature provides a useful starting point, thanks. http://www.mathworks.com/help/images/ref/blockproc.html
I will have to think about how to do the parallel execution from the function.
Blockproc provides two 'padding' methods: replicate and symmetric. I guess what I need could be called margin, or overlap perhaps.
For the margin case it might make sense that such a function merely returns an array of block definitions, rather than blocks of pixel data. But this would not be so applicable for the replicate and symmetric cases I think.
R
On Saturday, August 31, 2013 6:49:31 PM UTC+2, Johannes Schönberger wrote:
Hi Riaan,
Unfortunately we do not have (at least I do not know of) a function similar to Matlab's `blockproc`. Such feature would be a great addition to skimage!
Regards, Johannes
Am 31.08.2013 um 16:04 schrieb Riaan van den Dool <riaan...@gmail.comjavascript:>:
Hi guys
I would like to use scikitimage to process large images, for example
(5696, 13500).
In the interest of speed I need to divide the image into smaller
subimages with the possibility of processing these in parallel.
If I define the subimages so that neighbouring subimages overlap then
edge effects should not be a problem for the algorithm operating on each subimage.
This is probably a specific case of the more general border/edgeeffect
handling issue as addressed by the mode parameter here:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a imagedivision function/strategy
implemented in scikitimage?
• Is this something that might be included in future if an
implementation is available?
• Please share any references to articles or code that deals
with this.
Riaan
 You received this message because you are subscribed to the Google
Groups "scikitimage" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to scikitimage...@googlegroups.com javascript:.
For more options, visit https://groups.google.com/groups/opt_out.
Some hints:
 pad image with skimage.util.pad, which allows a large number of padding methods  spawn a pool of processes using Python's multiprocessing package in the standard library  use shared memory to provide read access to complete image  define slices of image blocks and add them to a processing queue
Am 31.08.2013 um 20:05 schrieb Riaan van den Dool riaanvddool@gmail.com:
The blockproc function's signature provides a useful starting point, thanks. http://www.mathworks.com/help/images/ref/blockproc.html
I will have to think about how to do the parallel execution from the function.
Blockproc provides two 'padding' methods: replicate and symmetric. I guess what I need could be called margin, or overlap perhaps.
For the margin case it might make sense that such a function merely returns an array of block definitions, rather than blocks of pixel data. But this would not be so applicable for the replicate and symmetric cases I think.
R
On Saturday, August 31, 2013 6:49:31 PM UTC+2, Johannes Schönberger wrote: Hi Riaan,
Unfortunately we do not have (at least I do not know of) a function similar to Matlab's `blockproc`. Such feature would be a great addition to skimage!
Regards, Johannes
Am 31.08.2013 um 16:04 schrieb Riaan van den Dool riaan...@gmail.com:
Hi guys
I would like to use scikitimage to process large images, for example (5696, 13500).
In the interest of speed I need to divide the image into smaller subimages with the possibility of processing these in parallel.
If I define the subimages so that neighbouring subimages overlap then edge effects should not be a problem for the algorithm operating on each subimage.
This is probably a specific case of the more general border/edgeeffect handling issue as addressed by the mode parameter here: http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a imagedivision function/strategy implemented in scikitimage? • Is this something that might be included in future if an implementation is available? • Please share any references to articles or code that deals with this. Riaan
 You received this message because you are subscribed to the Google Groups "scikitimage" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikitimage...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
 You received this message because you are subscribed to the Google Groups "scikitimage" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikitimage+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Thanks
On Saturday, August 31, 2013 8:17:25 PM UTC+2, Johannes Schönberger wrote:
Some hints:
 pad image with skimage.util.pad, which allows a large number of padding
methods
 spawn a pool of processes using Python's multiprocessing package in the
standard library
 use shared memory to provide read access to complete image
 define slices of image blocks and add them to a processing queue
Am 31.08.2013 um 20:05 schrieb Riaan van den Dool <riaan...@gmail.comjavascript:>:
The blockproc function's signature provides a useful starting point,
thanks.
http://www.mathworks.com/help/images/ref/blockproc.html
I will have to think about how to do the parallel execution from the
function.
Blockproc provides two 'padding' methods: replicate and symmetric. I
guess what I need could be called margin, or overlap perhaps.
For the margin case it might make sense that such a function merely
returns an array of block definitions, rather than blocks of pixel data. But this would not be so applicable for the replicate and symmetric cases I think.
R
On Saturday, August 31, 2013 6:49:31 PM UTC+2, Johannes Schönberger
wrote:
Hi Riaan,
Unfortunately we do not have (at least I do not know of) a function
similar to Matlab's `blockproc`. Such feature would be a great addition to skimage!
Regards, Johannes
Am 31.08.2013 um 16:04 schrieb Riaan van den Dool riaan...@gmail.com:
Hi guys
I would like to use scikitimage to process large images, for example
(5696, 13500).
In the interest of speed I need to divide the image into smaller
subimages with the possibility of processing these in parallel.
If I define the subimages so that neighbouring subimages overlap
then edge effects should not be a problem for the algorithm operating on each subimage.
This is probably a specific case of the more general
border/edgeeffect handling issue as addressed by the mode parameter here:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a imagedivision function/strategy
implemented in scikitimage?
• Is this something that might be included in future if an
implementation is available?
• Please share any references to articles or code that deals
with this.
Riaan
 You received this message because you are subscribed to the Google
Groups "scikitimage" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to scikitimage...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 You received this message because you are subscribed to the Google
Groups "scikitimage" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to scikitimage...@googlegroups.com javascript:.
For more options, visit https://groups.google.com/groups/opt_out.
You also might want to look into joblib which makes it very easy to do parallel computations. This is used frequently in sklearn to speedup code.
http://pythonhosted.org/joblib/
On Saturday, August 31, 2013 3:26:51 PM UTC4, Riaan van den Dool wrote:
Thanks
On Saturday, August 31, 2013 8:17:25 PM UTC+2, Johannes Schönberger wrote:
Some hints:
 pad image with skimage.util.pad, which allows a large number of
padding methods
 spawn a pool of processes using Python's multiprocessing package in
the standard library
 use shared memory to provide read access to complete image
 define slices of image blocks and add them to a processing queue
Am 31.08.2013 um 20:05 schrieb Riaan van den Dool riaan...@gmail.com:
The blockproc function's signature provides a useful starting point,
thanks.
http://www.mathworks.com/help/images/ref/blockproc.html
I will have to think about how to do the parallel execution from the
function.
Blockproc provides two 'padding' methods: replicate and symmetric. I
guess what I need could be called margin, or overlap perhaps.
For the margin case it might make sense that such a function merely
returns an array of block definitions, rather than blocks of pixel data. But this would not be so applicable for the replicate and symmetric cases I think.
R
On Saturday, August 31, 2013 6:49:31 PM UTC+2, Johannes Schönberger
wrote:
Hi Riaan,
Unfortunately we do not have (at least I do not know of) a function
similar to Matlab's `blockproc`. Such feature would be a great addition to skimage!
Regards, Johannes
Am 31.08.2013 um 16:04 schrieb Riaan van den Dool riaan...@gmail.com:
Hi guys
I would like to use scikitimage to process large images, for example
(5696, 13500).
In the interest of speed I need to divide the image into smaller
subimages with the possibility of processing these in parallel.
If I define the subimages so that neighbouring subimages overlap
then edge effects should not be a problem for the algorithm operating on each subimage.
This is probably a specific case of the more general
border/edgeeffect handling issue as addressed by the mode parameter here:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.co...
My questions: • Is there already a imagedivision function/strategy
implemented in scikitimage?
• Is this something that might be included in future if an
implementation is available?
• Please share any references to articles or code that deals
with this.
Riaan
 You received this message because you are subscribed to the Google
Groups "scikitimage" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to scikitimage...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 You received this message because you are subscribed to the Google
Groups "scikitimage" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to scikitimage...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger jsch@demuc.de wrote:
 pad image with skimage.util.pad, which allows a large number of padding methods
 spawn a pool of processes using Python's multiprocessing package in the standard library
 use shared memory to provide read access to complete image
 define slices of image blocks and add them to a processing queue
How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.
Stéfan
participants (4)

Colin Lea

Johannes Schönberger

Riaan van den Dool

Stéfan van der Walt