Dividing a large image into smaller overlapping blocks for parallel processing

[Riaan van den Dool]

I have created a gist with my thoughts of what such a function could look 
like.

https://gist.github.com/riaanvddool/6428299

The examples shown are a bit contrived, with border_size=(0,0) and 
non-overlapping 'rolling' windows, but the idea should be clear I think.

The proc_func function can either process and return a value synchronously, 
or create a separate job/process for each window, depending on 
implementation. By keeping this logic separate from the windowing function 
any multiprocessing-type solution can be used according to preference. In 
the case of asynchronous processing the results tuple will either be filled 
with None, or any other value returned by proc_func, ie it will not be the 
asynchronous result itself and the proc_func will have to implement a way 
for the asynchronous results to be returned when available (callback 
function as an example).

Your thoughts?

Riaan







 

On Sunday, September 1, 2013 7:25:46 PM UTC+2, Johannes Schönberger wrote:
>
> Great! 
>
> Am 01.09.2013 um 17:53 schrieb Colin Lea <coli... at gmail.com <javascript:>>: 
>
>
> > I just made a function blocproc to do this. I'll add a pull request 
> later today. It works in a similar way as the matlab feature. 
> > 
> > 
> > 
> > On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger 
> wrote: 
> > Yes that should be very useful. Nevertheless, I think a function like 
> Matlab's blockproc would be a really good addition. 
> > 
> > Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt <ste... at sun.ac.za>: 
> > 
> > > On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <js... at demuc.de> 
> wrote: 
> > >> - pad image with skimage.util.pad, which allows a large number of 
> padding methods 
> > >> - spawn a pool of processes using Python's multiprocessing package in 
> the standard library 
> > >> - use shared memory to provide read access to complete image 
> > >> - define slices of image blocks and add them to a processing queue 
> > > 
> > > How about we add an `overlap` parameter to 
> > > `skimage.utill.view_as_windows`?  That should solve this problem. 
> > > 
> > > Stéfan 
> > > 
> > > -- 
> > > You received this message because you are subscribed to the Google 
> Groups "scikit-image" group. 
> > > To unsubscribe from this group and stop receiving emails from it, send 
> an email to scikit-image... at googlegroups.com. 
> > > For more options, visit https://groups.google.com/groups/opt_out. 
> > > 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "scikit-image" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to scikit-image... at googlegroups.com <javascript:>. 
> > For more options, visit https://groups.google.com/groups/opt_out. 
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-image/attachments/20130903/d4fe4f84/attachment.html>