Mailman 3 Re: Dividing a large image into smaller overlapping blocks for parallel processing - scikit-image

newer
Re: Get border pixels of labelled...

Re: Dividing a large image into smaller overlapping blocks for parallel processing

Johannes Schönberger

1 Sep 2013 1 Sep '13

3:07 p.m.

Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition. Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt :

...

On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger wrote:

...
- pad image with skimage.util.pad, which allows a large number of padding methods - spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue

How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.

Stéfan

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

Show replies by date

Colin Lea

1 Sep 1 Sep

9:23 p.m.

New subject: Dividing a large image into smaller overlapping blocks for parallel processing

I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature. On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote:

...

Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.

Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt javascript:>:

...
...
- pad image with skimage.util.pad, which allows a large number of

...
- spawn a pool of processes using Python's multiprocessing package in

On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger javascript:> wrote: padding methods the standard library

...
- use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue

How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.

Stéfan

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com javascript:. For more options, visit https://groups.google.com/groups/opt_out.

Johannes Schönberger

10:55 p.m.

New subject: Dividing a large image into smaller overlapping blocks for parallel processing

Great! Am 01.09.2013 um 17:53 schrieb Colin Lea :

...

I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.

On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.

Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt :

...
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger wrote:

...
- pad image with skimage.util.pad, which allows a large number of padding methods - spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue

How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.

Stéfan

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

Riaan van den Dool

4 Sep 4 Sep

12:57 a.m.

New subject: Dividing a large image into smaller overlapping blocks for parallel processing

I have created a gist with my thoughts of what such a function could look like. https://gist.github.com/riaanvddool/6428299 The examples shown are a bit contrived, with border_size=(0,0) and non-overlapping 'rolling' windows, but the idea should be clear I think. The proc_func function can either process and return a value synchronously, or create a separate job/process for each window, depending on implementation. By keeping this logic separate from the windowing function any multiprocessing-type solution can be used according to preference. In the case of asynchronous processing the results tuple will either be filled with None, or any other value returned by proc_func, ie it will not be the asynchronous result itself and the proc_func will have to implement a way for the asynchronous results to be returned when available (callback function as an example). Your thoughts? Riaan On Sunday, September 1, 2013 7:25:46 PM UTC+2, Johannes Schönberger wrote:

...

Great!

Am 01.09.2013 um 17:53 schrieb Colin Lea javascript:>:

...
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.

On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.

Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt :

...
...
- pad image with skimage.util.pad, which allows a large number of

...
- spawn a pool of processes using Python's multiprocessing package in

On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger wrote: padding methods the standard library

...
- use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue

How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.

Stéfan

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com javascript:. For more options, visit https://groups.google.com/groups/opt_out.

Colin Lea

6:53 a.m.

New subject: Dividing a large image into smaller overlapping blocks for parallel processing

Thanks Riaan, I've already made a PR for this. See here: https://github.com/scikit-image/scikit-image/pull/723/ On Tuesday, September 3, 2013 3:27:51 PM UTC-4, Riaan van den Dool wrote:

...

I have created a gist with my thoughts of what such a function could look like.

https://gist.github.com/riaanvddool/6428299

The examples shown are a bit contrived, with border_size=(0,0) and non-overlapping 'rolling' windows, but the idea should be clear I think.

The proc_func function can either process and return a value synchronously, or create a separate job/process for each window, depending on implementation. By keeping this logic separate from the windowing function any multiprocessing-type solution can be used according to preference. In the case of asynchronous processing the results tuple will either be filled with None, or any other value returned by proc_func, ie it will not be the asynchronous result itself and the proc_func will have to implement a way for the asynchronous results to be returned when available (callback function as an example).

Your thoughts?

Riaan

On Sunday, September 1, 2013 7:25:46 PM UTC+2, Johannes Schönberger wrote:

...
Great!

Am 01.09.2013 um 17:53 schrieb Colin Lea :

...
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.

On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.

Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt :

...
...
- pad image with skimage.util.pad, which allows a large number of

On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger wrote: padding methods

...
- spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue

How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.

Stéfan

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

Riaan van den Dool

9:36 a.m.

New subject: Dividing a large image into smaller overlapping blocks for parallel processing

Yes Colin, I saw, and I have commented on the PR as well. I am submitting this as an alternative approach, because I am not sure that the approach you took is 100% in line with what I need. R On Wednesday, September 4, 2013 3:23:15 AM UTC+2, Colin Lea wrote:

...

Thanks Riaan, I've already made a PR for this. See here: https://github.com/scikit-image/scikit-image/pull/723/

On Tuesday, September 3, 2013 3:27:51 PM UTC-4, Riaan van den Dool wrote:

...
I have created a gist with my thoughts of what such a function could look like.

https://gist.github.com/riaanvddool/6428299

The examples shown are a bit contrived, with border_size=(0,0) and non-overlapping 'rolling' windows, but the idea should be clear I think.

The proc_func function can either process and return a value synchronously, or create a separate job/process for each window, depending on implementation. By keeping this logic separate from the windowing function any multiprocessing-type solution can be used according to preference. In the case of asynchronous processing the results tuple will either be filled with None, or any other value returned by proc_func, ie it will not be the asynchronous result itself and the proc_func will have to implement a way for the asynchronous results to be returned when available (callback function as an example).

Your thoughts?

Riaan

On Sunday, September 1, 2013 7:25:46 PM UTC+2, Johannes Schönberger wrote:

...
Great!

Am 01.09.2013 um 17:53 schrieb Colin Lea :

...
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.

On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.

Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt :

...
...
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <

...
...
...
- pad image with skimage.util.pad, which allows a large number of

js...@demuc.de> wrote: padding methods

...
...
...
- spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue

How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.

Stéfan

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

Riaan van den Dool

11 Sep 11 Sep

11:50 a.m.

New subject: Dividing a large image into smaller overlapping blocks for parallel processing

The current state of the PR looks promising. I have added a comment with some points for discussion around design choices. https://github.com/scikit-image/scikit-image/pull/723#issuecomment-24189987 R On Wednesday, September 4, 2013 6:06:24 AM UTC+2, Riaan van den Dool wrote:

...

Yes Colin, I saw, and I have commented on the PR as well.

I am submitting this as an alternative approach, because I am not sure that the approach you took is 100% in line with what I need.

R

On Wednesday, September 4, 2013 3:23:15 AM UTC+2, Colin Lea wrote:

...
Thanks Riaan, I've already made a PR for this. See here: https://github.com/scikit-image/scikit-image/pull/723/

On Tuesday, September 3, 2013 3:27:51 PM UTC-4, Riaan van den Dool wrote:

...
I have created a gist with my thoughts of what such a function could look like.

https://gist.github.com/riaanvddool/6428299

The examples shown are a bit contrived, with border_size=(0,0) and non-overlapping 'rolling' windows, but the idea should be clear I think.

The proc_func function can either process and return a value synchronously, or create a separate job/process for each window, depending on implementation. By keeping this logic separate from the windowing function any multiprocessing-type solution can be used according to preference. In the case of asynchronous processing the results tuple will either be filled with None, or any other value returned by proc_func, ie it will not be the asynchronous result itself and the proc_func will have to implement a way for the asynchronous results to be returned when available (callback function as an example).

Your thoughts?

Riaan

On Sunday, September 1, 2013 7:25:46 PM UTC+2, Johannes Schönberger wrote:

...
Great!

Am 01.09.2013 um 17:53 schrieb Colin Lea :

...
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.

On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.

Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt :

...
...
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <

...
...
> - pad image with skimage.util.pad, which allows a large number of

js...@demuc.de> wrote: padding methods

...
...
> - spawn a pool of processes using Python's multiprocessing package in the standard library > - use shared memory to provide read access to complete image > - define slices of image blocks and add them to a processing queue

How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.

Stéfan

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

3879

Age (days ago)

3889

Last active (days ago)

List overview

Download

6 comments

3 participants

participants (3)

Colin Lea
Johannes Schönberger
Riaan van den Dool

Re: Dividing a large image into smaller overlapping blocks for parallel processing

Johannes Schönberger

Colin Lea

Johannes Schönberger

Riaan van den Dool

Colin Lea

Riaan van den Dool

Riaan van den Dool

tags

participants (3)