Re: Dividing a large image into smaller overlapping blocks for parallel processing
Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition. Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt <stefan@sun.ac.za>:
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <jsch@demuc.de> wrote:
- pad image with skimage.util.pad, which allows a large number of padding methods - spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue
How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.
Stéfan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature. On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote:
Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.
Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt <ste...@sun.ac.za<javascript:>>:
- pad image with skimage.util.pad, which allows a large number of
- spawn a pool of processes using Python's multiprocessing package in
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <js...@demuc.de<javascript:>> wrote: padding methods the standard library
- use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue
How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.
Stéfan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com <javascript:>. For more options, visit https://groups.google.com/groups/opt_out.
Great! Am 01.09.2013 um 17:53 schrieb Colin Lea <colincsl@gmail.com>:
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.
On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.
Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt <ste...@sun.ac.za>:
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <js...@demuc.de> wrote:
- pad image with skimage.util.pad, which allows a large number of padding methods - spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue
How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.
Stéfan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
I have created a gist with my thoughts of what such a function could look like. https://gist.github.com/riaanvddool/6428299 The examples shown are a bit contrived, with border_size=(0,0) and non-overlapping 'rolling' windows, but the idea should be clear I think. The proc_func function can either process and return a value synchronously, or create a separate job/process for each window, depending on implementation. By keeping this logic separate from the windowing function any multiprocessing-type solution can be used according to preference. In the case of asynchronous processing the results tuple will either be filled with None, or any other value returned by proc_func, ie it will not be the asynchronous result itself and the proc_func will have to implement a way for the asynchronous results to be returned when available (callback function as an example). Your thoughts? Riaan On Sunday, September 1, 2013 7:25:46 PM UTC+2, Johannes Schönberger wrote:
Great!
Am 01.09.2013 um 17:53 schrieb Colin Lea <coli...@gmail.com <javascript:>>:
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.
On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.
Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt <ste...@sun.ac.za>:
- pad image with skimage.util.pad, which allows a large number of
- spawn a pool of processes using Python's multiprocessing package in
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <js...@demuc.de> wrote: padding methods the standard library
- use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue
How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.
Stéfan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com <javascript:>. For more options, visit https://groups.google.com/groups/opt_out.
Thanks Riaan, I've already made a PR for this. See here: https://github.com/scikit-image/scikit-image/pull/723/ On Tuesday, September 3, 2013 3:27:51 PM UTC-4, Riaan van den Dool wrote:
I have created a gist with my thoughts of what such a function could look like.
https://gist.github.com/riaanvddool/6428299
The examples shown are a bit contrived, with border_size=(0,0) and non-overlapping 'rolling' windows, but the idea should be clear I think.
The proc_func function can either process and return a value synchronously, or create a separate job/process for each window, depending on implementation. By keeping this logic separate from the windowing function any multiprocessing-type solution can be used according to preference. In the case of asynchronous processing the results tuple will either be filled with None, or any other value returned by proc_func, ie it will not be the asynchronous result itself and the proc_func will have to implement a way for the asynchronous results to be returned when available (callback function as an example).
Your thoughts?
Riaan
On Sunday, September 1, 2013 7:25:46 PM UTC+2, Johannes Schönberger wrote:
Great!
Am 01.09.2013 um 17:53 schrieb Colin Lea <coli...@gmail.com>:
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.
On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.
Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt <ste...@sun.ac.za>:
- pad image with skimage.util.pad, which allows a large number of
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <js...@demuc.de> wrote: padding methods
- spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue
How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.
Stéfan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Yes Colin, I saw, and I have commented on the PR as well. I am submitting this as an alternative approach, because I am not sure that the approach you took is 100% in line with what I need. R On Wednesday, September 4, 2013 3:23:15 AM UTC+2, Colin Lea wrote:
Thanks Riaan, I've already made a PR for this. See here: https://github.com/scikit-image/scikit-image/pull/723/
On Tuesday, September 3, 2013 3:27:51 PM UTC-4, Riaan van den Dool wrote:
I have created a gist with my thoughts of what such a function could look like.
https://gist.github.com/riaanvddool/6428299
The examples shown are a bit contrived, with border_size=(0,0) and non-overlapping 'rolling' windows, but the idea should be clear I think.
The proc_func function can either process and return a value synchronously, or create a separate job/process for each window, depending on implementation. By keeping this logic separate from the windowing function any multiprocessing-type solution can be used according to preference. In the case of asynchronous processing the results tuple will either be filled with None, or any other value returned by proc_func, ie it will not be the asynchronous result itself and the proc_func will have to implement a way for the asynchronous results to be returned when available (callback function as an example).
Your thoughts?
Riaan
On Sunday, September 1, 2013 7:25:46 PM UTC+2, Johannes Schönberger wrote:
Great!
Am 01.09.2013 um 17:53 schrieb Colin Lea <coli...@gmail.com>:
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.
On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.
Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt <ste...@sun.ac.za>:
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <
- pad image with skimage.util.pad, which allows a large number of
js...@demuc.de> wrote: padding methods
- spawn a pool of processes using Python's multiprocessing package in the standard library - use shared memory to provide read access to complete image - define slices of image blocks and add them to a processing queue
How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.
Stéfan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
The current state of the PR looks promising. I have added a comment with some points for discussion around design choices. https://github.com/scikit-image/scikit-image/pull/723#issuecomment-24189987 R On Wednesday, September 4, 2013 6:06:24 AM UTC+2, Riaan van den Dool wrote:
Yes Colin, I saw, and I have commented on the PR as well.
I am submitting this as an alternative approach, because I am not sure that the approach you took is 100% in line with what I need.
R
On Wednesday, September 4, 2013 3:23:15 AM UTC+2, Colin Lea wrote:
Thanks Riaan, I've already made a PR for this. See here: https://github.com/scikit-image/scikit-image/pull/723/
On Tuesday, September 3, 2013 3:27:51 PM UTC-4, Riaan van den Dool wrote:
I have created a gist with my thoughts of what such a function could look like.
https://gist.github.com/riaanvddool/6428299
The examples shown are a bit contrived, with border_size=(0,0) and non-overlapping 'rolling' windows, but the idea should be clear I think.
The proc_func function can either process and return a value synchronously, or create a separate job/process for each window, depending on implementation. By keeping this logic separate from the windowing function any multiprocessing-type solution can be used according to preference. In the case of asynchronous processing the results tuple will either be filled with None, or any other value returned by proc_func, ie it will not be the asynchronous result itself and the proc_func will have to implement a way for the asynchronous results to be returned when available (callback function as an example).
Your thoughts?
Riaan
On Sunday, September 1, 2013 7:25:46 PM UTC+2, Johannes Schönberger wrote:
Great!
Am 01.09.2013 um 17:53 schrieb Colin Lea <coli...@gmail.com>:
I just made a function blocproc to do this. I'll add a pull request later today. It works in a similar way as the matlab feature.
On Sunday, September 1, 2013 5:37:22 AM UTC-4, Johannes Schönberger wrote: Yes that should be very useful. Nevertheless, I think a function like Matlab's blockproc would be a really good addition.
Am 01.09.2013 um 11:34 schrieb Stéfan van der Walt <ste...@sun.ac.za>:
On Sat, Aug 31, 2013 at 8:17 PM, Johannes Schönberger <
> - pad image with skimage.util.pad, which allows a large number of
js...@demuc.de> wrote: padding methods
> - spawn a pool of processes using Python's multiprocessing package in the standard library > - use shared memory to provide read access to complete image > - define slices of image blocks and add them to a processing queue
How about we add an `overlap` parameter to `skimage.utill.view_as_windows`? That should solve this problem.
Stéfan
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
-- You received this message because you are subscribed to the Google Groups "scikit-image" group. To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
participants (3)
-
Colin Lea -
Johannes Schönberger -
Riaan van den Dool