Mailman 3 June 2017 - scikit-image

Re: [scikit-image] Refactoring ndimage
by Egor Panfilov 27 Jun '17

27 Jun '17

Hi Jaime, Very nice to hear that there is a major improvement to `scipy` on its way :). Regarding your questions: - Typically, our `master` branch is pretty stable, and we're fixing the arising bugs as fast as possible, so you might rely on our CI to be green. >From my point of view, `skimage` could be a good place to host more or less domain-specific tests, although I'm not really sure what does specificity mean in case of `ndimage`: convolutions, rescaling, interpolation, basic filtering are pretty generic; - Please, also notice that we've recently switched from `nose` to `pytest` (thanks to Nelle ;)), and any new tests should not have `nose` dependency; - Not sure that I'll find enough time to contribute to the code by myself, but it won't harm if you could provide to the public some directions on where to start with `scipy.ndimage` C-code. Keep up the great work, and best of luck! :) Regards, Egor Panfilov 2017-06-26 12:30 GMT+03:00 Jaime Fernández del Río <jaime.frio(a)gmail.com>: > Hi all, > > I have started working on an ambitious refactor of scipy.ndimage. Some > more context on what I would like to achieve can be found here > <https://mail.python.org/pipermail/scipy-dev/2017-June/021986.html>. > > scikit-image is probably the biggest "customer" of ndimage, so even though > the API and performance are not expected to change, you may want to have a > say. > > Assuming there is no major concern with the overall plan, I would also > like to poke this community for help with: > > - figuring out a sensible test strategy: Ralf Gommers points out that > we should probably use scikit-image's test suite regularly to complement > the relatively few tests currently in ndimage: is the development version > of scikit-image stable enough for this purpose or do you suggest using a > specific stable release? Any other ideas? > - PR reviewing. The idea is to make this refactor as incremental as > possible. That means many smaller interdependent PRs. Even if you don't > have commit rights to the scipy repository you are probably better versed > than most of us in image processing algorithms, so if anyone can keep an > eye on ndimage PRs > <https://github.com/scipy/scipy/pulls?q=is%3Apr+is%3Aopen+label%3Ascipy.ndim…> > and provide occasional reviews it would be greatly appreciated. > - expert consulting: some of the code in ndimage is pretty hard to > understand, e.g. I struggle with spline interpolation. If anyone with the > proper academic knowledge is willing to answer stupid questions, please let > me know. > - the refactoring itself: if anyone would like to be a part of this, > it would be my pleasure! A basic command of the C language is needed, but > I'm more than willing to mentor beginners. > > Thanks! > > Jaime > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes > de dominación mundial. > > _______________________________________________ > scikit-image mailing list > scikit-image(a)python.org > https://mail.python.org/mailman/listinfo/scikit-image > >

1 0

Re: [scikit-image] Refactoring ndimage
by Jaime Fernández del Río 27 Jun '17

27 Jun '17

On Tue, Jun 27, 2017 at 12:09 AM, Robert McLeod <robbmcleod(a)gmail.com> wrote: > Jaime, > > One thing that might be a good selling point would be if you built-in > threaded operations on image stacks, which I think are pretty common in all > forms of microscopy now. I'm thinking particularly of `ni.convolve()`, > where one might want to repeatedly call the function on every image frame > in the stack. Generally multi-threading kernel operations like convolve on > single images is tricky, but if you have a stack it's much more > straight-forward to parallelize. Right now I would do such a thing with a > thread pool in Python, but it would scale a lot better if there was C-level > support. E.g.: > I think the biggest hurdle for adding any such low-level support for multithreading in scipy is not having a portable threading library that will work on the myriad platforms that scipy is supposed to work on. In any case, such a change would have to be a next step after making ndimage something sane to work with, which is the purpose of the refactoring I was describing. Jaime > > import multiprocessing as mp > import numpy as np > import scipy.ndimage as ni > > # funcHandle must release the GIL > def stackFunc( funcHandle, images, funcArgs ): > tPool = mp.ThreadPool( self.n_threads ) > > slices = self.images.shape[0] > # Build parameters list for the threaded processes, consisting of > index > tArgs = [None] * slices > for J in np.arange(slices): > tArgs[J] = (J, images, funcArgs) > > # All operations are done 'in-place' > tPool.map( funcHandle, tArgs ) > tPool.close() > tPool.join() > > > Sincerely, > > Robert > > > On Mon, Jun 26, 2017 at 2:30 AM, Jaime Fernández del Río < > jaime.frio(a)gmail.com> wrote: > >> Hi all, >> >> I have started working on an ambitious refactor of scipy.ndimage. Some >> more context on what I would like to achieve can be found here >> <https://mail.python.org/pipermail/scipy-dev/2017-June/021986.html>. >> >> scikit-image is probably the biggest "customer" of ndimage, so even >> though the API and performance are not expected to change, you may want to >> have a say. >> >> Assuming there is no major concern with the overall plan, I would also >> like to poke this community for help with: >> >> - figuring out a sensible test strategy: Ralf Gommers points out that >> we should probably use scikit-image's test suite regularly to complement >> the relatively few tests currently in ndimage: is the development version >> of scikit-image stable enough for this purpose or do you suggest using a >> specific stable release? Any other ideas? >> - PR reviewing. The idea is to make this refactor as incremental as >> possible. That means many smaller interdependent PRs. Even if you don't >> have commit rights to the scipy repository you are probably better versed >> than most of us in image processing algorithms, so if anyone can keep an >> eye on ndimage PRs >> <https://github.com/scipy/scipy/pulls?q=is%3Apr+is%3Aopen+label%3Ascipy.ndim…> >> and provide occasional reviews it would be greatly appreciated. >> - expert consulting: some of the code in ndimage is pretty hard to >> understand, e.g. I struggle with spline interpolation. If anyone with the >> proper academic knowledge is willing to answer stupid questions, please let >> me know. >> - the refactoring itself: if anyone would like to be a part of this, >> it would be my pleasure! A basic command of the C language is needed, but >> I'm more than willing to mentor beginners. >> >> Thanks! >> >> Jaime >> >> -- >> (\__/) >> ( O.o) >> ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes >> de dominación mundial. >> >> _______________________________________________ >> scikit-image mailing list >> scikit-image(a)python.org >> https://mail.python.org/mailman/listinfo/scikit-image >> >> > > > -- > Robert McLeod, Ph.D. > robert.mcleod(a)unibas.ch > robert.mcleod(a)bsse.ethz.ch <robert.mcleod(a)ethz.ch> > robbmcleod(a)gmail.com > > _______________________________________________ > scikit-image mailing list > scikit-image(a)python.org > https://mail.python.org/mailman/listinfo/scikit-image > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

1 0

Re: [scikit-image] Refactoring ndimage
by Jaime Fernández del Río 27 Jun '17

27 Jun '17

On Mon, Jun 26, 2017 at 11:23 PM, Nelle Varoquaux <nelle.varoquaux(a)gmail.com > wrote: > Hi Jaime, > > Will you be at scipy? Some of the scikit-image crowd will be there for > a sprint: that might be a good place to have a short meeting to > discuss with the people that actually know something about ndimage > (ie, not me…). > Unfortunately no. I have yet to figure a convincing excuse to make these conferences something that my employer pays for. And outside of work my vacation days budget is almost entirely taken by family plans. What I could do is attend the Euroscipy sprints: there seem to be relatively cheap flights from Zürich, and it's even within driving distance. Are there any plans for that? Jaime > > Cheers, > N > > On 26 June 2017 at 14:20, Stefan van der Walt <stefanv(a)berkeley.edu> > wrote: > > Hi Jaime > > > > On Mon, Jun 26, 2017, at 02:30, Jaime Fernández del Río wrote: > > > > I have started working on an ambitious refactor of scipy.ndimage. Some > more > > context on what I would like to achieve can be found here. > > > > > > I hope you can hear the loud applause coming in from all directions! > > > > One of the big barriers to fixing issues in ndimage is the obscurity of > the > > code. I am delighted that you have decided to give it a good look-over > and > > refactor. > > > > My C knowledge is not nearly as good as yours, so I don't know whether > I'd > > be very helpful as a reviewer, but I can try. > > > > Best regards > > Stéfan > > > > > > _______________________________________________ > > scikit-image mailing list > > scikit-image(a)python.org > > https://mail.python.org/mailman/listinfo/scikit-image > > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

1 0

Parallel processing segmentation while keeping labels
by Cedric Espenel 26 Jun '17

26 Jun '17

Hi, I'm working on a project where I have to segment out nucleus of cells in a volume. I use random_walker with define markers for every nuclei. If I try to run it on the all volume the segmentation works well but it's very slow. So I used util.view_as_blocks function from scikit to split my image and markers and I loop over the different block using joblib: rw_dapi_chunks = Parallel(n_jobs = 4)(delayed(segmentation.random_walker)(chunks[i,j], chunks_markers[i,j], beta = 3000, mode='cg_mg') for i in range(2) for j in range(2)) So I end up with a list of 4 images and if I combine these images (btw I'm not sure what is the best way for doing that? I just add them up in an empty array) everything work well, except that I end-up with nuclei with 2 different labels: [image: Images intégrées 1] Am I doing something wrong? Is there a way to split your images in block for parallel computing but being able to get the labeling right at the end? Best, Cedric

1 0

Re: [scikit-image] Refactoring ndimage
by Robert McLeod 26 Jun '17

26 Jun '17

Jaime, One thing that might be a good selling point would be if you built-in threaded operations on image stacks, which I think are pretty common in all forms of microscopy now. I'm thinking particularly of `ni.convolve()`, where one might want to repeatedly call the function on every image frame in the stack. Generally multi-threading kernel operations like convolve on single images is tricky, but if you have a stack it's much more straight-forward to parallelize. Right now I would do such a thing with a thread pool in Python, but it would scale a lot better if there was C-level support. E.g.: import multiprocessing as mp import numpy as np import scipy.ndimage as ni # funcHandle must release the GIL def stackFunc( funcHandle, images, funcArgs ): tPool = mp.ThreadPool( self.n_threads ) slices = self.images.shape[0] # Build parameters list for the threaded processes, consisting of index tArgs = [None] * slices for J in np.arange(slices): tArgs[J] = (J, images, funcArgs) # All operations are done 'in-place' tPool.map( funcHandle, tArgs ) tPool.close() tPool.join() Sincerely, Robert On Mon, Jun 26, 2017 at 2:30 AM, Jaime Fernández del Río < jaime.frio(a)gmail.com> wrote: > Hi all, > > I have started working on an ambitious refactor of scipy.ndimage. Some > more context on what I would like to achieve can be found here > <https://mail.python.org/pipermail/scipy-dev/2017-June/021986.html>. > > scikit-image is probably the biggest "customer" of ndimage, so even though > the API and performance are not expected to change, you may want to have a > say. > > Assuming there is no major concern with the overall plan, I would also > like to poke this community for help with: > > - figuring out a sensible test strategy: Ralf Gommers points out that > we should probably use scikit-image's test suite regularly to complement > the relatively few tests currently in ndimage: is the development version > of scikit-image stable enough for this purpose or do you suggest using a > specific stable release? Any other ideas? > - PR reviewing. The idea is to make this refactor as incremental as > possible. That means many smaller interdependent PRs. Even if you don't > have commit rights to the scipy repository you are probably better versed > than most of us in image processing algorithms, so if anyone can keep an > eye on ndimage PRs > <https://github.com/scipy/scipy/pulls?q=is%3Apr+is%3Aopen+label%3Ascipy.ndim…> > and provide occasional reviews it would be greatly appreciated. > - expert consulting: some of the code in ndimage is pretty hard to > understand, e.g. I struggle with spline interpolation. If anyone with the > proper academic knowledge is willing to answer stupid questions, please let > me know. > - the refactoring itself: if anyone would like to be a part of this, > it would be my pleasure! A basic command of the C language is needed, but > I'm more than willing to mentor beginners. > > Thanks! > > Jaime > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes > de dominación mundial. > > _______________________________________________ > scikit-image mailing list > scikit-image(a)python.org > https://mail.python.org/mailman/listinfo/scikit-image > > -- Robert McLeod, Ph.D. robert.mcleod(a)unibas.ch robert.mcleod(a)bsse.ethz.ch <robert.mcleod(a)ethz.ch> robbmcleod(a)gmail.com

1 0

Re: [scikit-image] Refactoring ndimage
by Nelle Varoquaux 26 Jun '17

26 Jun '17

Hi Jaime, Will you be at scipy? Some of the scikit-image crowd will be there for a sprint: that might be a good place to have a short meeting to discuss with the people that actually know something about ndimage (ie, not me…). Cheers, N On 26 June 2017 at 14:20, Stefan van der Walt <stefanv(a)berkeley.edu> wrote: > Hi Jaime > > On Mon, Jun 26, 2017, at 02:30, Jaime Fernández del Río wrote: > > I have started working on an ambitious refactor of scipy.ndimage. Some more > context on what I would like to achieve can be found here. > > > I hope you can hear the loud applause coming in from all directions! > > One of the big barriers to fixing issues in ndimage is the obscurity of the > code. I am delighted that you have decided to give it a good look-over and > refactor. > > My C knowledge is not nearly as good as yours, so I don't know whether I'd > be very helpful as a reviewer, but I can try. > > Best regards > Stéfan > > > _______________________________________________ > scikit-image mailing list > scikit-image(a)python.org > https://mail.python.org/mailman/listinfo/scikit-image >

1 0

Re: [scikit-image] Refactoring ndimage
by Stefan van der Walt 26 Jun '17

26 Jun '17

Hi Jaime On Mon, Jun 26, 2017, at 02:30, Jaime Fernández del Río wrote: > I have started working on an ambitious refactor of scipy.ndimage. Some > more context on what I would like to achieve can be found here[1]. I hope you can hear the loud applause coming in from all directions! One of the big barriers to fixing issues in ndimage is the obscurity of the code. I am delighted that you have decided to give it a good look- over and refactor. My C knowledge is not nearly as good as yours, so I don't know whether I'd be very helpful as a reviewer, but I can try. Best regards Stéfan Links: 1. https://mail.python.org/pipermail/scipy-dev/2017-June/021986.html

1 0

Fwd: [SciPy-User] EuroSciPy 2017 call for contributions - extension of deadline
by Nelle Varoquaux 26 Jun '17

26 Jun '17

Hi everyone, I thought some of you might be interested in this dead line extension. Cheers, N ---------- Forwarded message ---------- From: Pierre de Buyl <pierre.debuyl(a)kuleuven.be> Date: 26 June 2017 at 03:49 Subject: [SciPy-User] EuroSciPy 2017 call for contributions - extension of deadline To: scipy-user(a)python.org, numpy-discussion(a)python.org (Apologies if you receive multiple copies of this message) 10th European Conference on Python in Science August 28 - September 1, 2017 in Erlangen, Germany The Call for Papers is extended to July 02, 2017 23:00 CEST Description: The EuroSciPy meeting is a cross-disciplinary gathering focused on the use and development of the Python language in scientific research. This event strives to bring together both users and developers of scientific tools, as well as academic research and state of the art industry. Erlangen is one of Germany's major science hubs and located north of Munich (90 minutes by train). The Call for Papers is extended to July 02, 2017 23:00 CEST Regards, The EuroSciPy team https://www.euroscipy.org/2017/ _______________________________________________ SciPy-User mailing list SciPy-User(a)python.org https://mail.python.org/mailman/listinfo/scipy-user

1 0

Refactoring ndimage
by Jaime Fernández del Río 26 Jun '17

26 Jun '17

Hi all, I have started working on an ambitious refactor of scipy.ndimage. Some more context on what I would like to achieve can be found here <https://mail.python.org/pipermail/scipy-dev/2017-June/021986.html>. scikit-image is probably the biggest "customer" of ndimage, so even though the API and performance are not expected to change, you may want to have a say. Assuming there is no major concern with the overall plan, I would also like to poke this community for help with: - figuring out a sensible test strategy: Ralf Gommers points out that we should probably use scikit-image's test suite regularly to complement the relatively few tests currently in ndimage: is the development version of scikit-image stable enough for this purpose or do you suggest using a specific stable release? Any other ideas? - PR reviewing. The idea is to make this refactor as incremental as possible. That means many smaller interdependent PRs. Even if you don't have commit rights to the scipy repository you are probably better versed than most of us in image processing algorithms, so if anyone can keep an eye on ndimage PRs <https://github.com/scipy/scipy/pulls?q=is%3Apr+is%3Aopen+label%3Ascipy.ndim…> and provide occasional reviews it would be greatly appreciated. - expert consulting: some of the code in ndimage is pretty hard to understand, e.g. I struggle with spline interpolation. If anyone with the proper academic knowledge is willing to answer stupid questions, please let me know. - the refactoring itself: if anyone would like to be a part of this, it would be my pleasure! A basic command of the C language is needed, but I'm more than willing to mentor beginners. Thanks! Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

1 0

Re: [scikit-image] Why does img_as force incoming data
by Juan Nunez-Iglesias 21 Jun '17

21 Jun '17

Hi David, Yes, img_float.astype('uint16') will work, but you'll need to make sure you map your float image to the range 0-65535. Juan. On 22 Jun 2017, 9:29 AM +1000, David Protter <david.protter(a)gmail.com>, wrote: > Thank you! Will this also work going back the other way? (Float to uint?) > > > On Jun 21, 2017 5:16 PM, "Stefan van der Walt" <stefanv(a)berkeley.edu> wrote: > > > Hi Dave > > > > > > On Wed, Jun 21, 2017, at 15:22, David Protter wrote: > > > > Hi all, new here and having a lot of fun using Skimage for scientific image analysis. > > > > > > Glad to hear it! > > > > > > > I’m doign some normalization on images coming in as uint16, to expand their dynamic range. After normalization, images are float64, and I’m trying to convert them back to uint16. However, it seems like all the img_as functions ignore the kind of float coming in, since anything of kind float is constrained to [-1 to 1]. Below is the section out of the convert() function that seems to do this. > > > > > > Floating point images are expected to be between -1 and 1, as described here: > > > > > > http://scikit-image.org/docs/stable/user_guide/data_types.html > > > > > > However, many functions support working on data in its original range, by specifying `preserve_range=True`. > > > > > > To convert your data from uint16 to float without changing its range, do: > > > > > > img_float = img.astype(float) > > > > > > Best regards > > > Stéfan > > > > _______________________________________________ > scikit-image mailing list > scikit-image(a)python.org > https://mail.python.org/mailman/listinfo/scikit-image

1 0