Mailman 3 January 2017 - scikit-image

Re: [scikit-image] Fw: Re: Compilation Problems
by Stefan van der Walt 10 Jan '17

10 Jan '17

On Tue, Jan 10, 2017, at 11:15, Oleksandr Huziy wrote: > I am afraid he would still need the openssl library or is it included > in the python sources? Ah, I see what you mean; I thought some of those binaries would be static compiles, but I guess they may not be. A simple experiment should suffice to clear that up :) Stéfan

1 0

Re: [scikit-image] Fw: Re: Compilation Problems
by Stefan van der Walt 10 Jan '17

10 Jan '17

You may want to follow Nathaniel's suggestion and install a version of Python that has SSL support. Are you able to make a local Python installation into your home directory? Stéfan On Tue, Jan 10, 2017, at 07:43, Oleksandr Huziy wrote: > Hi James: > > Have you tried? > > pip install scikit-image --user > > Or with the cloned directory: > pip install . --user > > > Cheers > > > > > > 2017-01-10 10:33 GMT-05:00 James Board via scikit-image <scikit- > image(a)python.org>: >> I don't have root access on this >> machine and I personally didn't install the python >> version. I can't tell if the sysadmins installed >> openssl-devel before building python. >> >> How sure are you that this is causing my problem? I >> could build python and openssl-devel myself. But >> without root access, that could be a lot of work because >> I'll have to track down each dependency manually. >> >> Thank you for replying. >> >> >> > >> > -------------------------------------------- >> > On Sun, 1/8/17, Nathaniel Smith <njs(a)pobox.com> >> > wrote: >> > >> > Subject: Re: [scikit-image] Compilation Problems >> > To: "James Board" <jpboard2(a)yahoo.com> >> > Cc: scikit-image(a)python.org >> > Date: Sunday, January 8, 2017, 10:16 PM >> >> > >> > You appear to be using a self-built >> > Python. Did you by chance forget >> > to install openssl-devel before building Python, so that >> > you >> > ended up >> > with a Python that has no ssl module? >> > >> > -n >> > >> > On Sun, Jan 8, 2017 at 6:11 PM, James Board via >> > scikit-image >> > <scikit-image(a)python.org> >> > wrote: >> > > I'm on a Red-Hat-like Linux machine and I tried to >> > install scikit several ways and failed. Here was one >> > attempt: >> > > >> > >> git clone https://github.com/scikit-image/scikit-image.git >> > >> cd scikit-image >> > >> python --version >> > > Python 2.7.9 >> > > >> > >> pip install -e >> > > Traceback (most recent call last): >> > > File >> > "/usr/local/python/2.7.9/gnu//bin/pip", line 9, in >> > <module> >> > > load_entry_point('pip==8.1.0', >> > 'console_scripts', 'pip')() >> > > File >> > "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", >> > line 568, in load_entry_point >> > > File >> > "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", >> > line 2720, in load_entry_point >> > > File >> > "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", >> > line 2380, in load >> > > File >> > "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", >> > line 2386, in resolve >> > > File >> > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip- >> > 8.1.0-py2.7.egg/pip/__init__.py", >> > line 15, in <module> >> > > from pip.vcs import git, >> > mercurial, subversion, bazaar # noqa >> > > File >> > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip- >> > 8.1.0-py2.7.egg/pip/vcs/subversion.py", >> > line 9, in <module> >> > > from pip.index import Link >> > > File >> > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip- >> > 8.1.0-py2.7.egg/pip/index.py", >> > line 30, in <module> >> > > from pip.wheel import Wheel, >> > wheel_ext >> > > File >> > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip- >> > 8.1.0-py2.7.egg/pip/wheel.py", >> > line 39, in <module> >> > > from >> > pip._vendor.distlib.scripts import ScriptMaker >> > > File >> > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip- >> > 8.1.0-py2.7.egg/pip/_vendor/distlib/scripts.py", >> > line 14, in <module> >> > > from .compat import sysconfig, >> > detect_encoding, ZipFile >> > > File >> > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip- >> > 8.1.0-py2.7.egg/pip/_vendor/distlib/compat.py", >> > line 31, in <module> >> > > from urllib2 import (Request, >> > urlopen, URLError, HTTPError, >> > > ImportError: cannot import name HTTPSHandler >> > > >> > > Here's another attempt that failed: >> > > >> > >> easy_install --install-dir=/p/home/jpboard/SCIKIT >> > -U scikit-image >> > > Creating /p/home/jpboard/SCIKIT/site.py >> > > Searching for scikit-image >> > > Reading https://pypi.python.org/simple/scikit-image/ >> > > Download error on >> > https://pypi.python.org/simple/scikit-image/: >> > unknown url type: https -- Some packages may not be >> > found! >> > > Couldn't find index page for 'scikit-image' (maybe >> > misspelled?) >> > > Scanning index of all packages (this may take a >> > while) >> > > Reading https://pypi.python.org/simple/ >> > > Download error on https://pypi.python.org/simple/: unknown url >> > > type: >> > https -- Some packages may not be found! >> > > No local packages or download links found for >> > scikit-image >> > > error: Could not find suitable distribution for >> > Requirement.parse('scikit-image') >> > > >> > > Can anyone tell what I'm doing >> > wrong? I tried all available options for me >> > in the scikit documentation. What should I do next? >> > > >> > > _______________________________________________ >> > > scikit-image mailing list >> > > scikit-image(a)python.org >> > > https://mail.python.org/mailman/listinfo/scikit-image >> > >> > >> > >> > -- >> > Nathaniel J. Smith -- https://vorpus.org >> > >> _______________________________________________ >> scikit-image mailing list >> scikit-image(a)python.org >> https://mail.python.org/mailman/listinfo/scikit-image > > > > -- > Sasha > _________________________________________________ > scikit-image mailing list > scikit-image(a)python.org > https://mail.python.org/mailman/listinfo/scikit-image

2 1

Older versions of scikit-image
by James Board 10 Jan '17

10 Jan '17

Where can I download scikit-image version 0.93? There is a list of versions here http://scikit-image.org/docs/dev/install.html But the links to older versions don't take me to an actual download page.

2 1

Re: [scikit-image] Fw: Re: Compilation Problems
by James Board 10 Jan '17

10 Jan '17

I made some progress. I tried this command in the cloned version of scikit-image and it got fairly deep into the compilation process: python setup.py install --prefix ~/SCIKIT However, it ultimately failed with these error messages: Processing dependencies for scikit-image==0.13.dev0 Searching for PyWavelets>=0.4.0 Reading https://pypi.python.org/simple/PyWavelets/ Download error on https://pypi.python.org/simple/PyWavelets/: unknown url type: https -- Some packages may not be found! Couldn't find index page for 'PyWavelets' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading https://pypi.python.org/simple/ Download error on https://pypi.python.org/simple/: unknown url type: https -- Some packages may not be found! No local packages or download links found for PyWavelets>=0.4.0 error: Could not find suitable distribution for Requirement.parse('PyWavelets>=0.4.0') What is the cause of this?

2 1

Re: [scikit-image] Fw: Re: Compilation Problems
by James Board 10 Jan '17

10 Jan '17

I just tried both. Here are the error messages: pip install scikit-image --user Traceback (most recent call last): File "/usr/local/python/2.7.9/gnu/bin/pip", line 9, in <module> load_entry_point('pip==8.1.0', 'console_scripts', 'pip')() File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 568, in load_entry_point File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2720, in load_entry_point File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2380, in load File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2386, in resolve File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/__init__.py", line 15, in <module> from pip.vcs import git, mercurial, subversion, bazaar # noqa File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/vcs/subversion.py", line 9, in <module> from pip.index import Link File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/index.py", line 30, in <module> from pip.wheel import Wheel, wheel_ext File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/wheel.py", line 39, in <module> from pip._vendor.distlib.scripts import ScriptMaker File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/_vendor/distlib/scripts.py", line 14, in <module> from .compat import sysconfig, detect_encoding, ZipFile File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/_vendor/distlib/compat.py", line 31, in <module> from urllib2 import (Request, urlopen, URLError, HTTPError, ImportError: cannot import name HTTPSHandler And in the cloned directory: > cd scikit-image > pip install . --user Traceback (most recent call last): File "/usr/local/python/2.7.9/gnu/bin/pip", line 9, in <module> load_entry_point('pip==8.1.0', 'console_scripts', 'pip')() File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 568, in load_entry_point File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2720, in load_entry_point File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2380, in load File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2386, in resolve File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/__init__.py", line 15, in <module> from pip.vcs import git, mercurial, subversion, bazaar # noqa File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/vcs/subversion.py", line 9, in <module> from pip.index import Link File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/index.py", line 30, in <module> from pip.wheel import Wheel, wheel_ext File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/wheel.py", line 39, in <module> from pip._vendor.distlib.scripts import ScriptMaker File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/_vendor/distlib/scripts.py", line 14, in <module> from .compat import sysconfig, detect_encoding, ZipFile File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/_vendor/distlib/compat.py", line 31, in <module> from urllib2 import (Request, urlopen, URLError, HTTPError, ImportError: cannot import name HTTPSHandler

1 0

Fw: Re: Compilation Problems
by James Board 10 Jan '17

10 Jan '17

I don't have root access on this machine and I personally didn't install the python version. I can't tell if the sysadmins installed openssl-devel before building python. How sure are you that this is causing my problem? I could build python and openssl-devel myself. But without root access, that could be a lot of work because I'll have to track down each dependency manually. Thank you for replying. > > -------------------------------------------- > On Sun, 1/8/17, Nathaniel Smith <njs(a)pobox.com> > wrote: > > Subject: Re: [scikit-image] Compilation Problems > To: "James Board" <jpboard2(a)yahoo.com> > Cc: scikit-image(a)python.org > Date: Sunday, January 8, 2017, 10:16 PM > > You appear to be using a self-built > Python. Did you by chance forget > to install openssl-devel before building Python, so that > you > ended up > with a Python that has no ssl module? > > -n > > On Sun, Jan 8, 2017 at 6:11 PM, James Board via > scikit-image > <scikit-image(a)python.org> > wrote: > > I'm on a Red-Hat-like Linux machine and I tried to > install scikit several ways and failed. Here was one > attempt: > > > >> git clone https://github.com/scikit-image/scikit-image.git > >> cd scikit-image > >> python --version > > Python 2.7.9 > > > >> pip install -e > > Traceback (most recent call last): > > File > "/usr/local/python/2.7.9/gnu//bin/pip", line 9, in > <module> > > load_entry_point('pip==8.1.0', > 'console_scripts', 'pip')() > > File > "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", > line 568, in load_entry_point > > File > "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", > line 2720, in load_entry_point > > File > "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", > line 2380, in load > > File > "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", > line 2386, in resolve > > File > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/__init__.py", > line 15, in <module> > > from pip.vcs import git, > mercurial, subversion, bazaar # noqa > > File > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/vcs/subversion.py", > line 9, in <module> > > from pip.index import Link > > File > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/index.py", > line 30, in <module> > > from pip.wheel import Wheel, > wheel_ext > > File > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/wheel.py", > line 39, in <module> > > from > pip._vendor.distlib.scripts import ScriptMaker > > File > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/_vendor/distlib/scripts.py", > line 14, in <module> > > from .compat import sysconfig, > detect_encoding, ZipFile > > File > "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/_vendor/distlib/compat.py", > line 31, in <module> > > from urllib2 import (Request, > urlopen, URLError, HTTPError, > > ImportError: cannot import name HTTPSHandler > > > > Here's another attempt that failed: > > > >> easy_install --install-dir=/p/home/jpboard/SCIKIT > -U scikit-image > > Creating /p/home/jpboard/SCIKIT/site.py > > Searching for scikit-image > > Reading https://pypi.python.org/simple/scikit-image/ > > Download error on > https://pypi.python.org/simple/scikit-image/: > unknown url type: https -- Some packages may not be > found! > > Couldn't find index page for 'scikit-image' (maybe > misspelled?) > > Scanning index of all packages (this may take a > while) > > Reading https://pypi.python.org/simple/ > > Download error on https://pypi.python.org/simple/: unknown url type: > https -- Some packages may not be found! > > No local packages or download links found for > scikit-image > > error: Could not find suitable distribution for > Requirement.parse('scikit-image') > > > > Can anyone tell what I'm doing > wrong? I tried all available options for me > in the scikit documentation. What should I do next? > > > > _______________________________________________ > > scikit-image mailing list > > scikit-image(a)python.org > > https://mail.python.org/mailman/listinfo/scikit-image > > > > -- > Nathaniel J. Smith -- https://vorpus.org >

2 1

Compilation Problems
by James Board 09 Jan '17

09 Jan '17

I'm on a Red-Hat-like Linux machine and I tried to install scikit several ways and failed. Here was one attempt: > git clone https://github.com/scikit-image/scikit-image.git > cd scikit-image > python --version Python 2.7.9 > pip install -e Traceback (most recent call last): File "/usr/local/python/2.7.9/gnu//bin/pip", line 9, in <module> load_entry_point('pip==8.1.0', 'console_scripts', 'pip')() File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 568, in load_entry_point File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2720, in load_entry_point File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2380, in load File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2386, in resolve File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/__init__.py", line 15, in <module> from pip.vcs import git, mercurial, subversion, bazaar # noqa File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/vcs/subversion.py", line 9, in <module> from pip.index import Link File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/index.py", line 30, in <module> from pip.wheel import Wheel, wheel_ext File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/wheel.py", line 39, in <module> from pip._vendor.distlib.scripts import ScriptMaker File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/_vendor/distlib/scripts.py", line 14, in <module> from .compat import sysconfig, detect_encoding, ZipFile File "/usr/local/python/2.7.9/gnu/lib/python2.7/site-packages/pip-8.1.0-py2.7.egg/pip/_vendor/distlib/compat.py", line 31, in <module> from urllib2 import (Request, urlopen, URLError, HTTPError, ImportError: cannot import name HTTPSHandler Here's another attempt that failed: > easy_install --install-dir=/p/home/jpboard/SCIKIT -U scikit-image Creating /p/home/jpboard/SCIKIT/site.py Searching for scikit-image Reading https://pypi.python.org/simple/scikit-image/ Download error on https://pypi.python.org/simple/scikit-image/: unknown url type: https -- Some packages may not be found! Couldn't find index page for 'scikit-image' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading https://pypi.python.org/simple/ Download error on https://pypi.python.org/simple/: unknown url type: https -- Some packages may not be found! No local packages or download links found for scikit-image error: Could not find suitable distribution for Requirement.parse('scikit-image') Can anyone tell what I'm doing wrong? I tried all available options for me in the scikit documentation. What should I do next?

2 1

Re: [scikit-image] Image analysis pipeline improvement suggestions
by simone codeluppi 06 Jan '17

06 Jan '17

Hi Nathan thanks a lot for the infos! I looked into it and it seems promising, I just need to understand if it is possible to save numpy data into feather. In the pipeline I developed, in the majority of the intermediate steps the generated numpy arrays are saved as single file (as .lzma using joblib.dump) that are then written on a single hdf5 file at the end of the processing step. However, as it is now one of the steps require parallel writing to the file. After your suggestion and some discussion on stack overflow I think that the best solution will be to modify the code and skip the parallel writing. The main issue with the RAM however is related to some of the large image stacks I need to filter (using scipy already made filters). Unfortunately the only way out will be to chunk the images, process them on different cores and then re-join them if the full image can fit in RAM. Thanks a lot for the help! Simone On Wed, Dec 28, 2016 at 2:37 PM Nathan Faggian <nathan.faggian(a)gmail.com> wrote: > Hi Simone, > > I have had a little experience with HDF5 and am interested to see where > you go with this. I wonder if you could use "feather": > https://github.com/wesm/feather > > There was a recent post from Wes McKinney about feather, which sparked my > interest: > http://wesmckinney.com/blog/high-perf-arrow-to-pandas/ > > Do you use HDF5 to store intermediates? if so, I would try storing > intermediates to a file format like feather and then reducing to a HDF5 > file at the end. The reduction should be IO bound and not dependent on RAM > so would suit your cluster. > > If you need to read a large array then I think HDF5 supports that (for > single write but multiple reads) without the need for MPI - so this could > map well to a tool like distributed: > http://distributed.readthedocs.io/en/latest/ > > Not sure this helps, there is an assumption (on my part) that your > intermediate calculations are not terabytes in size. > > Good luck! > > Nathan > > > On 29 December 2016 at 05:07, simone codeluppi <simone.codeluppi(a)gmail.com > > wrote: > > Hi all! > > I would like to pick your brain for some suggestion on how to modify my > image analysis pipeline. > > I am analyzing terabytes of image stacks generated using a microscope. The > current code I generated rely heavily on scikit-image, numpy and scipy. In > order to speed up the analysis the code runs on a HPC computer ( > https://www.nsc.liu.se/systems/triolith/) with MPI (mpi4py) for > parallelization and hdf5 (h5py) for file storage. The development cycle of > the code has been pretty painful mainly due to my non familiarity with mpi > and problems in compiling parallel hdf5 (with many open/closing bugs). > However, the big drawback is that each core has only 2Gb of RAM (no shared > ram across nodes) and in order to run some of the processing steps i ended > up reserving one node (16 cores) but running only 3 cores in order to have > enough ram (image chunking won’t work in this case). As you can imagine > this is extremely inefficient and i end up getting low priority in the > queue system. > > > Our lab currently bought a new 4 nodes server with shared RAM running > hadoop. My goal is to move the parallelization of the processing to dask. I > tested it before in another system and works great. The drawback is that, > if I understood correctly, parallel hdf5 works only with MPI > (driver=’mpio’). Hdf5 gave me quite a bit of headache but works well in > keeping a good structure of the data and i can save everything as numpy > arrays….very handy. > > > If I will move to hadoop/dask what do you think will be a good solution > for data storage? Do you have any additional suggestion that can improve > the layout of the pipeline? Any help will be greatly appreciated. > > > Simone > -- > *Bad as he is, the Devil may be abus'd,* > *Be falsy charg'd, and causelesly accus'd,* > *When men, unwilling to be blam'd alone,* > *Shift off these Crimes on Him which are their* > *Own* > > *Daniel Defoe* > > simone.codeluppi(a)gmail.com > > simone(a)codeluppi.org > > > _______________________________________________ > scikit-image mailing list > scikit-image(a)python.org > https://mail.python.org/mailman/listinfo/scikit-image > > > -- *Bad as he is, the Devil may be abus'd,* *Be falsy charg'd, and causelesly accus'd,* *When men, unwilling to be blam'd alone,* *Shift off these Crimes on Him which are their* *Own* *Daniel Defoe* simone.codeluppi(a)gmail.com simone(a)codeluppi.org

1 0

Re: [scikit-image] Image analysis pipeline improvement suggestions
by Stefanie Lück 05 Jan '17

05 Jan '17

Hi Simone, unfortunately I can not contribute anything to your questions but would you mind to share some of your experience with the hdf5 file format and parallelization? I am also analysing a lot of big microscopy images and would love to hear your opinion or get advice for an image pipeline. Best regards, Stefanie 2016-12-28 23:37 GMT+01:00 Nathan Faggian <nathan.faggian(a)gmail.com>: > Hi Simone, > > I have had a little experience with HDF5 and am interested to see where > you go with this. I wonder if you could use "feather": > https://github.com/wesm/feather > > There was a recent post from Wes McKinney about feather, which sparked my > interest: > http://wesmckinney.com/blog/high-perf-arrow-to-pandas/ > > Do you use HDF5 to store intermediates? if so, I would try storing > intermediates to a file format like feather and then reducing to a HDF5 > file at the end. The reduction should be IO bound and not dependent on RAM > so would suit your cluster. > > If you need to read a large array then I think HDF5 supports that (for > single write but multiple reads) without the need for MPI - so this could > map well to a tool like distributed: > http://distributed.readthedocs.io/en/latest/ > > Not sure this helps, there is an assumption (on my part) that your > intermediate calculations are not terabytes in size. > > Good luck! > > Nathan > > > On 29 December 2016 at 05:07, simone codeluppi <simone.codeluppi(a)gmail.com > > wrote: > >> Hi all! >> >> I would like to pick your brain for some suggestion on how to modify my >> image analysis pipeline. >> >> I am analyzing terabytes of image stacks generated using a microscope. >> The current code I generated rely heavily on scikit-image, numpy and scipy. >> In order to speed up the analysis the code runs on a HPC computer ( >> https://www.nsc.liu.se/systems/triolith/) with MPI (mpi4py) for >> parallelization and hdf5 (h5py) for file storage. The development cycle of >> the code has been pretty painful mainly due to my non familiarity with mpi >> and problems in compiling parallel hdf5 (with many open/closing bugs). >> However, the big drawback is that each core has only 2Gb of RAM (no shared >> ram across nodes) and in order to run some of the processing steps i ended >> up reserving one node (16 cores) but running only 3 cores in order to have >> enough ram (image chunking won’t work in this case). As you can imagine >> this is extremely inefficient and i end up getting low priority in the >> queue system. >> >> >> Our lab currently bought a new 4 nodes server with shared RAM running >> hadoop. My goal is to move the parallelization of the processing to dask. I >> tested it before in another system and works great. The drawback is that, >> if I understood correctly, parallel hdf5 works only with MPI >> (driver=’mpio’). Hdf5 gave me quite a bit of headache but works well in >> keeping a good structure of the data and i can save everything as numpy >> arrays….very handy. >> >> >> If I will move to hadoop/dask what do you think will be a good solution >> for data storage? Do you have any additional suggestion that can improve >> the layout of the pipeline? Any help will be greatly appreciated. >> >> >> Simone >> -- >> *Bad as he is, the Devil may be abus'd,* >> *Be falsy charg'd, and causelesly accus'd,* >> *When men, unwilling to be blam'd alone,* >> *Shift off these Crimes on Him which are their* >> *Own* >> >> *Daniel Defoe* >> >> simone.codeluppi(a)gmail.com >> >> simone(a)codeluppi.org >> >> >> _______________________________________________ >> scikit-image mailing list >> scikit-image(a)python.org >> https://mail.python.org/mailman/listinfo/scikit-image >> >> > > _______________________________________________ > scikit-image mailing list > scikit-image(a)python.org > https://mail.python.org/mailman/listinfo/scikit-image > >

1 0

GLCM strange results
by Stefanie Lück 05 Jan '17

05 Jan '17

Hi everybody, first of all, a happy to new year! I am trying to extract GLCM features of my image but I don't understand the results. The GLCM output images are looking quite different, nevertheless that the source is quite similar. I attached three example images and the corresponding GLCM output. I don't really understand why the second example has much brighter values then the other two images. Can anybody tell me what I am doing wrong? Any help would be very appreciated, thanks in advance, Stefanie Here is the code I used: import cv2 import matplotlib.pyplot as plt img = cv2.imread(crop_path + image, 0) glcm_img = np.copy(img) glcm_img[:] = 0 for i in range(glcm_img.shape[0]): for j in range(glcm_img.shape[1]): if i < 3 or j < 3: continue if i > (glcm_img.shape[0] - 4) or j > (glcm_img.shape[0] - 4): continue glcm_window = img[i - 3: i + 4, j - 3: j + 4] glcm = greycomatrix(glcm_window, [1], [0], symmetric=True, normed=True) contrast = greycoprops(glcm, 'dissimilarity') glcm_img[i, j] = contrast plt.imsave(glcm_path + image, glcm_img, cmap=plt.cm.gray)

1 0