[AstroPy] astropy and photutils slow

Nathan Goldbaum nathan12343 at gmail.com
Tue Sep 13 18:14:26 EDT 2016


Hi David,

On Tue, Sep 13, 2016 at 4:47 PM, David Nidever <davidnidever at gmail.com>
wrote:

> Hi Stephen,
>
> This is what I had:
>
> In [7]: np.__version__
> Out[7]: '1.8.0rc1'
>
>
This version indicates to me that you're using the Apple system python
(e.g. /usr/bin/python on OSX). I want to strongly encourage you to avoid
using the system python, since it's possible to actually break the
operating system by using it.

In addition, the apple system python in particular has a number of
questionable aspects (including a pre-release version of Numpy 1.8.0 out of
the box is one example) and this leads to a number of weird issues that can
be avoided by simply not using it.

Instead, try to install python from a package manager like homebrew or
macports or use a self-contained python environment from e.g. Anaconda.

-Nathan


> In [10]: astropy.__version__
> Out[10]: u'1.2.1'
>
> and I'm using python 2.7.
>
> I upgraded to the newest version and I'm finding a similar
> performance to what you showed.
>
> Thanks!
> David
>
>
> On Tue, Sep 13, 2016 at 1:17 PM, Stephen Bailey <stephenbailey at lbl.gov>
> wrote:
>
>> What version of numpy and astropy do you have?  (numpy.__version__ and
>> astropy.__version__).
>>
>> As another data point, using numpy/1.11.1 and astropy/1.2.1 I get much
>> faster results than you:
>>
>> On Tue, Sep 13, 2016 at 12:05 PM, David Nidever <davidnidever at gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I'm pretty new to astropy and to python in general.  I've been trying out
>>> some of the astropy and photutils photometry tools, but I'm finding that
>>> they
>>> are running pretty slow.  I wanted to double-check that I'm not doing
>>> something
>>> wrong.
>>>
>>> Below is an example of running the astropy sigma clipped statistics
>>> routine
>>> on a fake 2Kx4K image (since I'm working on DECam data).  This takes
>>> almost 20s to run even though similar statistics using numpy take ~0.1s
>>> or so.
>>>
>>> >>> import numpy as np
>>> >>> from astropy.stats import sigma_clipped_stats
>>> >>> im = np.random.rand(4096,2048)*10.0+100.0
>>> >>> timeit mean, median, std = sigma_clipped_stats(im, sigma=3.0,
>>> iters=5)
>>> 1 loop, best of 3: 18.9 s per loop
>>>
>>
>> 1.84s per loop for me.  Still slow, but not 18.9s.
>>
>>
>>> It seems like it might be the use of the numpy masked array module that
>>> is
>>> slowing things down.  Just using the masked array median versus the
>>> normal numpy median takes ~30x longer.
>>>
>>> >>> timeit np.median(im)
>>> 10 loops, best of 3: 109 ms per loop
>>>
>>
>> 103 ms for me; i.e. this isn't just that I have a super fast machine and
>> you have a super slow one.
>>
>>
>>> >>> timeit np.ma.median(im)
>>> 1 loop, best of 3: 2.71 s per loop
>>>
>>
>> 104 ms for me.  i.e. with numpy 1.11.1 I don't see a radical difference
>> between masked array or not.
>>
>> Stephen
>>
>>
>>>
>>> I'm finding similar speed issues with the photutils routines (e.g.
>>> background2D)
>>> that I believe also use masked arrays.
>>>
>>> Is there something I can do to speed things up?
>>>
>>> Thanks!
>>> David
>>>
>>> --
>>> Dr. David Lee Nidever
>>> Survey Data Scientist, NOAO
>>> 950 N. Cherry Ave.
>>> Tucson, AZ 85719
>>> (520) 318-8368
>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at scipy.org
>>> https://mail.scipy.org/mailman/listinfo/astropy
>>>
>>>
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> https://mail.scipy.org/mailman/listinfo/astropy
>>
>>
>
>
> --
> Dr. David Lee Nidever
> Survey Data Scientist, NOAO
> 950 N. Cherry Ave.
> Tucson, AZ 85719
> (520) 318-8368
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://mail.scipy.org/mailman/listinfo/astropy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20160913/7c64f449/attachment.html>


More information about the AstroPy mailing list