[AstroPy] Using quantities is slow

Erik Bray embray at stsci.edu
Mon Aug 24 18:26:40 EDT 2015


On 08/24/2015 09:57 AM, Kevin Gullikson wrote:
> Hi,
>
> I am writing an MCMC fitting code, and my current implementation uses the (very
> useful) astropy quantities framework to take care of unit conversions and such.
> However, the code is surprisingly slow so I profiled it with %prun and it looks
> like the quantity utilities that wrap numpy functions are taking a huge chunk of
> time. Is that just the price of convenience?

Hi Kevin,

It's hard to say exactly where your sticking point is without seeing some code. 
  But it sounds like you're probably evaluating some function that performs a 
computation using quantities in a loop for interpolation.  If the unit 
conversion needs to be computed every time it goes through that loop that's 
going to always add an enormous amount of overhead (though I think this will be 
improved when Numpy 1.10 is out and we can take advantage of __numpy_ufunc__ 
support).

Better to convert the input arrays so that they're all expressed in the same 
base units.  The exact heuristics to use for this could be very simple, or could 
be quite complex (perhaps to minimize the amount of error).  Then perform the 
fit with unadorned arrays, and restore the units later.

I'm working on similar code to do exactly this for astropy.modeling, so that 
models which have units attached the parameters and/or inputs can be fitted 
efficiently.  And in any case the fitting algorithms in SciPy don't natively 
support units, so it's necessary to make some conversions first.

Perhaps the routine for choosing the best set of conversions to do given a list 
of quantities could be made general enough to be useful outside of 
astropy.modeling (though once that unit support is fully working you might be 
interested to try implementing your MCMC code within the astropy.modeling 
framework).

Erik

> Here is an excerpt of the output from profiling my code.
>
>           1827310507 function calls (1823056217 primitive calls) in 3659.725 seconds
>
>     Ordered by: internal time
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>   25719008  400.735    0.000 1944.942    0.000 quantity.py:263(__array_prepare__)
> 159341911  157.611    0.000  182.754    0.000 {getattr}
> 32101370/31134490  148.248    0.000  180.318    0.000 {method 'format' of 'unicode' objects}
>   27459392  143.898    0.000  397.546    0.000 quantity.py:520(_new_view)
>   25719008  141.664    0.000  566.152    0.000 quantity.py:421(__array_wrap__)
>    6381408  137.927    0.000  733.341    0.000 core.py:2013(_expand_and_gather)
>   63814347  136.370    0.000  373.226    0.000 {hasattr}
> 188542970  135.470    0.000  300.222    0.000 {isinstance}
>   30360032  112.239    0.000  136.142    0.000 core.py:695(__hash__)
>   44089728   99.326    0.000  164.752    0.000 abc.py:128(__instancecheck__)
>   42350301   97.285    0.000  191.681    0.000 {method 'view' of 'numpy.ndarray' objects}
>   30553408   95.704    0.000  236.855    0.000 quantity.py:695(__getattr__)
>    6961536   90.321    0.000  675.250    0.000 quantity.py:756(__mul__)
>     386752   87.576    0.000  900.927    0.002 Orbit.py:133(get_true_anomaly)
>     193376   78.895    0.000 1178.998    0.006 Orbit.py:15(__init__)
>   13729696   65.363    0.000  353.957    0.000 core.py:891(_to)
>   58012800   59.299    0.000  128.197    0.000 quantity.py:260(__array_finalize__)
>     386752   59.201    0.000   79.588    0.000 interpolate.py:1581(_evaluate_linear)
>   73869632   55.839    0.000   55.839    0.000 _weakrefset.py:70(__contains__)
>   15083328   53.805    0.000  266.936    0.000 core.py:2014(add_unit)
>    2900640   42.100    0.000  535.141    0.000 quantity.py:780(__div__)




More information about the AstroPy mailing list