[AstroPy] Using quantities is slow

Erik Bray embray at stsci.edu
Mon Aug 24 18:41:26 EDT 2015


On 08/24/2015 06:32 PM, Derek Homeier wrote:
> On 24 Aug 2015, at 3:57 pm, Kevin Gullikson <kevin.gullikson at gmail.com> wrote:
>>
>> I am writing an MCMC fitting code, and my current implementation uses the (very useful) astropy quantities framework to take care of unit conversions and such. However, the code is surprisingly slow so I profiled it with %prun and it looks like the quantity utilities that wrap numpy functions are taking a huge chunk of time. Is that just the price of convenience?
>>
> I am not an expert on the quantities/units implementation, but I would imagine that they could indeed
> introduce a significant overhead, as the class adds a number of validations and checks for a function
> calling a quantity object. Especially with rather numerous operations on comparatively small arrays
> this could become problematic.
> Did you experiment with using decorators as described in
> http://astropy.readthedocs.org/en/stable/units/quantity.html#functions-accepting-quantities
>
> - iiuc this would allow bypassing some of the validations?

I don't think this is really relevant to the OP's question (though always good 
to know about).  This decorator just makes it easy to bypass writing a lot of 
boilerplate code for checking the units of its inputs--it doesn't remove any 
overhead (in fact it adds it--but this was used for cases where that overhead 
already existed--the code was just highly repetitive).

> Another option might be to only pass plain numpy arrays to the innermost loops of your code with .value,
> of course at the expense of some of the convenience gained in the first place...

Right, that's the big problem.  The thing Quantity supports (which most similar 
libraries in Python don't) is that it *is* a Numpy ndarray, and with few 
exceptions [1] works transparently with all of Numpy's universal functions. 
However, Numpy currently doesn't make it easy to wrap calls to ufuncs, so 
there's some overhead associated with redoing unit conversions on every function 
call (and even if Numpy did make this easier, as it will in 1.10, performing 
unit conversions repeatedly in a loop will add significant overhead to any 
fitting operation, so better to do the unit conversions first so that all 
quantities are expressed in the same base units and orders of magnitude, then as 
Derek suggested pass in the raw arrays (from .value) to high performance code.

Having a mechanism to automate such conversions will be useful I think.

Erik


[1] http://astropy.readthedocs.org/en/stable/known_issues.html#quantity-issues



More information about the AstroPy mailing list