[AstroPy] Using quantities is slow
embray at stsci.edu
Mon Aug 24 18:41:26 EDT 2015
On 08/24/2015 06:32 PM, Derek Homeier wrote:
> On 24 Aug 2015, at 3:57 pm, Kevin Gullikson <kevin.gullikson at gmail.com> wrote:
>> I am writing an MCMC fitting code, and my current implementation uses the (very useful) astropy quantities framework to take care of unit conversions and such. However, the code is surprisingly slow so I profiled it with %prun and it looks like the quantity utilities that wrap numpy functions are taking a huge chunk of time. Is that just the price of convenience?
> I am not an expert on the quantities/units implementation, but I would imagine that they could indeed
> introduce a significant overhead, as the class adds a number of validations and checks for a function
> calling a quantity object. Especially with rather numerous operations on comparatively small arrays
> this could become problematic.
> Did you experiment with using decorators as described in
> - iiuc this would allow bypassing some of the validations?
I don't think this is really relevant to the OP's question (though always good
to know about). This decorator just makes it easy to bypass writing a lot of
boilerplate code for checking the units of its inputs--it doesn't remove any
overhead (in fact it adds it--but this was used for cases where that overhead
already existed--the code was just highly repetitive).
> Another option might be to only pass plain numpy arrays to the innermost loops of your code with .value,
> of course at the expense of some of the convenience gained in the first place...
Right, that's the big problem. The thing Quantity supports (which most similar
libraries in Python don't) is that it *is* a Numpy ndarray, and with few
exceptions  works transparently with all of Numpy's universal functions.
However, Numpy currently doesn't make it easy to wrap calls to ufuncs, so
there's some overhead associated with redoing unit conversions on every function
call (and even if Numpy did make this easier, as it will in 1.10, performing
unit conversions repeatedly in a loop will add significant overhead to any
fitting operation, so better to do the unit conversions first so that all
quantities are expressed in the same base units and orders of magnitude, then as
Derek suggested pass in the raw arrays (from .value) to high performance code.
Having a mechanism to automate such conversions will be useful I think.
More information about the AstroPy