[AstroPy] Using quantities is slow

Kevin Gullikson kevin.gullikson at gmail.com
Tue Aug 25 10:57:36 EDT 2015


Thanks for the suggestions everyone! I ended up using units for the
interface code that gets called once, and using plain numpy arrays for the
likelihood calculations. That gave me a ~4x speedup!


Kevin Gullikson
PhD Candidate
University of Texas Astronomy
RLM 15.310E

On Tue, Aug 25, 2015 at 12:25 AM, Adam Ginsburg <adam.g.ginsburg at gmail.com>
wrote:

> I've encountered performance problems with units before, and one issue
> is that creating new composite units (e.g., u.km/u.s) comes with some
> overhead.  If such operations are a significant overhead, you can set
> them up in advance, e.g.:
>
> _kms = u.km/u.s
>
> and use that elsewhere in the code.  You can also get a small boost
> from using u.Quantity(x, unit) instead of x*unit.
>
> In [1]: %timeit 5*u.km/u.s
> The slowest run took 4.81 times longer than the fastest. This could
> mean that an intermediate result is being cached
> 10000 loops, best of 3: 40.6 µs per loop
>
> In [2]: %timeit u.Quantity(5, u.km/u.s)
> The slowest run took 4.38 times longer than the fastest. This could
> mean that an intermediate result is being cached
> 10000 loops, best of 3: 30.1 µs per loop
>
> In [3]: _kms = u.km/u.s
>
> In [4]: %timeit u.Quantity(5, _kms)
> The slowest run took 6.71 times longer than the fastest. This could
> mean that an intermediate result is being cached
> 100000 loops, best of 3: 7.89 µs per loop
>
> (thanks to Tom Robitaille who pointed all this out to me)
>
> On Tue, Aug 25, 2015 at 2:58 AM, Paul Kuin <npkuin at gmail.com> wrote:
> > I'm converting my code to use astropy.units, and decided to use it
> mainly at
> > the interface level, and use my own set of fixed units inside the
> software
> > units.
> >
> > That said, it seems to me that it would be useful if you could set some
> flag
> > to tell the code that the units are not going to change until you unset
> the
> > flag. Then the "y = xx.value" steps are not needed. Or perhaps this is a
> > really stupid idea because I don't know the ins and outs. .. ;-)
> >
> > Paul
> >
> > On Mon, Aug 24, 2015 at 11:41 PM, Erik Bray <embray at stsci.edu> wrote:
> >>
> >> On 08/24/2015 06:32 PM, Derek Homeier wrote:
> >> > On 24 Aug 2015, at 3:57 pm, Kevin Gullikson <
> kevin.gullikson at gmail.com>
> >> > wrote:
> >> >>
> >> >> I am writing an MCMC fitting code, and my current implementation uses
> >> >> the (very useful) astropy quantities framework to take care of unit
> >> >> conversions and such. However, the code is surprisingly slow so I
> profiled
> >> >> it with %prun and it looks like the quantity utilities that wrap
> numpy
> >> >> functions are taking a huge chunk of time. Is that just the price of
> >> >> convenience?
> >> >>
> >> > I am not an expert on the quantities/units implementation, but I would
> >> > imagine that they could indeed
> >> > introduce a significant overhead, as the class adds a number of
> >> > validations and checks for a function
> >> > calling a quantity object. Especially with rather numerous operations
> on
> >> > comparatively small arrays
> >> > this could become problematic.
> >> > Did you experiment with using decorators as described in
> >> >
> >> >
> http://astropy.readthedocs.org/en/stable/units/quantity.html#functions-accepting-quantities
> >> >
> >> > - iiuc this would allow bypassing some of the validations?
> >>
> >> I don't think this is really relevant to the OP's question (though
> always
> >> good
> >> to know about).  This decorator just makes it easy to bypass writing a
> lot
> >> of
> >> boilerplate code for checking the units of its inputs--it doesn't remove
> >> any
> >> overhead (in fact it adds it--but this was used for cases where that
> >> overhead
> >> already existed--the code was just highly repetitive).
> >>
> >> > Another option might be to only pass plain numpy arrays to the
> innermost
> >> > loops of your code with .value,
> >> > of course at the expense of some of the convenience gained in the
> first
> >> > place...
> >>
> >> Right, that's the big problem.  The thing Quantity supports (which most
> >> similar
> >> libraries in Python don't) is that it *is* a Numpy ndarray, and with few
> >> exceptions [1] works transparently with all of Numpy's universal
> >> functions.
> >> However, Numpy currently doesn't make it easy to wrap calls to ufuncs,
> so
> >> there's some overhead associated with redoing unit conversions on every
> >> function
> >> call (and even if Numpy did make this easier, as it will in 1.10,
> >> performing
> >> unit conversions repeatedly in a loop will add significant overhead to
> any
> >> fitting operation, so better to do the unit conversions first so that
> all
> >> quantities are expressed in the same base units and orders of magnitude,
> >> then as
> >> Derek suggested pass in the raw arrays (from .value) to high performance
> >> code.
> >>
> >> Having a mechanism to automate such conversions will be useful I think.
> >>
> >> Erik
> >>
> >>
> >> [1]
> >>
> http://astropy.readthedocs.org/en/stable/known_issues.html#quantity-issues
> >> _______________________________________________
> >> AstroPy mailing list
> >> AstroPy at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/astropy
> >
> >
> >
> >
> > --
> >
> > * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * *
> > Dr. N.P.M. Kuin      (n.kuin at ucl.ac.uk)
> > phone +44-(0)1483 (prefix) -204927 (work)
> > mobile +44(0)7806985366  skype ID: npkuin
> > Mullard Space Science Laboratory  – University College London  –
> > Holmbury St Mary – Dorking – Surrey RH5 6NT–  U.K.
> >
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at scipy.org
> > http://mail.scipy.org/mailman/listinfo/astropy
> >
>
>
>
> --
> Adam Ginsburg
> Fellow, European Southern Observatory
> http://www.adamgginsburg.com/
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150825/cbeb0a89/attachment.html>


More information about the AstroPy mailing list