[Numpy-discussion] "import numpy" performance

[Paul Ivanov]

On Mon, Jul 2, 2012 at 2:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Mon, Jul 2, 2012 at 10:06 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Mon, Jul 2, 2012 at 9:43 PM, Benjamin Root <ben.root at ou.edu> wrote:
>>>
>>> On Mon, Jul 2, 2012 at 4:34 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>>>> I think this ship has sailed, but it'd be worth looking into lazy
>>>> importing, where 'numpy.fft' isn't actually imported until someone
>>>> starts using it. There are a bunch of libraries that do this, and one
>>>> would have to fiddle to get compatibility with all the different
>>>> python versions and make sure you're not killing performance (might
>>>> have to be in C) but something along the lines of
>>>>
>>>> class _FFTModule(object):
>>>>   def __getattribute__(self, name):
>>>>     mod = importlib.import_module("numpy.fft")
>>>>     _FFTModule.__getattribute__ = mod.__getattribute__
>>>>     return getattr(mod, name)
>>>> fft = _FFTModule()
>>>
>>> Not sure how this would impact projects like ipython that does
>>> tab-completion support, but I know that that would drive me nuts in my basic
>>> tab-completion setup I have for my regular python terminal.  Of course, in
>>> the grand scheme of things, that really isn't all that important, I don't
>>> think.
>>
>> We used to do it for scipy. It did interfere with tab completion. It
>> did drive many people nuts.
>
> Sounds like a bug in your old code, or else the REPLs have gotten
> better? I just pasted the above code into both ipython and python
> prompts, and typing 'fft.<tab>' worked fine in both cases. dir(fft)
> works first try as well.
>
> (If you try this, don't forget to 'import importlib' first, and note
> importlib is 2.7+ only. Obviously importlib is not necessary but it
> makes the minimal example less tedious.)

For anyone interested, I worked out a small lazy-loading class that
we use in nitime [1], which does not need importlib and thus works
on python versions before 2.7 and also has a bit of repr pretty
printing.

I wrote about this to Scipy-Dev [2], and in the original nitime
PR [3] tested that it works in python 2.5, 2.6, 2.7, 3.0, 3.1 and
3.2.

Since that time, we've only changed how we deal with the one
known limitation: reloading a lazily-loaded module was a noop in
that PR, but now throws an error (there's one line commented out
if the noop behavior is preferred).

Here's a link to the rendered docs [4], but if you just grab the
LazyImport class from [1], you can do

  fft = LazyImport('numpy.fft')

1. https://github.com/nipy/nitime/blob/master/nitime/lazyimports.py
2. http://mail.scipy.org/pipermail/scipy-dev/2011-September/016606.html
3. https://github.com/nipy/nitime/pull/88
4. http://nipy.sourceforge.net/nitime/api/generated/nitime.lazyimports.html#module-nitime.lazyimports

best,
-- 
Paul Ivanov
314 address only used for lists,  off-list direct email at:
http://pirsquared.org | GPG/PGP key id: 0x0F3E28F7