Would it not be possible to import just the necessary module of numpy to meet the necessary functionality of your application. i.e. import numpy.core or whatever you're using you could even do: import numpy.core as numpy I think, to simplify your code, I'm no expert though. Hanni 2008/7/1 Andrew Dalke <dalke@dalkescientific.com>:

On Jul 1, 2008, at 2:22 AM, Robert Kern wrote:

...

Your use case isn't so typical and so suffers on the import time end of the balance.

I'm working on my presentation for EuroSciPy. "Isn't so typical" seems to be a good summary of my first slide. :)

...

...

Any chance of cutting down on the number, in order to improve startup costs?

Not at this point in time, no. That would break too much code.

Understood.

Thanks for the response,

Andrew dalke@dalkescientific.com

_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

You are correct, it appears to take slightly longer to import numpy.core and longer again to import numpy.core as numpy I should obviously check first in future. Hanni 2008/7/1 Matthieu Brucher <matthieu.brucher@gmail.com>:

Hi,

IIRC, il you do import numpy.core as numpy, it starts by importing numpy, so it will be even slower.

Matthieu

2008/7/1 Hanni Ali <hanni.ali@gmail.com>:

...

Would it not be possible to import just the necessary module of numpy to meet the necessary functionality of your application.

i.e.

import numpy.core

or whatever you're using

you could even do:

import numpy.core as numpy

I think, to simplify your code, I'm no expert though.

Hanni

2008/7/1 Andrew Dalke <dalke@dalkescientific.com>:

...

On Jul 1, 2008, at 2:22 AM, Robert Kern wrote:

...

Your use case isn't so typical and so suffers on the import time end of the balance.

I'm working on my presentation for EuroSciPy. "Isn't so typical" seems to be a good summary of my first slide. :)

...

...

Any chance of cutting down on the number, in order to improve startup costs?

Not at this point in time, no. That would break too much code.

Understood.

Thanks for the response,

Andrew dalke@dalkescientific.com

_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

-- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

On Jul 4, 2008, at 2:22 PM, Andrew Dalke wrote:

[josiah:numpy/build/lib.macosx-10.3-fat-2.5] dalke% time python -c 'pass' 0.015u 0.042s 0:00.06 83.3% 0+0k 0+0io 0pf+0w [josiah:numpy/build/lib.macosx-10.3-fat-2.5] dalke% time python -c 'import numpy' 0.084u 0.231s 0:00.33 93.9% 0+0k 0+8io 0pf+0w [josiah:numpy/build/lib.macosx-10.3-fat-2.5] dalke%

For one of my clients I wrote a tool to analyze import times. I don't have it, but here's something similar I just now whipped up:

Based on those results I've been digging into the code trying to figure out why numpy imports so many files, and at the same time I've been trying to guess at the use case Robert Kern regards as typical when he wrote: Your use case isn't so typical and so suffers on the import time end of the balance and trying to figure out what code would break if those modules weren't all eagerly imported and were instead written as most other Python modules are written. I have two thoughts for why mega-importing might be useful: - interactive users get to do tab complete and see everything (eg, "import numpy" means "numpy.fft.ifft" works, without having to do "import numpy.fft" manually) - class inspectors don't need to to directory checks to find possible modules (This is a stretch, since every general purpose inspector I know of has to know how to frob the directories to find directories.) Are these the reasons numpy imports everything or are there other reasons? The first guess comes from the comment in numpy/__init__.py "The following sub-packages must be explicitly imported:" meaning, I take it, that the other modules (core, lib, random, linalg, fft, testing) do not need to be explicitly imported. Is the numpy recommendation that people should do: import numpy numpy.fft.ifft(data) ? If so, the documentation should be updated to say that "random", "ma", "ctypeslib" and several other libraries are included in that list. Why is the last so important that it should be in the top- level namespace? In my opinion, this assistance is counter to standard practice in effectively every other Python package. I don't see the benefit. You may ask if there are possible improvements. There's no obvious place taking up a bunch of time but there are plenty of small places which add up. For examples: 1) I wondered why 'cPickle' needed to be imported. One of the places it's used is numpy.lib.format which is only imported by numpy.lib.io. It's easy to defer the 'import format' to be inside the functions which need it. Note that io.py already defers the import of zipfile, so function-local imports are not inappropriate. 'io' imports 'tempfile', needing 0.016 seconds. This can be a deferred cost only incurred by those who use io.savez, which already has some function-local imports. The reason for the high import costs? Here's what tempfile itself imports. tempfile: 0.016 (io) errno: 0.000 (tempfile) random: 0.010 (tempfile) binascii: 0.003 (random) _random: 0.003 (random) fcntl: 0.003 (tempfile) thread: 0.000 (tempfile) (This is read as 'tempfile' is imported by 'io' and takes 0.016 seconds total, including all children, and the directly imported children of 'tempfile' are 'errno', 'random', 'fcntl' and 'thread'. 'random' imports 'binascii' and '_random'.) BTW, the load and save commands in io do an incorrect check. if isinstance(file, type("")): fid = _file(file,"rb") else: fid = file Filenames can be unicode strings. This test should either be isinstance(file, basestring) or not hasatttr(file, 'read') 2) What's the point of "add_newdocs"? According to the top of the module # This is only meant to add docs to objects defined in C- extension modules. # The purpose is to allow easier editing of the docstrings without # requiring a re-compile. which implies this aids development, but not deployment. The import takes a miniscule 0.006 seconds of the 0.225 ("import lib" and its subimports takes 0.141 seconds) but seems to add no direct end-user benefit. Shouldn't this documentation be pushed into the C code at least for each release? 3) I see that numpy/core/numerictypes.py imports 'string', which takes 0.008 seconds. I wondered why. It's part of "english_lower", "english_upper", and "english_capitalize", which are functions defined in that module. The implementation can't be improved, and using string.translate is the right approach. However, 3a) the two functions have no leading underscore and have docstrings to imply that this is part of the public API (although they are not included in __all__). Are they meant for general use? Note that english_capitalize is over-engineered for the use-case in that file. There are no empty type names, so the test "if s" is never false. 3b) there are only 33 types in that module so a hand-written lookup table mapping the name to the appropriate name/alias would work. Yes, it makes adding new types less than completely auomatic, but that's done rarely. Getting rid of these functions, and thus getting rid of the import speeds numpy startup time by 3.5%. 4) numpy.testing takes 0.041 seconds to import. The text I quoted above says that it's a numpy requirement that 'testing' always be imported, even though I'm hard pressed to figure out why that's important. Assuming it is important, 0.020 seconds is spent importing 'difflib' difflib: 0.020 (utils) heapq: 0.016 (difflib) itertools: 0.003 (heapq) operator: 0.003 (heapq) bisect: 0.005 (heapq) _bisect: 0.003 (bisect) _heapq: 0.003 (heapq) which is only used in numpy.testing.utils:assert_string . That can be deferred. Similarly, numpytest: 0.012 (numpy.testing) glob: 0.005 (numpytest) fnmatch: 0.002 (glob) shlex: 0.006 (numpytest) collections: 0.003 (shlex) numpy.testing.utils: 0.000 (numpytest) but notice that 'glob' while imported is never used in 'numpytest', and that 'shlex' can easily be a deferred import. This saves (for the common case) 0.01 seconds. 5) There some additional savings in _datasource _datasource: 0.016 (io) shutil: 0.003 (_datasource) stat: 0.000 (shutil) urlparse: 0.003 (_datasource) bz2: 0.003 (_datasource) gzip: 0.006 (_datasource) zlib: 0.003 (gzip) This module provides the "Datasource" class, which is accessed through "numpy.lib.io.Datasource". Deferring the 'bz2' and 'gzip' imports until needed saves 0.01 seconds. This will require some modification to the code more than shifting the import statement. These together add up to about 0.08 seconds, which is about 30% of the 'import numpy' cost. I could probably get another 0.05 seconds if I dug around more, but I can't without knowing what use case numpy is trying to achieve. Why are all those ancillary modules (testing, ctypeslib) eagerly loaded when there seems no need for that feature? Andrew dalke@dalkescientific.com

On Jul 30, 2008, at 10:51 PM, Alan McIntyre wrote:

I suppose it's necessary for providing the test() and bench() functions in subpackages, but I that isn't a good reason to impose upon all users the time required to set up numpy.testing.

I just posted this in my reply to Stéfan, but I'll say it again here. numpy defines numpy.test numpy.bench and numpy.testing.test The two 'test's use the same implementation. This is a likely unneeded duplication and one should be removed. The choice depends on if people think the name should be 'numpy.test' or 'numpy.testing.test'. BTW, where's the on-line documentation for these functions? They are actually bound methods, and I wondered if the doc programs handle them okay. If they should be top-level functions then I would prefer the be actual functions to hide an import. In that case, replace from testing import Tester test = Tester().test with def test(label='fast', verbose=1, extra_argv=None, doctests=False, coverage=False, **kwargs): from testing import Tester Tester.test(label, verbose, extra_argv, doctests, coverage, **kwargs) or something similar. This would keep the API unchanged (assuming those are important in the top-level) and reduce the number of imports. Else I would keep/move them in 'numpy.testing' and require that if someone wants to use 'test' or 'bench' then to get them after a 'from numpy import testing'.

Thanks for taking the time to find those; I just removed the unused glob and delayed the import of shlex, difflib, and inspect in numpy.testing.

Thanks! Andrew dalke@dalkescientific.com

On Jul 31, 2008, at 4:21 AM, Alan McIntyre wrote:

They actually do two different things; numpy.test() runs test for all of numpy, and numpy.testing.test() runs tests for numpy.testing only. There are similar functions in numpy.lib, numpy.core, etc.

Really? This is the code from numpy/__init__.py: from testing import Tester test = Tester().test bench = Tester().bench This is the code from numpy/testing/__init__.py: test = Tester().test ... ahhh, here's the magic, from testing/nosetester.py:NoseTester if package is None: f = sys._getframe(1) package = f.f_locals.get('__file__', None) assert package is not None package = os.path.dirname(package) Why are 'test' and 'bench' part of the general API instead something only used during testing? Andrew dalke@dalkescientific.com

On Mon, Aug 4, 2008 at 18:15, Andrew Dalke <dalke@dalkescientific.com> wrote:

I'm working on the patches for reducing the import overhead. I want to make sure I don't break anything. I'm trying to figure out how to run all of the tests. I expected, based on the following

Alan McIntyre wrote:

...

They actually do two different things; numpy.test() runs test for all of numpy, and numpy.testing.test() runs tests for numpy.testing only. There are similar functions in numpy.lib, numpy.core, etc.

Robert Kern wrote:

...

By now, we have most of the denizens here trained to do numpy.test() when testing their new installations.

README:

...

After installation, tests can be run (from outside the source directory) with:

python -c 'import numpy; numpy.test()'

that 'numpy.test()' runs everthing.

When I run numpy.test() I don't seem to run all of the tests. That is, I don't see the output I get when I run numpy.lib.test() . Here's a copy of my output, to show you what I mean.

You have old stuff in your checkout/installation. Make sure you have deleted all of the *.pycs and directories which have been deleted in SVN.

Also, I can't figure out what when I run a test I get a new Python prompt.

I can't parse that sentence. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Aug 5, 2008, at 2:00 AM, Robert Kern wrote:

You have old stuff in your checkout/installation. Make sure you have deleted all of the *.pycs and directories which have been deleted in SVN.

Now that I've fixed that, I can tell that I made a mistake related to the self-test code. I can't figure it out. Many of the modules have code which look like from testing import Tester test = Tester().test bench = Tester().bench As I understand it, this code is migrating to use nosetests. Because people expect 'import numpy; numpy.test()' to work, there will be a compatibility period where this API is unchanged. I found that importing 'testing' costs 0.013 seconds, which is 10% of my current import time. I would like to defer the import until needed, so I rewrote the 'test' code as: def test(label='fast', verbose=1, extra_argv=None, doctests=False, coverage=False, **kwargs): from testing import Tester import numpy Tester(numpy).test(label, verbose, extra_argv, doctests, coverage, **kwargs) In my view there's no difference between them, but Tester().test does introspection to figure out the module location. (In fact, if I don't pass it the module explicitly then it expects that locals() ["__file__"] for sys._getframe(-1) will exist, which is not the case with my function. The underling code should instead check for that variable in globals().) I ended up with recursion errors, and I don't know why. Any idea of what to do? [josiah:~] dalke% python -c 'import numpy; numpy.test()' Running unit tests for numpy NumPy version 1.2.0.dev5607 NumPy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Python version 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] nose version 0.10.3 Running unit tests for numpy NumPy version 1.2.0.dev5607 NumPy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Python version 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] nose version 0.10.3 Running unit tests for numpy NumPy version 1.2.0.dev5607 NumPy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy .... Python version 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] nose version 0.10.3 EEEEEEEEEEE ====================================================================== ERROR: numpy.test ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/__init__.py", line 107, in test coverage, **kwargs) File "/Users/dalke/cvses/numpy/build/lib.macosx-10.3-fat-2.5/numpy/ testing/nosetester.py", line 270, in test t = NumpyTestProgram(argv=argv, exit=False, plugins=plugins) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/core.py", line 219, in __init__ argv=argv, testRunner=testRunner, testLoader=testLoader) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/unittest.py", line 758, in __init__ self.parseArgs(argv) ... return self.call(*arg, **kw) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ manager.py", line 145, in simple result = meth(*arg, **kw) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 214, in wantMethod return self.validateAttrib(attribs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 164, in validateAttrib if not value(key, attribs): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 118, in eval_in_context return eval(expr, None, ContextHelper(attribs)) File "<string>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 50, in __getitem__ return self.obj.get(name, False) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/plugins/ attrib.py", line 66, in get log.debug('Get %s from %s.%s', name, self.cls, self.method) RuntimeError: maximum recursion depth exceeded

Andrew dalke@dalkescientific.com

On Aug 5, 2008, at 4:19 AM, Robert Kern wrote:

def test(...): ... test.__test__ = False

That did it - thanks! Does "import numpy; numpy.bench()" work for anyone? When I try it I get [josiah:~] dalke% python -c 'import numpy; numpy.bench()' ---------------------------------------------------------------------- Ran 0 tests in 0.003s OK I can go ahead and remove those if they don't work for anyone. Andrew dalke@dalkescientific.com

On Aug 5, 2008, at 3:53 PM, Alan McIntyre wrote:

At the moment, bench() doesn't work. That's something I'll try to look at this week, but from Friday until the 15th I'm going to be driving most of the time and may not get as much done as I'd like.

Thanks for the confirmation. The import speedup patch I just submitted keeps the 'bench' definitions there (including one that was missing). But instead of defining it as a bound method, I used functions that import the testing submodule and construct/call the right objects. It should behave the same. I think. Andrew dalke@dalkescientific.com

On Jul 30, 2008, at 10:59 PM, Stéfan van der Walt wrote:

I.e. most people don't start up NumPy all the time -- they import NumPy, and then do some calculations, which typically take longer than the import time.

Is that interactively, or is that through programs?

For a benefit of 0.03s, I don't think it's worth it.

The final number with all the hundredths of a second added up to 0.08 seconds, which was about 30% of the 'import numpy' cost.

Numpy has a very flat namespace, for better or worse, which implies many imports.

I don't get the feeling that numpy is flat. Python's stdlib is flat. Numpy has many 2- and 3-level modules.

...

Is the numpy recommendation that people should do:

import numpy numpy.fft.ifft(data)

That's the way many people use it.

The normal Python way is: from numpy import fft fft.ifft(data) because in most packages, parent modules don't import all of their children. I acknowledge that existing numpy code will break with my desired change, as this example from the tutorial import numpy import pylab # Build a vector of 10000 normal deviates with variance 0.5^2 and mean 2 mu, sigma = 2, 0.5 v = numpy.random.normal(mu,sigma,10000) and I am not saying to change this code. Instead, I am asking for limits on the eagerness, with a long-term goal of minimizing its use.

...

Why is [ctypeslib] so important that it should be in the top- level namespace?

It's a single Python file -- does it make much of a difference?

The file imports other files. Here's the import chain: ctypeslib: 0.047 (numpy) ctypes: -1.000 (ctypeslib) _ctypes: 0.003 (ctypes) gestalt: -1.000 (ctypes) ma: 0.005 (numpy) extras: 0.001 (ma) numpy.lib.index_tricks: 0.000 (extras) numpy.lib.polynomial: 0.000 (extras) (The "-1.000" indicates a bug in my instrumentation script, which I worked around with a -1.0 value.) Every numpy program, because it eagerly imports 'ctypeslib' to make it be accessible as a top-level variable, ends up importing ctypes.

...

...

if 1: ... t1 = time.time() ... import ctypes ... t2 = time.time() ... t2-t1 0.032159090042114258

That's 10% of the import time.

...

In my opinion, this assistance is counter to standard practice in effectively every other Python package. I don't see the benefit.

How do you propose we change this?

If I had my way, remove things like (in numpy/__init__.py) import linalg import fft import random import ctypeslib import ma but leave the list of submodules in "__all__" so that "from numpy import *" works. Perhaps add a top-level function to 'import_all()' which mimics the current behavior, and have iPython know about it so interactive users get it automatically. Or something like that. Yes, I know the numpy team won't change this behavior. I want to know why you all will consider changing. Something more concrete: change the top-level definitions in 'numpy' from from testing import Tester test = Tester().test bench = Tester().bench with def test(label='fast', verbose=1, extra_argv=None, doctests=False, coverage=False, **kwargs): from testing import Tester Tester.test(label, verbose, extra_argv, doctests, coverage, **kwargs and do something similar for 'bench'. Note that numpy currently implements numpy.test <-- this is a Tester().test numpy.testing.test <-- another Tester().test bound method so there's some needless and distracting, but extremely minor, duplication.

...

Getting rid of these functions, and thus getting rid of the import speeds numpy startup time by 3.5%.

While I appreciate you taking the time to find these niggles, but we are short on developer time as it is. Asking them to spend their precious time on making a 3.5% improvement in startup time does not make much sense. If you provide a patch, on the other hand, it would only take a matter of seconds to decide whether to apply or not. You've already done most of the sleuth work.

I wrote that I don't know the reasons for why the design was as it is. Are those functions ("english_upper", "english_lower", "english_capitalize") expected as part of the public interface for the module? The lack of a "_" prefix and their verbose docstrings implies that they are for general use. In that case, they can't easily be gotten rid of. Yet it doesn't make sense for them to be part of 'numerictypes'. Why would I submit a patch if there's no way those definitions will disappear, for reasons I am not aware of? I am not asking you all to make these changes. I'm asking about how much change is acceptable, what are the restrictions, and why are they there? I also haven't yet figured out how to get the regression tests to run, and I'm not going to contribute patches without at least passing that bare minimum. BTW, how do I do that? In the top-level there's a 'test.sh' command but when I run it I get: % mkdir tmp % bash test.sh Running from numpy source directory. Traceback (most recent call last): File "setupscons.py", line 56, in <module> raise DistutilsError('\n'.join(msg)) distutils.errors.DistutilsError: You cannot build numpy with scons without the numscons package (Failure was: No module named numscons) test.sh: line 11: cd: /Users/dalke/cvses/numpy/tmp: No such file or directory and when I run 'nosetests' in the top-level directory I get: ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python intepreter from there. I couldn't find (in a cursory search) instructions for running self- tests or regression tests.

...

I could probably get another 0.05 seconds if I dug around more, but I can't without knowing what use case numpy is trying to achieve. Why are all those ancillary modules (testing, ctypeslib) eagerly loaded when there seems no need for that feature?

Need is relative. You need fast startup time, but most of our users need quick access to whichever functions they want (and often use from an interactive terminal). I agree that "testing" and "ctypeslib" do not belong in that category, but they don't seem to do much harm either.

If there is no need for those features then I'll submit a patch to remove them. There is some need, and there are many ways to handle that need. The current solution in numpy is to import everything. Again I ask, does *everything* (like 'testing' and 'ctypeslib') need to be imported eagerly? In your use case of user-driven exploratory development the answer is no - the users described above rarely desire access to those package because those packages are best used in automated environments. Eg, why write tests which are only used once? Andrew dalke@dalkescientific.com

On Jul 31, 2008, at 3:53 AM, David Cournapeau wrote:

You are supposed to run the tests on an installed numpy, not in the sources:

import numpy numpy.test(verbose = 10)

Doesn't that make things more cumbersome to test? That is, if I were to make a change I would need to: - python setup.py build (to put the code into the build/* subdirectory) - cd the build directory, or switch to a terminal which was already there - manually do the import/test code you wrote, or a write two-line program for it I would rather do 'nosetests' in the source tree, if at all feasible, although that might only be possible for the Python source. Hmm. And it looks like testing/nosetester.py (which implements the 'test' function above) is meant to make it easier to run nose, except my feeling is the extra level of wrapping makes things more complicated. The nosetest command-line appears to be more flexible, with support for, for examples, dropping into the debugger on errors, and reseting the coverage test files. I'm speaking out of ignorance, btw. Cheers, Andrew dalke@dalkescientific.com

On Thu, Jul 31, 2008 at 03:41:15PM +0900, David Cournapeau wrote:

Yes. Nothing that an easy make file cannot solve, nonetheless (I am sure I am not the only one with a makefile/script which automates the above, to test a new svn updated numpy in one command).

That's why distutils have a test target. You can do "python setup.py test", and if you have setup you setup.py properly it should work (obviously it is easy to make this statement, and harder to get the thing working). Gaël

On Thu, Jul 31, 2008 at 11:05:33PM +0900, David Cournapeau wrote:

Gael Varoquaux wrote:

...

That's why distutils have a test target. You can do "python setup.py test", and if you have setup you setup.py properly it should work (obviously it is easy to make this statement, and harder to get the thing working).

I have already seen some discussion about distutils like this, if you mean something like this:

http://blog.ianbicking.org/pythons-makefile.html

but I would take with rake and make over this anytime. I just don't understand why something like rake does not exist in python, but well, let's not go there.

Well, actually, in the enthought tools suite we use setuptools for packaging (I don't want to start a controversy, I am not advocating the use of setuptools, just stating a fact) and nose for testing, and getting "setup.py test" to wrok, including do the build test and download nose if not there, is a matter of addig those two lines to the setup.py: tests_require = [ 'nose >= 0.10.3', ], test_suite = 'nose.collector', Obviously, the build part has to be well-tuned for the machinery to work, but there is a lot of value here. Gaël

On Thu, Jul 31, 2008 at 11:16:12PM +0900, David Cournapeau wrote:

Gael Varoquaux wrote:

...

Obviously, the build part has to be well-tuned for the machinery to work, but there is a lot of value here.

Ah yes, setuptools does have this. But this is specific to setuptools, bare distutils does not have this test command, right ?

Dunno, sorry. The scale of my ignore of distutils and related subjects would probably impress you :). Gaël, looking forward to your tutorial on scons.

On Thu, Jul 31, 2008 at 2:12 AM, Andrew Dalke <dalke@dalkescientific.com> wrote:

Hmm. And it looks like testing/nosetester.py (which implements the 'test' function above) is meant to make it easier to run nose, except my feeling is the extra level of wrapping makes things more complicated. The nosetest command-line appears to be more flexible, with support for, for examples, dropping into the debugger on errors, and reseting the coverage test files.

You can actually pass those sorts of options to nose through the extra_argv parameter in test(). That might be a little cumbersome, but (as far as I know) it's something I'm going to do so infrequently it's not a big deal.

On Thu, Jul 31, 2008 at 04:42, Stéfan van der Walt <stefan@sun.ac.za> wrote:

2008/7/31 Andrew Dalke <dalke@dalkescientific.com>:

...

I wrote that I don't know the reasons for why the design was as it is. Are those functions ("english_upper", "english_lower", "english_capitalize") expected as part of the public interface for the module? The lack of a "_" prefix and their verbose docstrings implies that they are for general use. In that case, they can't easily be gotten rid of. Yet it doesn't make sense for them to be part of 'numerictypes'.

Anything underneath numpy.core that is not exposed as numpy.something is not for public consumption.

That said, the reason those particular docstrings are verbose is because I wanted people to know why those functions exist there (e.g. "This is an internal utility function...."). But you still can't remove them since they are being used inside numerictypes. That's why I labeled them "internal utility functions" instead of leaving them with minimal docstrings such that you would have to guess. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Thu, Jul 31, 2008 at 12:43:17PM +0200, Andrew Dalke wrote:

Startup performance has not been a numpy concern. It a concern for me, and it has been (for other packages) a concern for some of my clients.

I am curious, if startup performance is a problem, I guess it is because you are running lots of little scripts where startup time is big compared to run time. Did you think of forking them from an already started process. I had this same problem (with libraries way slower than numpy to load) and used os.fork to a great success. Gaël

On Thu, Jul 31, 2008 at 10:34:04AM -0400, Kevin Jacobs <jacobs@bioinformed.com> wrote:

The morale of this discussion, for me, is that just because _you_ don't care about a particular aspect or feature, doesn't mean that others don't or shouldn't. Your workarounds may not be viable for me and vice-versa. So let's just go with the spirit of open source and encourage those motivated to controbute to do so, provided their suggestions are sensible and do not break code.

I fully agree ehre. And if people improve numpy's startup time with breaking or obfuscating stuff, I am very happy. I was just trying to help :). Yes, the value of open source is that different people improve the same tools to meet different goals, thus we should always keep on open ear to other people's requirements, especially if they come up with high-quality code. Gaël

On Fri, Aug 1, 2008 at 5:02 AM, Robert Kern <robert.kern@gmail.com> wrote:

5) Proxy objects ... I would really like to avoid proxy objects. They have caused fragility in the past.

One recurrent problem around import times optimization is that it is some work to improve it, but it takes one line to destroy it all. For example, inspect import came back, and this alone is ~10-15 % of my import time on mac os x (from ~ 180 to ~160). This would be the main advantage of lazy import; but does it really worth the trouble, since it brings some complexity as you mentionned last time we had this discussion ? Maybe a simple test script to check for known costly import would be enough (running from time to time ?). Maybe ctypes can be loaded "in the fly", too. Those are the two obvious hotspot ( ~ 25 % altogether). with a recent SVN checkout

6) I'm not a fan of having environment variables control the way numpy gets imported, but I'm willing to consider it. For example, I might go for having proxy objects for linalg et al. *only* if a particular environment variable were set. But there had better be a very large improvement in import times.

linalg does not seem to have a huge impact. It is typically much faster to load than ctypeslib or inspect. cheers, David

On Thu, Jul 31, 2008 at 16:09, Andrew Dalke <dalke@dalkescientific.com> wrote:

On Jul 31, 2008, at 10:02 PM, Robert Kern wrote:

...

1) Everything exposed by "from numpy import *" still needs to work.

Does that include numpy.Tester? I don't mean numpy.test() nor numpy.bench().

Does that include numpy.PackageLoader? I don't mean numpy.pkgload.

Probably not. I would consider those to be implementation details that got left in rather than a deliberate API exposure. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Thu, Jul 31, 2008 at 10:02 PM, Robert Kern <robert.kern@gmail.com> wrote:

On Thu, Jul 31, 2008 at 05:43, Andrew Dalke <dalke@dalkescientific.com> wrote:

...

On Jul 31, 2008, at 12:03 PM, Robert Kern wrote:

...

...

But you still can't remove them since they are being used inside numerictypes. That's why I labeled them "internal utility functions" instead of leaving them with minimal docstrings such that you would have to guess.

My proposal is to replace that code with a table mapping the type name to the uppercase/lowercase/capitalized forms, thus eliminating the (small) amount of time needed to import string.

It makes adding new types slightly more difficult.

I know it's a tradeoff.

Probably not a bad one. Write up the patch, and then we'll see how much it affects the import time.

I would much rather that we discuss concrete changes like this rather than rehash the justifications of old decisions. Regardless of the merits about the old decisions (and I agreed with your position at the time), it's a pointless and irrelevant conversation. The decisions were made, and now we have a user base to whom we have promised not to break their code so egregiously again. The relevant conversation is what changes we can make now.

Some general guidelines:

1) Everything exposed by "from numpy import *" still needs to work. a) The layout of everything under numpy.core is an implementation detail. b) _underscored functions and explicitly labeled internal functions can probably be modified. c) Ask about specific functions when in doubt.

2) The improvement in import times should be substantial. Feel free to bundle up the optimizations for consideration.

3) Moving imports from module-level down into the functions where they are used is generally okay if we get a reasonable win from it. The local imports should be commented, explaining that they are made local in order to improve the import times.

4) __import__ hacks are off the table.

5) Proxy objects ... I would really like to avoid proxy objects. They have caused fragility in the past.

6) I'm not a fan of having environment variables control the way numpy gets imported, but I'm willing to consider it. For example, I might go for having proxy objects for linalg et al. *only* if a particular environment variable were set. But there had better be a very large improvement in import times.

I just want to say that I agree with Andrew that slow imports just suck. But it's not really that bad, for example on my system: In [1]: %time import numpy CPU times: user 0.11 s, sys: 0.01 s, total: 0.12 s Wall time: 0.12 s so that's ok. For comparison: In [1]: %time import sympy CPU times: user 0.12 s, sys: 0.02 s, total: 0.14 s Wall time: 0.14 s But I am still unhappy about it, I'd like if the package could import much faster, because it adds up, when you need to import 7 packages like that, it's suddenly 1s and that's just too much. But of course everything within the constrains that Robert has outlined. From the theoretical point of view, I don't understand why python cannot just import numpy (or any other package) immediatelly, and only at the moment the user actually access something, to import it in real. Mercurial uses a lazy import module, that does exactly this. Maybe that's an option? Look into mercurial/demandimport.py. Use it like this: In [1]: import demandimport In [2]: demandimport.enable() In [3]: %time import numpy CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 s That's pretty good, huh? :) Unfortunately, numpy cannot work with lazy import (yet): In [5]: %time from numpy import array ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (17, 0)) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) [skip] /usr/lib/python2.5/site-packages/numpy/lib/index_tricks.py in <module>() 14 import function_base 15 import numpy.core.defmatrix as matrix ---> 16 makemat = matrix.matrix 17 18 # contributed by Stefan van der Walt /home/ondra/ext/sympy/demandimport.pyc in __getattribute__(self, attr) 73 return object.__getattribute__(self, attr) 74 self._load() ---> 75 return getattr(self._module, attr) 76 def __setattr__(self, attr, val): 77 self._load() AttributeError: 'module' object has no attribute 'matrix' BTW, neither can SymPy. However, maybe it shows some possibilities and maybe it's possible to fix numpy to work with such a lazy import. On the other hand, I can imagine it can bring a lot more troubles, so it should probably only be optional. Ondrej

On Jul 31, 2008, at 11:42 AM, Stéfan van der Walt wrote:

Maybe when we're convinced that there is a lot to be gained from making such a change. From my perspective, it doesn't look good:

I) Major code breakage II) Confused users III) More difficult function discovery for beginners

I'm not asking for a change. I fully realize this. I happen to think it's a mistake and there are other ways to have addressed the underlying requirement, but I know that's not going to change. (For example, follow matplotlib approach where there's a special library designed to be imported in interactive use. But I am *not* proposing this change.) I point out that this make numpy different than most other Python packages. Had this not been done then I) would not be a problem, II) is I think a wash, because people starting with numpy will still wonder why

...

...

import PIL PIL.Image Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute 'Image' import PIL.Image PIL.Image <module 'PIL.Image' from '/Library/Frameworks/Python.framework/ Versions/2.5/lib/python2.5/site-packages/PIL/Image.pyc'>

and

...

...

import xml xml.etree Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute 'etree' from xml import etree xml.etree <module 'xml.etree' from '/Library/Frameworks/Python.framework/ Versions/2.5/lib/python2.5/xml/etree/__init__.pyc'>

occur. III) assumes there couldn't have been other solutions. And it assumes that the difficulties are large, which I haven't seen in my experience.

I) Slight improvement in startup speed.

The user base for numpy might be .. 10,000 people? 100,000 people? Let's go with the latter, and assume that with command-line scripts, CGI scripts, and the other programs that people write in order to help do research means that numpy is started on average 10 times a day. 100,000 people * 10 times / day * 0.1 seconds per startup = almost 28 people-hours spent each day waiting for numpy to start. I'm willing to spend a few days to achieve that. Perhaps there's fewer people than I'm estimating. OTOH, perhaps there are more imports of numpy per day. An order of magnitude less time is still a couple of hours each day as the world waits to import all of the numpy libraries. If on average people import numpy 10 times a day and it could be made 0.1 seconds faster then that's 1 second per person per day. If it takes on average 5 minutes to learn to import the module directly and the onus is all on numpy, then after 1 year of use the efficiency has made up for it, and the benefits continue to grow. Slight improvements add up when multiplied by everyone. The goals of numpy when it started aren't going to be the same as when it's a mature, widely used and deployed package.

Andrew dalke@dalkescientific.com

Hi All, I've been reading this discussion with interest. I would just to highlight an alternate use of numpy to interactive use. We have a cluster of machines which process tasks on an individual basis where a master tasks may spawn 600 slave tasks to be processed. These tasks are spread across the cluster and processed as scripts in a individual python thread. Although reducing the process time by 300 seconds for the master task is only about a 1.5% speedup (total time can be i excess of 24000s). We process large number of these tasks in any given year and every little helps! Hanni 2008/7/31 Stéfan van der Walt <stefan@sun.ac.za>

2008/7/31 Andrew Dalke <dalke@dalkescientific.com>:

...

The user base for numpy might be .. 10,000 people? 100,000 people? Let's go with the latter, and assume that with command-line scripts, CGI scripts, and the other programs that people write in order to help do research means that numpy is started on average 10 times a day.

100,000 people * 10 times / day * 0.1 seconds per startup = almost 28 people-hours spent each day waiting for numpy to start.

I don't buy that argument. No single person is agile enough to do anything useful in the half a second or so it takes to start up NumPy. No one is *waiting* for NumPy to start. Just by answering this e-mail I could have (and maybe should have) started NumPy three hundred and sixty times.

I don't want to argue about this, though. Write the patches, file a ticket, and hopefully someone will deem them important enough to apply them.

Stéfan _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

Nathan Bell wrote:

There are other components of NumPy/SciPy that are more worthy of optimization. Given that programmer time is a scarce resource, it's more sensible to direct our efforts towards making the other 98.5% of the computation faster.

To be fair, when I took a look at the problem last month, it took a few of us (Robert and me IIRC) maximum 2 man hours altogether to divide by two numpy import times on linux, without altering at all the API. Maybe there are more things which can be done to get to a more 'flat' profile. cheers, David

Stéfan van der Walt wrote:

No one is *waiting* for NumPy to start.

I am, and probably 10 times, a day, yes. And it's a major issue for CGI, though maybe no one's using that anymore anyway.

Just by answering this e-mail I could have (and maybe should have) started NumPy three hundred and sixty times.

sure, but I like wasting my time on mailing lists.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Andrew Dalke wrote:

If I had my way, remove things like (in numpy/__init__.py)

import linalg import fft import random import ctypeslib import ma

as a side benefit, this might help folks using py2exe, py2app and friends -- as it stands all those sub-modules need to be included in your app bundle regardless of whether they are used. I recall having to explicitly add them by hand, too, though that may have been a matplotlib.numerix issue.

but leave the list of submodules in "__all__" so that "from numpy import *" works.

Of course, no one should be doing that anyway.... ;-) And for what it's worth, I've found myself very frustrated by how long it takes to start up python and import numpy. I often do whip out the interpreter to do something fast, and I didn't used to have to wait for it. On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7 seconds to import numpy! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

David Cournapeau wrote:

Christopher Barker wrote:

...

On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7 seconds to import numpy!

Hot or cold ? If hot, there is something horribly wrong with your setup.

hot -- it takes about 10 cold. I've been wondering about that. time python -c "import numpy" real 0m8.383s user 0m0.320s sys 0m7.805s and similar results if run multiple times in a row. Any idea what could be wrong? I have no clue where to start, though I suppose a complete clean out and re-install of python comes to mind. oh, and this is a dual G5 PPC (which should have a faster disk than your Macbook) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Thu, Jul 31, 2008 at 1:12 PM, Christopher Barker <Chris.Barker@noaa.gov>wrote:

David Cournapeau wrote:

...

Christopher Barker wrote:

...

On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7 seconds to import numpy!

Hot or cold ? If hot, there is something horribly wrong with your setup.

hot -- it takes about 10 cold.

I've been wondering about that.

time python -c "import numpy"

real 0m8.383s user 0m0.320s sys 0m7.805s

and similar results if run multiple times in a row.

Any idea what could be wrong? I have no clue where to start, though I suppose a complete clean out and re-install of python comes to mind.

Is only 'import numpy' slow, or other packages import slowly too ? Are there remote directories in your pythonpath ? Do you have old `eggs` in the site-packages directory that point to remote directories (installed with setuptools developp) ? Try cleaning the site-packages directory. That did the trick for me once. David

oh, and this is a dual G5 PPC (which should have a faster disk than your Macbook)

-Chris

-- Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

David Cournapeau wrote:

...

time python -c "import numpy"

real 0m8.383s user 0m0.320s sys 0m7.805s

What does python -c "import sys; print sys.path" say ?

A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also wondering how the order is determined -- if it looked in site-packages first, it would find numpy a whole lot faster. I also tried: python -v -v -c "import numpy" &>junk2.txt which results in: # installing zipimport hook import zipimport # builtin # installed zipimport hook # trying /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.so # trying /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/sitemodule.so # trying /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.py ... ... And a LOT more: $ grep "# trying" junk2.txt | wc -l 7446 For comaprison: $ python -v -v -c "import sys" &>junk3.txt $ grep "# trying" junk3.txt | wc -l 618 which still seems like a lot. So I think I've found the problem, it's looking in 7446 places ! but why? I suspect the thing to do is to re-install from scratch, and only add in packages I'm really using now. I wonder if making sure all eggs are unzipped would help, too. Thanks for all your help on what is really OT at this point. -Chris $ python -c "import sys; [sys.stdout.write(p+'\n') for p in sys.path]; print len(sys.path) " /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/altgraph-0.6.7-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Pyrex-0.9.5.1a-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Shapely-1.0-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/ipython-0.8.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/py2app-0.4.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/modulegraph-0.7.2.dev_r21-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PyPubSub-3.0a5-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/macholib-1.2.1.dev_r23-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib-0.98.0-py2.5-macosx-10.3-fat.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.3-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Pylons-0.9.7beta5-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Tempita-0.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/WebError-0.8-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/WebOb-0.9.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Mako-0.2.0-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/decorator-2.2.0-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/simplejson-1.8.1-py2.5-macosx-10.3-ppc.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/FormEncode-1.0.1-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PasteScript-1.6.3-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PasteDeploy-1.3.2-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Paste-1.6-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Beaker-0.9.5-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/WebHelpers-0.6dev_20080613-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Routes-1.9-py2.5.egg /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/bdist_mpkg-0.4.3-py2.5.egg /Users/cbarker/HAZMAT/Hazpy/trunk/lib /usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5/site-packages /usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5/site-packages/wx-2.8-mac-unicode /Library/Frameworks/Python.framework/Versions/2.5/lib/python25.zip /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5 /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-darwin /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac/lib-scriptpackages /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PIL /usr/local/lib/wxPython-unicode-2.8.8.1/lib/python2.5 41

...

Any idea what could be wrong? I have no clue where to start, though I suppose a complete clean out and re-install of python comes to mind.

oh, and this is a dual G5 PPC (which should have a faster disk than your Macbook)

disk should not matter. If hot, everything should be in the IO buffer, opening a file is of the order of a few micro seconds (that's certainly the order on Linux; the VM on Mac OS X is likely not as good, but still).

cheers,

David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, Aug 1, 2008 at 11:53, Gael Varoquaux <gael.varoquaux@normalesup.org> wrote:

On Fri, Aug 01, 2008 at 09:18:48AM -0700, Christopher Barker wrote:

...

...

What does python -c "import sys; print sys.path" say ?

...

A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also wondering how the order is determined -- if it looked in site-packages first, it would find numpy a whole lot faster.

AFAIK this is a setuptools issue. From what I hear, it might be fixed in the svn version of setuptools, but they still have to make a release that has this feature.

The two issues I can see are: import path priority, it should be screwed up like it is, and speed. Speed is obviously a hard problem.

...

I suspect the thing to do is to re-install from scratch, and only add in packages I'm really using now.

Avoid eggs if you can. This has been my policy. I am not sure how much this is just superstition or a real problem, though.

Superstition. [~]$ python -c "import sys; print len(sys.path)" 269 [~]$ python -v -v -c "import numpy" 2> foo.txt [~]$ wc -l foo.txt 42500 foo.txt [~]$ time python -c "import numpy" python -c "import numpy" 0.18s user 0.46s system 88% cpu 0.716 total So cut it out. Chris, please profile your import so we actually have some real information to work with instead of prejudices. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

Christopher Barker wrote:

$ time python -c "import numpy"

real 0m0.973s user 0m0.290s sys 0m0.682s

much more in line with what others are getting.

I'll start installing stuff that I actually am using now, and maybe I'll see when (if) it breaks down.

OK, I just installed wxPython, and whoa! time python -c "import numpy" real 0m2.793s user 0m0.294s sys 0m2.494s so it's taking almost two seconds more to import numpy, now that wxPython is installed. I haven't even imported it yet. importing wx isn't as bad: $ time python -c "import wx" real 0m1.589s user 0m0.274s sys 0m1.000s -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Sat, Aug 2, 2008 at 00:06, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:

Christopher Barker wrote:

...

OK, I just installed wxPython, and whoa!

time python -c "import numpy"

real 0m2.793s user 0m0.294s sys 0m2.494s

so it's taking almost two seconds more to import numpy, now that wxPython is installed. I haven't even imported it yet. importing wx isn't as bad:

$ time python -c "import wx"

real 0m1.589s user 0m0.274s sys 0m1.000s

Since numpy wo wx + wc import times adds up to numpy import times, this suggests that numpy may import wx. Which it shouldn't, obviously. There is something strange happening here. Please check wether wx really is imported when you do import numpy:

python -c "import numpy; import sys; print sys.modules"

And if it is, we have to know why it is imported at all when doing import numpy.

It isn't. The problem is on Chris's file system. Whatever is wrong with his file system (Bill Spotz's identical problem suggests too many temporary but unused inodes) increases the traversal of the file system. wx has a .pth file which adds entries to sys.path. Every time one tries to import something, the entries on sys.path are examined for the module. So increasing the number of entries on sys.path exacerbates the problem. But the problem really is his disk; it's not a problem with numpy or Python or anything else. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

Robert Kern wrote:

It isn't. The problem is on Chris's file system.

Thanks for all your help, Robert. Interestingly, I haven't noticed any problems anywhere else, but who knows? I guess this is what Linux Torvalds meant when he said that OS-X's file system was "brain dead"

Whatever is wrong with his file system (Bill Spotz's identical problem suggests too many temporary but unused inodes)

I didn't see anything about Bill having similar issues -- was it on this list?

But the problem really is his disk; it's not a problem with numpy or Python or anything else.

so the question is: what can I do about it? Do I have any other choice than wiping the disk and re-installing? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

OK, So I'm an idiot. After reading this, I thought "I haven't rebooted for a while". It turns out it's been 35 days. I think I've been having slow startup for a longer than that, but none the less, I re-booted (which took a long time), and presto: $ time python -c "import numpy" real 0m0.686s user 0m0.322s sys 0m0.363s much better! I suspect OS-X did some disk-cleaning on re-boot. Frankly, 35 days is pretty pathetic for an uptime, but as I said, I think this issue has been going on longer. Perhaps OS-X runs a disk check every n re-boots, like some linux distros do. Sorry about the noise, and thanks, particularly to Robert, for taking an interest in this. -Chris Robert Kern wrote:

On Mon, Aug 4, 2008 at 14:24, Christopher Barker <Chris.Barker@noaa.gov> wrote:

...

...

It isn't. The problem is on Chris's file system. Thanks for all your help, Robert. Interestingly, I haven't noticed any

Robert Kern wrote: problems anywhere else, but who knows?

I guess this is what Linux Torvalds meant when he said that OS-X's file system was "brain dead"

...

Whatever is wrong with his file system (Bill Spotz's identical problem suggests too many temporary but unused inodes) I didn't see anything about Bill having similar issues -- was it on this list?

...

From my earlier message in this thread:

""" Looking at the Shark results you sent me, it looks like all of your time is getting sucked up by the system call getdirentries(). Googling for some of the function names in that stack brings me to the message "Slow python initialization" on the Pythonmac-SIG:

http://mail.python.org/pipermail/pythonmac-sig/2005-December/015542.html

The ultimate resolution was that Bill Spotz, the original poster, ran Disk Utility and used the Disk Repair option to clean up a large number of unused inodes. This solved the problem for him:

http://mail.python.org/pipermail/pythonmac-sig/2005-December/015548.html """

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

I just added ticket 874 http://scipy.org/scipy/numpy/ticket/874 which on my machine takes the import time from 0.15 seconds down to 0.093 seconds. A bit over a month ago it was about 0.33 seconds. :) The biggest trick I didn't apply was to defer importing the re module, and only compile the patterns when they are needed. That would save about 0.013 seconds, but the result would be rather fragile as people rarely consider deferring pattern definitions. I'll submit a patch for it, expected it to be rejected. ;) I noticed that two places import the standard Python math library, which doesn't seem necessary: numpy/lib/__init__.py: import math __all__ = ['emath','math'] 'math' is never used in that module. It does mean that 'numpy.math' is the same as Python's math module, which I didn't expect. I didn't understand why it was explicitly in the __all_ list so I didn't change it. All tests pass without it. numpy/lib/index_tricks.py: import math .. size.append(math.ceil((key[k].stop - start)/ (step*1.0))) Wouldn't numpy.ceil be okay here? All tests pass when I use numpy.ceil. This would save about 0.002 seconds, so it's a small improvement. I think there's a subtle and minor bug in _datasource. def __del__(self): # Remove temp directories if self._istmpdest: rmtree(self._destpath) Is 'rmtree' guaranteed to be present in the module scope if the object is garbage collected during Python shutdown? Andrew dalke@dalkescientific.com

On Sat, Aug 2, 2008 at 1:18 AM, Christopher Barker <Chris.Barker@noaa.gov> wrote:

A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also wondering how the order is determined -- if it looked in site-packages first, it would find numpy a whole lot faster.

I don't think the number itself is an issue. Putting eggs first is the way it has to be I think, that's just how eggs are supposed to work.

I also tried:

python -v -v -c "import numpy" &>junk2.txt

which results in:

# installing zipimport hook import zipimport # builtin # installed zipimport hook # trying /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.so # trying /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/sitemodule.so # trying /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.py

... ...

And a LOT more:

$ grep "# trying" junk2.txt | wc -l 7446

For comaprison: $ python -v -v -c "import sys" &>junk3.txt $ grep "# trying" junk3.txt | wc -l 618

which still seems like a lot.

So I think I've found the problem, it's looking in 7446 places ! but why?

Part of it is how python looks for modules. Again, I don't think the number itself is the issue: non existing files should not impact much because python import is basically doing a stat, and a stat on a non existing file, in the hot situation, takes nothing. IOW, I don't think the problem is the numbers themselves. It has to be something else. A simple profiling like python -m cProfile -o foo.stats foo.py and then: python -c "import pstats; p = pstats.Stats("foo.stats"); p.sort_stats('cumulative').print_stats(50)" May give useful information. This and using shark as Robert suggested should point to some direction, cheers, David

On Fri, Aug 1, 2008 at 17:22, Christopher Barker <Chris.Barker@noaa.gov> wrote:

David Cournapeau wrote:

...

IOW, I don't think the problem is the numbers themselves. It has to be something else. A simple profiling like

python -m cProfile -o foo.stats foo.py

and then:

python -c "import pstats; p = pstats.Stats("foo.stats"); p.sort_stats('cumulative').print_stats(50)"

OK, see the results -- I think (though i may be wrong) this means that the problem isn't in finding the numpy package:

Can you send foo.stats, too? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Fri, Aug 1, 2008 at 17:50, Christopher Barker <Chris.Barker@noaa.gov> wrote:

Robert Kern wrote:

...

Can you send foo.stats, too?

sure can.

Also, I've got Shark up and running, and have run it on a script that does nothing but import numpy, but really have no idea what I'm looking at, or how to send it to someone that does (you?). I'll keep poking at it some more.

File/Save As..., pick a file name. When asked about whether to embed source files or strip them out, choose Strip. Then email the resulting .mshark file to me. It looks like your Python just takes a truly inordinate amount of time to execute any code. Some of the problematic modules like httplib have been moved to local imports, but the time it takes for your Python to execute the code in that module is still ridiculously large. Can you profile just importing httplib instead of numpy? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Fri, Aug 1, 2008 at 17:50, Christopher Barker <Chris.Barker@noaa.gov> wrote:

Robert Kern wrote:

...

Can you send foo.stats, too?

sure can.

Also, I've got Shark up and running, and have run it on a script that does nothing but import numpy, but really have no idea what I'm looking at, or how to send it to someone that does (you?). I'll keep poking at it some more.

Looking at the Shark results you sent me, it looks like all of your time is getting sucked up by the system call getdirentries(). Googling for some of the function names in that stack brings me to the message "Slow python initialization" on the Pythonmac-SIG: http://mail.python.org/pipermail/pythonmac-sig/2005-December/015542.html The ultimate resolution was that Bill Spotz, the original poster, ran Disk Utility and used the Disk Repair option to clean up a large number of unused inodes. This solved the problem for him: http://mail.python.org/pipermail/pythonmac-sig/2005-December/015548.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Thu, Jul 31, 2008 at 11:45, Christopher Barker <Chris.Barker@noaa.gov> wrote:

On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7 seconds to import numpy!

Can you try running a Python process that just imports numpy under Shark.app? http://developer.apple.com/tools/shark_optimize.html This will help us see what's eating up the time at the C level, at least. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco