requirements for moving __import__ over to importlib?
I'm going to start this off with the caveat that hg.python.org/sandbox/bcannon#bootstrap_importlib is not completely at feature parity, but getting there shouldn't be hard. There is a FAILING file that has a list of the tests that are not passing because importlib bootstrapping and a comment as to why (I think) they are failing. But no switch would ever happen until the test suite passes. Anyway, to start this conversation I'm going to open with why I think removing most of the C code in Python/import.c and replacing it with importlib/_bootstrap.py is a positive thing. One is maintainability. Antoine mentioned how if change occurs everyone is going to have to be able to fix code in importlib, and that's the point! I don't know about the rest of you but I find Python code easier to work with than C code (and if you don't you might be subscribed to the wrong mailing list =). I would assume the ability to make changes or to fix bugs will be a lot easier with importlib than import.c. So maintainability should be easier when it comes to imports. Two is APIs. PEP 302 introduced this idea of an API for objects that can perform imports so that people can control it, enhance it, introspect it, etc. But as it stands right now, import.c implements none of PEP 302 for any built-in import mechanism. This mostly stems from positive thing #1 I just mentioned. but since I was able to do this code from scratch I was able to design for (and extend) PEP 302 compliance in order to make sure the entire import system was exposed cleanly. This means it is much easier now to write a custom importer for quirky syntax, a different storage mechanism, etc. Third is multi-VM support. IronPython, Jython, and PyPy have all said they would love importlib to become the default import implementation so that all VMs have the same implementation. Some people have even said they will use importlib regardless of what CPython does simply to ease their coding burden, but obviously that still leads to the possibility of subtle semantic differences that would go away if all VMs used the same implementation. So switching would lead to one less possible semantic difference between the various VMs. So, that is the positives. What are the negatives? Performance, of course. Now I'm going to be upfront and say I really did not want to have this performance conversation now as I have done *NO* profiling or analysis of the algorithms used in importlib in order to tune performance (e.g. the function that handles case-sensitivity, which is on the critical path for importing source code, has a platform check which could go away if I instead had platform-specific versions of the function that were assigned to a global variable at startup). I also know that people have a bad habit of latching on to micro-benchmark numbers, especially for something like import which involves startup or can easily be measured. I mean I wrote importlib.test.benchmark to help measure performance changes in any algorithmic changes I might make, but it isn't a real-world benchmark like what Unladen Swallow gave us (e.g. the two start-up benchmarks that use real-world apps -- hg and bzr -- aren't available on Python 3 so only normal_startup and nosite_startup can be used ATM). IOW I really do not look forward to someone saying "importlib is so much slower at importing a module containing ``pass``" when (a) that never happens, and (b) most programs do not spend their time importing but instead doing interesting work. For instance, right now importlib does ``python -c "import decimal"`` (which, BTW, is the largest module in the stdlib) 25% slower on my machine with a pydebug build (a non-debug build would probably be in my favor as I have more Python objects being used in importlib and thus more sanity checks). But if you do something (very) slightly more interesting like ``python -m calendar`` where is a slight amount of work then importlib is currently only 16% slower. So it all depends on how we measure (as usual). So, if there is going to be some baseline performance target I need to hit to make people happy I would prefer to know what that (real-world) benchmark is and what the performance target is going to be on a non-debug build. And if people are not worried about the performance then I'm happy with that as well. =)
Brett, thanks for persevering on importlib! Given how complicated imports are in Python, I really appreciate you pushing this forward. I've been knee deep in both import.c and importlib at various times. ;) On Feb 07, 2012, at 03:07 PM, Brett Cannon wrote:
I think it's *really* critical that importlib be well-documented. Not just its API, but also design documents (what classes are there, and why it's decomposed that way), descriptions of how to extend and subclass, maybe even examples for doing some typical hooks. Maybe even a guided tour or tutorial for people digging into importlib for the first time.
So, that is the positives. What are the negatives? Performance, of course.
That's okay. Get it complete, right, and usable first and then unleash the Pythonic hoards to bang on performance.
Identifying the use cases are important here. For example, even if it were a lot slower, Mailman wouldn't care (*I* might care because it takes longer to run my test, but my users wouldn't). But Bazaar or Mercurial users would care a lot. -Barry
On Tue, Feb 7, 2012 at 21:24, Barry Warsaw <barry@python.org> wrote:
Yeah, startup performance getting worse kinda sucks for command-line apps. And IIRC it's been getting worse over the past few releases... Anyway, I think there was enough of a python3 port for Mercurial (from various GSoC students) that you can probably run some of the very simple commands (like hg parents or hg id), which should be enough for your purposes, right? Cheers, Dirkjan
Le 07/02/2012 23:21, Brett Cannon a écrit :
hg clone http://selenic.com/repo/hg/ cd hg python3 contrib/setup3k.py build
Hi Brett, I think this message went unanswered, so here’s a late reply: Le 07/02/2012 23:21, Brett Cannon a écrit :
# get Mercurial from a repo or tarball hg clone http://selenic.com/repo/hg/ cd hg # convert files in place (don’t commit after this :) python3.2 contrib/setup3k.py # the makefile is not py3k-aware, need to run manually # the current stable head fails with a TypeError for me PYTHONPATH=. python3.2 build/scripts-3.2 Cheers
I just tried this and I get a str/bytes issue. I also think your setup3k.py command is missing ``build`` and your build/scripts-3.2 is missing ``/hg``. On Wed, Feb 22, 2012 at 19:26, Éric Araujo <merwok@netwok.org> wrote:
On Tue, Feb 7, 2012 at 15:24, Barry Warsaw <barry@python.org> wrote:
That's fine and not difficult to do.
Right, which is why I'm looking for some agreed upon, concrete benchmark I can use which isn't fluff. -Brett
On Tue, 7 Feb 2012 15:07:24 -0500 Brett Cannon <brett@python.org> wrote:
From a cursory look, I think you're gonna have to break (special-case) some abstractions and have some inner loop coded in C for the common cases. That said, I think profiling and solving performance issues is critical *before* integrating this work. It doesn't need to be done by you, but the python-dev community shouldn't feel strong-armed to solve the issue.
Well, import time is so important that the Mercurial developers have written an on-demand import mechanism, to reduce the latency of command-line operations. But it's not only important for Mercurial and the like. Even if you're developing a Web app, making imports slower will make restarts slower, and development more tedious in the first place.
- No significant slowdown in startup time. - Within 25% of current performance when importing, say, the "struct" module (Lib/struct.py) from bytecode. Regards Antoine.
On 7 February 2012 20:49, Antoine Pitrou <solipsis@pitrou.net> wrote:
One question here, I guess - does the importlib integration do anything to make writing on-demand import mechanisms easier (I'd suspect not, but you never know...) If it did, then performance issues might be somewhat less of a sticking point, as usual depending on use cases. Paul.
On Tue, Feb 7, 2012 at 16:19, Paul Moore <p.f.moore@gmail.com> wrote:
Depends on what your feature set is. I have a fully working mixin you can add to any loader which makes it lazy if you trigger the import on reading an attribute from the module: http://code.google.com/p/importers/source/browse/importers/lazy.py . But if you want to trigger the import on *writing* an attribute then I have yet to make that work in Python source (maybe people have an idea on how to make that work since __setattr__ doesn't mix well with __getattribute__).
On Tue, Feb 7, 2012 at 15:49, Antoine Pitrou <solipsis@pitrou.net> wrote:
Wouldn't shock me if it came to that, but obviously I would like to try to avoid it.
That part of the discussion I'm staying out of since I want to see this in so I'm biased.
Sure, but they are a somewhat extreme case.
Fine, startup cost from a hard crash I can buy when you are getting 1000 QPS, but development more tedious?
What's significant and measuring what exactly? I mean startup already has a ton of imports as it is, so this would wash out the point of measuring practically anything else for anything small. This is why I said I want a benchmark to target which does actual work since flat-out startup time measures nothing meaningful but busy work. I would get more out of code that just stat'ed every file in Lib since at least that did some work.
- Within 25% of current performance when importing, say, the "struct" module (Lib/struct.py) from bytecode.
Why struct? It's such a small module that it isn't really a typical module. The median file size of Lib is 11K (e.g. tabnanny.py), not 238 bytes (which is barely past Hello World). And is this just importing struct or is this from startup, e.g. ``python -c "import struct"``?
On Tue, 7 Feb 2012 17:16:18 -0500 Brett Cannon <brett@python.org> wrote:
I don't think Mercurial is extreme. Any command-line tool written in Python applies. For example, yum (Fedora's apt-get) is written in Python. And I'm sure many people do small administration scripts in Python. These tools may then be run in a loop by whatever other script.
Well, waiting several seconds when reloading a development server is tedious. Anyway, my point was that other cases (than command-line tools) can be negatively impacted by import time.
I don't understand your sentence. Yes, startup has a ton of imports and that's why I'm fearing it may be negatively impacted :) ("a ton" being a bit less than 50 currently)
"Actual work" can be very small in some cases. For example, if you run "hg branch" I'm quite sure it doesn't do a lot of work except importing many modules and then reading a single file in .hg (the one named ".hg/branch" probably, but I'm not a Mercurial dev). In the absence of more "real world" benchmarks, I think the startup benchmarks in the benchmarks repo are a good baseline. That said you could also install my 3.x port of Twisted here: https://bitbucket.org/pitrou/t3k/ and then run e.g. "python3 bin/trial -h".
I would get more out of code that just stat'ed every file in Lib since at least that did some work.
stat()ing files is not really representative of import work. There are many indirections in the import machinery. (actually, even import.c appears quite slower than a bunch of stat() calls would imply)
Precisely to measure the overhead. Typical module size will vary depending on development style. Some people may prefer writing many small modules. Or they may be using many small libraries, or using libraries that have adoptes such a development style. Measuring the overhead on small modules will make sure we aren't overly confident.
Just importing struct, as with the timeit snippets in the other thread. Regards Antoine.
On Tue, Feb 7, 2012 at 18:08, Antoine Pitrou <solipsis@pitrou.net> wrote:
So you want less than a 50% startup cost on the standard startup benchmarks?
OK, so less than 25% slowdown when importing a module with pre-existing bytecode that is very small. And here I was worrying you were going to suggest easy goals to reach for. ;)
On Wed, 8 Feb 2012 11:07:10 -0500 Brett Cannon <brett@python.org> wrote:
No, ~50 is the number of imports at startup. I think startup time should grow by less than 10%. (even better if it shrinks of course :))
And here I was worrying you were going to suggest easy goals to reach for. ;)
He. Well, if importlib enabled user-level functionality, I guess it could be attractive to trade a slice of performance against it. But from an user's point of view, bootstrapping importlib is mostly an implementation detail with not much of a positive impact. Regards Antoine.
On Tue, Feb 7, 2012 at 3:07 PM, Brett Cannon <brett@python.org> wrote:
One thing I'm a bit worried about is repeated imports, especially ones that are inside frequently-called functions. In today's versions of Python, this is a performance win for "command-line tool platform" systems like Mercurial and PEAK, where you want to delay importing as long as possible, in case the code that needs the import is never called at all... but, if it *is* used, you may still need to use it a lot of times. When writing that kind of code, I usually just unconditionally import inside the function, because the C code check for an already-imported module is faster than the Python "if" statement I'd have to clutter up my otherwise-clean function with. So, in addition to the things other people have mentioned as performance targets, I'd like to keep the slowdown factor low for this type of scenario as well. Specifically, the slowdown shouldn't be so much as to motivate lazy importers like Mercurial and PEAK to need to rewrite in-function imports to do the already-imported check ourselves. ;-) (Disclaimer: I haven't actually seen Mercurial's delayed/dynamic import code, so I can't say for 100% sure if they'd be affected the same way.)
On Tue, 7 Feb 2012 17:24:21 -0500 Brett Cannon <brett@python.org> wrote:
IOW you want the sys.modules case fast, which I will never be able to match compared to C code since that is pure execution with no I/O.
Why wouldn't continue using C code for that? It's trivial (just a dict lookup). Regards Antoine.
On Tue, Feb 7, 2012 at 17:42, Antoine Pitrou <solipsis@pitrou.net> wrote:
Sure, but it's all the code between the function call and hitting sys.modules which would also need to get shoved into the C code. As I said, I have not tried to optimize anything yet (and unfortunately a lot of the upfront costs are over stupid things like checking if __import__ is being called with a string for the module name).
Le mercredi 08 février 2012 à 11:01 -0500, Brett Cannon a écrit :
I guess my point was: why is there a function call in that case? The "import" statement could look up sys.modules directly. Or the built-in __import__ could still be written in C, and only defer to importlib when the module isn't found in sys.modules. Practicality beats purity. Regards Antoine.
On Wed, Feb 8, 2012 at 11:09, Antoine Pitrou <solipsis@pitrou.net> wrote:
Because people like to do wacky stuff with their imports and so fully bypassing __import__ would be bad.
It's a possibility, although that would require every function call to fetch the PyInterpreterState to get at the cached __import__ (so the proper sys and imp modules are used) and I don't know how expensive that would be (probably as not as expensive as calling out to Python code but I'm thinking out loud).
On Thu, Feb 9, 2012 at 2:09 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I quite like the idea of having builtin __import__ be a *very* thin veneer around importlib that just does the "is this in sys.modules already so we can just return it from there?" checks and delegates other more complex cases to Python code in importlib. Poking around in importlib.__import__ [1] (as well as importlib._gcd_import), I'm thinking what we may want to do is break up the logic a bit so that there are multiple helper functions that a C version can call back into so that we can optimise certain simple code paths to not call back into Python at all, and others to only do so selectively. Step 1: separate out the "fromlist" processing from __import__ into a separate helper function def _process_fromlist(module, fromlist): # Perform any required imports as per existing code: # http://hg.python.org/cpython/file/aba513307f78/Lib/importlib/_bootstrap.py#l... Step 2: separate out the relative import resolution from _gcd_import into a separate helper function. def _resolve_relative_name(name, package, level): assert hasattr(name, 'rpartition') assert hasattr(package, 'rpartition') assert level > 0 name = # Recalculate as per the existing code: # http://hg.python.org/cpython/file/aba513307f78/Lib/importlib/_bootstrap.py#l... return name Step 3: Implement builtin __import__ in C (pseudo-code below): def __import__(name, globals={}, locals={}, fromlist=[], level=0): if level > 0: name = importlib._resolve_relative_import(name) try: module = sys.modules[name] except KeyError: # Not cached yet, need to invoke the full import machinery # We already resolved any relative imports though, so # treat it as an absolute import return importlib.__import__(name, globals, locals, fromlist, 0) # Got a hit in the cache, see if there's any more work to do if not fromlist: # Duplicate relevant importlib.__import__ logic as C code # to find the right module to return from sys.modules elif hasattr(module, "__path__"): importlib._process_fromlist(module, fromlist) return module This would then be similar to the way main.c already works when it interacts with runpy - simple cases are handled directly in C, more complex cases get handed over to the Python module. Cheers, Nick. [1] http://hg.python.org/cpython/file/default/Lib/importlib/_bootstrap.py#l950 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Feb 8, 2012 at 20:26, Nick Coghlan <ncoghlan@gmail.com> wrote:
Fine by me.
I was actually already thinking of exposing this as importlib.resolve_name() so breaking it out makes sense. I also think it might be possible to expose a sort of importlib.find_module() that does nothing more than find the loader for a module (if available).
I suspect that if people want the case where you load from bytecode is fast then this will have to expand beyond this to include C functions and/or classes which can be used as accelerators; while this accelerates the common case of sys.modules, this (probably) won't make Antoine happy enough for importing a small module from bytecode (importing large modules like decimal are already fast enough).
On Fri, Feb 10, 2012 at 1:05 AM, Brett Cannon <brett@python.org> wrote:
No, my suggestion of keeping a de minimis C implementation for the builtin __import__ is purely about ensuring the case of repeated imports (especially those nested inside functions) remains as fast as it is today. To speed up *first time* imports (regardless of their origin), I think it makes a lot more sense to use better algorithms at the importlib level, and that's much easier in Python than it is in C. It's not like we've ever been philosophically *opposed* to smarter approaches, it's just that import.c was already hairy enough and we had grave doubts about messing with it too much (I still have immense respect for the effort that Victor put in to sorting out most of its problems with Unicode handling). Not having that millstone hanging around our necks should open up *lots* of avenues for improvement without breaking backwards compatibility (since we can really do what we like, so long as the PEP 302 APIs are still invoked in the right order and the various public APIs remain backwards compatible). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Brett Cannon <brett <at> python.org> writes:
IOW you want the sys.modules case fast, which I will never be able to match compared to C code since that is pure execution with no I/O.
Sure you can: have a really fast Python VM. Constructive: if you can run this code under PyPy it'd be easy to just: $ pypy -mtimeit "import struct" $ pypy -mtimeit -s "import importlib" "importlib.import_module('struct')" Or whatever the right API is. Alex
On Tue, Feb 7, 2012 at 18:26, Alex Gaynor <alex.gaynor@gmail.com> wrote:
I'm not worried about PyPy. =) I assume you will just flat-out use importlib regardless of what happens with CPython since it is/will be fully compatible and is already written for you.
On Tue, Feb 7, 2012 at 5:24 PM, Brett Cannon <brett@python.org> wrote:
Couldn't you just prefix the __import__ function with something like this: ... try: module = sys.modules[name] except KeyError: # slow code path (Admittedly, the import lock is still a problem; initially I thought you could just skip it for this case, but the problem is that another thread could be in the middle of executing the module.)
On Tue, Feb 7, 2012 at 21:27, PJ Eby <pje@telecommunity.com> wrote:
I practically do already. As of right now there are some 'if' checks that come ahead of it that I could shift around to fast path this even more (since who cares about types and such if the module name happens to be in sys.modules), but it isn't that far off as-is.
On 2/7/2012 4:51 PM, PJ Eby wrote:
importlib could provide a parameterized decorator for functions that are the only consumers of an import. It could operate much like this: def imps(mod): def makewrap(f): def wrapped(*args, **kwds): print('first/only call to wrapper') g = globals() g[mod] = __import__(mod) g[f.__name__] = f f(*args, **kwds) wrapped.__name__ = f.__name__ return wrapped return makewrap @imps('itertools') def ic(): print(itertools.count) ic() ic() # first/only call to wrapper <class 'itertools.count'> <class 'itertools.count'> -- Terry Jan Reedy
On Tue, Feb 7, 2012 at 6:40 PM, Terry Reedy <tjreedy@udel.edu> wrote:
If I were going to rewrite code, I'd just use lazy imports (see http://pypi.python.org/pypi/Importing ). They're even faster than this approach (or using plain import statements), as they have zero per-call function call overhead. It's just that not everything I write can depend on Importing. Throw an equivalent into the stdlib, though, and I guess I wouldn't have to worry about dependencies... (To be clearer; I'm talking about the http://peak.telecommunity.com/DevCenter/Importing#lazy-imports feature, which sticks a dummy module subclass instance into sys.modules, whose __gettattribute__ does a reload() of the module, forcing the normal import process to run, after first changing the dummy object's type to something that doesn't have the __getattribute__ any more. This ensures that all accesses after the first one are at normal module attribute access speed. That, and the "whenImported" decorator from Importing would probably be of general stdlib usefulness too.)
On 2/7/2012 9:35 PM, PJ Eby wrote:
My code above and Importing, as I understand it, both delay imports until needed by using a dummy object that gets replaced at first access. (Now that I am reminded, sys.modules is the better place for the dummy objects. I just wanted to show that there is a simple solution (though more specialized) even for existing code.) The cost of delay, which might mean never, is a bit of one-time extra overhead. Both have no extra overhead after the first call. Unless delayed importing is made standard, both require a bit of extra code somewhere.
And that is what I think (agree?) should be done to counteract the likely slowdown from using importlib.
-- Terry Jan Reedy
On Wed, Feb 8, 2012 at 12:54 PM, Terry Reedy <tjreedy@udel.edu> wrote:
Yeah, this is one frequently reinvented wheel that could definitely do with a standard implementation. Christian Heimes made an initial attempt at such a thing years ago with PEP 369, but an importlib based __import__ would let the implementation largely be pure Python (with all the increase in power and flexibility that implies). I'm not sure such an addition would help much with the base interpreter start up time though - most of the modules we bring in are because we're actually using them for some reason. The other thing that shouldn't be underrated here is the value in making the builtin import system PEP 302 compliant from a *documentation* perspective. I've made occasional attempts at fully documenting the import system over the years, and I always end up giving up because the combination of the pre-PEP 302 builtin mechanisms in import.c and the PEP 302 compliant mechanisms for things like zipimport just degenerate into a mess of special cases that are impossible to justify beyond "nobody got around to fixing this yet". The fact that we have an undocumented PEP 302 based reimplementation of imports squirrelled away in pkgutil to make pkgutil and runpy work is sheer insanity (replacing *that* with importlib might actually be a good first step towards full integration). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Feb 7, 2012 at 22:47, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'll see if I can come up with a pure Python way to handle setting attributes on the module since that is the one case that my importers project code can't handle.
It wouldn't. This would be for third-parties only.
I actually have never bothered to explain import as it is currently implemented in any of my PyCon import talks precisely because it is such a mess. It's just easier to explain from a PEP 302 perspective since you can actually comprehend that.
On 2/8/2012 11:13 AM, Brett Cannon wrote:
On Tue, Feb 7, 2012 at 22:47, Nick Coghlan <ncoghlan@gmail.com
It wouldn't. This would be for third-parties only.
such as hg. That is what I had in mind. Would the following work? Treat a function as a 'loop' in that it may be executed repeatedly. Treat 'import x' in a function as what it is, an __import__ call plus a local assignment. Apply a version of the usual optimization: put a sys.modules-based lazy import outside of the function (at the top of the module?) and leave the local assignment "x = sys.modules['x']" in the function. Change sys.modules.__delattr__ to replace a module with a dummy, so the function will still work after a deletion, as it does now. -- Terry Jan Reedy
On 2/8/2012 3:16 PM, Brett Cannon wrote:
The intent of what I proposed it to be transparent for imports within functions. It would be a minor optimization if anything, but it would mean that there is a lazy mechanism in place. For top-level imports, unless *all* are made lazy, then there *must* be some indication in the code of whether to make it lazy or not. -- Terry Jan Reedy
On Wed, Feb 8, 2012 at 4:08 PM, Brett Cannon <brett@python.org> wrote:
There's actually only a few things stopping all imports from being lazy. "from x import y" immediately de-lazies them, after all. ;-) The main two reasons you wouldn't want imports to *always* be lazy are: 1. Changing sys.path or other parameters between the import statement and the actual import 2. ImportErrors are likewise deferred until point-of-use, so conditional importing with try/except would break.
On Thu, Feb 9, 2012 at 11:28 AM, PJ Eby <pje@telecommunity.com> wrote:
3. Module level code may have non-local side effects (e.g. installing codecs, pickle handlers, atexit handlers) A white-listing based approach to lazy imports would let you manage all those issues without having to change all the code that actually *does* the imports. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Feb 8, 2012 at 20:28, PJ Eby <pje@telecommunity.com> wrote:
This actually depends on the type of ImportError. My current solution actually would trigger an ImportError at the import statement if no finder could locate the module. But if some ImportError was raised because of some other issue during load then that would come up at first use.
On Tue, Feb 7, 2012 at 22:47, Nick Coghlan <ncoghlan@gmail.com> wrote [SNIP]
It easily goes beyond runpy. You could ditch much of imp's C code (e.g. load_module()), you could write py_compile and compileall using importlib, you could rewrite zipimport, etc. Anything that touches import could be refactored to (a) use just Python code, and (b) reshare code so as to not re-invent the wheel constantly.
On Wed, Feb 8, 2012 at 11:15, Brett Cannon <brett@python.org> wrote:
And taking it even farther, all of the blackbox aspects of import go away. For instance, the implicit, hidden importers for built-in modules, frozen modules, extensions, and source could actually be set on sys.path_hooks. The Meta path importer that handles sys.path could actually exist on sys.meta_path.
Brett, thanks for persevering on importlib! Given how complicated imports are in Python, I really appreciate you pushing this forward. I've been knee deep in both import.c and importlib at various times. ;) On Feb 07, 2012, at 03:07 PM, Brett Cannon wrote:
I think it's *really* critical that importlib be well-documented. Not just its API, but also design documents (what classes are there, and why it's decomposed that way), descriptions of how to extend and subclass, maybe even examples for doing some typical hooks. Maybe even a guided tour or tutorial for people digging into importlib for the first time.
So, that is the positives. What are the negatives? Performance, of course.
That's okay. Get it complete, right, and usable first and then unleash the Pythonic hoards to bang on performance.
Identifying the use cases are important here. For example, even if it were a lot slower, Mailman wouldn't care (*I* might care because it takes longer to run my test, but my users wouldn't). But Bazaar or Mercurial users would care a lot. -Barry
On Tue, Feb 7, 2012 at 21:24, Barry Warsaw <barry@python.org> wrote:
Yeah, startup performance getting worse kinda sucks for command-line apps. And IIRC it's been getting worse over the past few releases... Anyway, I think there was enough of a python3 port for Mercurial (from various GSoC students) that you can probably run some of the very simple commands (like hg parents or hg id), which should be enough for your purposes, right? Cheers, Dirkjan
Le 07/02/2012 23:21, Brett Cannon a écrit :
hg clone http://selenic.com/repo/hg/ cd hg python3 contrib/setup3k.py build
Hi Brett, I think this message went unanswered, so here’s a late reply: Le 07/02/2012 23:21, Brett Cannon a écrit :
# get Mercurial from a repo or tarball hg clone http://selenic.com/repo/hg/ cd hg # convert files in place (don’t commit after this :) python3.2 contrib/setup3k.py # the makefile is not py3k-aware, need to run manually # the current stable head fails with a TypeError for me PYTHONPATH=. python3.2 build/scripts-3.2 Cheers
I just tried this and I get a str/bytes issue. I also think your setup3k.py command is missing ``build`` and your build/scripts-3.2 is missing ``/hg``. On Wed, Feb 22, 2012 at 19:26, Éric Araujo <merwok@netwok.org> wrote:
On Tue, Feb 7, 2012 at 15:24, Barry Warsaw <barry@python.org> wrote:
That's fine and not difficult to do.
Right, which is why I'm looking for some agreed upon, concrete benchmark I can use which isn't fluff. -Brett
On Tue, 7 Feb 2012 15:07:24 -0500 Brett Cannon <brett@python.org> wrote:
From a cursory look, I think you're gonna have to break (special-case) some abstractions and have some inner loop coded in C for the common cases. That said, I think profiling and solving performance issues is critical *before* integrating this work. It doesn't need to be done by you, but the python-dev community shouldn't feel strong-armed to solve the issue.
Well, import time is so important that the Mercurial developers have written an on-demand import mechanism, to reduce the latency of command-line operations. But it's not only important for Mercurial and the like. Even if you're developing a Web app, making imports slower will make restarts slower, and development more tedious in the first place.
- No significant slowdown in startup time. - Within 25% of current performance when importing, say, the "struct" module (Lib/struct.py) from bytecode. Regards Antoine.
On 7 February 2012 20:49, Antoine Pitrou <solipsis@pitrou.net> wrote:
One question here, I guess - does the importlib integration do anything to make writing on-demand import mechanisms easier (I'd suspect not, but you never know...) If it did, then performance issues might be somewhat less of a sticking point, as usual depending on use cases. Paul.
On Tue, Feb 7, 2012 at 16:19, Paul Moore <p.f.moore@gmail.com> wrote:
Depends on what your feature set is. I have a fully working mixin you can add to any loader which makes it lazy if you trigger the import on reading an attribute from the module: http://code.google.com/p/importers/source/browse/importers/lazy.py . But if you want to trigger the import on *writing* an attribute then I have yet to make that work in Python source (maybe people have an idea on how to make that work since __setattr__ doesn't mix well with __getattribute__).
On Tue, Feb 7, 2012 at 15:49, Antoine Pitrou <solipsis@pitrou.net> wrote:
Wouldn't shock me if it came to that, but obviously I would like to try to avoid it.
That part of the discussion I'm staying out of since I want to see this in so I'm biased.
Sure, but they are a somewhat extreme case.
Fine, startup cost from a hard crash I can buy when you are getting 1000 QPS, but development more tedious?
What's significant and measuring what exactly? I mean startup already has a ton of imports as it is, so this would wash out the point of measuring practically anything else for anything small. This is why I said I want a benchmark to target which does actual work since flat-out startup time measures nothing meaningful but busy work. I would get more out of code that just stat'ed every file in Lib since at least that did some work.
- Within 25% of current performance when importing, say, the "struct" module (Lib/struct.py) from bytecode.
Why struct? It's such a small module that it isn't really a typical module. The median file size of Lib is 11K (e.g. tabnanny.py), not 238 bytes (which is barely past Hello World). And is this just importing struct or is this from startup, e.g. ``python -c "import struct"``?
On Tue, 7 Feb 2012 17:16:18 -0500 Brett Cannon <brett@python.org> wrote:
I don't think Mercurial is extreme. Any command-line tool written in Python applies. For example, yum (Fedora's apt-get) is written in Python. And I'm sure many people do small administration scripts in Python. These tools may then be run in a loop by whatever other script.
Well, waiting several seconds when reloading a development server is tedious. Anyway, my point was that other cases (than command-line tools) can be negatively impacted by import time.
I don't understand your sentence. Yes, startup has a ton of imports and that's why I'm fearing it may be negatively impacted :) ("a ton" being a bit less than 50 currently)
"Actual work" can be very small in some cases. For example, if you run "hg branch" I'm quite sure it doesn't do a lot of work except importing many modules and then reading a single file in .hg (the one named ".hg/branch" probably, but I'm not a Mercurial dev). In the absence of more "real world" benchmarks, I think the startup benchmarks in the benchmarks repo are a good baseline. That said you could also install my 3.x port of Twisted here: https://bitbucket.org/pitrou/t3k/ and then run e.g. "python3 bin/trial -h".
I would get more out of code that just stat'ed every file in Lib since at least that did some work.
stat()ing files is not really representative of import work. There are many indirections in the import machinery. (actually, even import.c appears quite slower than a bunch of stat() calls would imply)
Precisely to measure the overhead. Typical module size will vary depending on development style. Some people may prefer writing many small modules. Or they may be using many small libraries, or using libraries that have adoptes such a development style. Measuring the overhead on small modules will make sure we aren't overly confident.
Just importing struct, as with the timeit snippets in the other thread. Regards Antoine.
On Tue, Feb 7, 2012 at 18:08, Antoine Pitrou <solipsis@pitrou.net> wrote:
So you want less than a 50% startup cost on the standard startup benchmarks?
OK, so less than 25% slowdown when importing a module with pre-existing bytecode that is very small. And here I was worrying you were going to suggest easy goals to reach for. ;)
On Wed, 8 Feb 2012 11:07:10 -0500 Brett Cannon <brett@python.org> wrote:
No, ~50 is the number of imports at startup. I think startup time should grow by less than 10%. (even better if it shrinks of course :))
And here I was worrying you were going to suggest easy goals to reach for. ;)
He. Well, if importlib enabled user-level functionality, I guess it could be attractive to trade a slice of performance against it. But from an user's point of view, bootstrapping importlib is mostly an implementation detail with not much of a positive impact. Regards Antoine.
On Tue, Feb 7, 2012 at 3:07 PM, Brett Cannon <brett@python.org> wrote:
One thing I'm a bit worried about is repeated imports, especially ones that are inside frequently-called functions. In today's versions of Python, this is a performance win for "command-line tool platform" systems like Mercurial and PEAK, where you want to delay importing as long as possible, in case the code that needs the import is never called at all... but, if it *is* used, you may still need to use it a lot of times. When writing that kind of code, I usually just unconditionally import inside the function, because the C code check for an already-imported module is faster than the Python "if" statement I'd have to clutter up my otherwise-clean function with. So, in addition to the things other people have mentioned as performance targets, I'd like to keep the slowdown factor low for this type of scenario as well. Specifically, the slowdown shouldn't be so much as to motivate lazy importers like Mercurial and PEAK to need to rewrite in-function imports to do the already-imported check ourselves. ;-) (Disclaimer: I haven't actually seen Mercurial's delayed/dynamic import code, so I can't say for 100% sure if they'd be affected the same way.)
On Tue, 7 Feb 2012 17:24:21 -0500 Brett Cannon <brett@python.org> wrote:
IOW you want the sys.modules case fast, which I will never be able to match compared to C code since that is pure execution with no I/O.
Why wouldn't continue using C code for that? It's trivial (just a dict lookup). Regards Antoine.
On Tue, Feb 7, 2012 at 17:42, Antoine Pitrou <solipsis@pitrou.net> wrote:
Sure, but it's all the code between the function call and hitting sys.modules which would also need to get shoved into the C code. As I said, I have not tried to optimize anything yet (and unfortunately a lot of the upfront costs are over stupid things like checking if __import__ is being called with a string for the module name).
Le mercredi 08 février 2012 à 11:01 -0500, Brett Cannon a écrit :
I guess my point was: why is there a function call in that case? The "import" statement could look up sys.modules directly. Or the built-in __import__ could still be written in C, and only defer to importlib when the module isn't found in sys.modules. Practicality beats purity. Regards Antoine.
On Wed, Feb 8, 2012 at 11:09, Antoine Pitrou <solipsis@pitrou.net> wrote:
Because people like to do wacky stuff with their imports and so fully bypassing __import__ would be bad.
It's a possibility, although that would require every function call to fetch the PyInterpreterState to get at the cached __import__ (so the proper sys and imp modules are used) and I don't know how expensive that would be (probably as not as expensive as calling out to Python code but I'm thinking out loud).
On Thu, Feb 9, 2012 at 2:09 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I quite like the idea of having builtin __import__ be a *very* thin veneer around importlib that just does the "is this in sys.modules already so we can just return it from there?" checks and delegates other more complex cases to Python code in importlib. Poking around in importlib.__import__ [1] (as well as importlib._gcd_import), I'm thinking what we may want to do is break up the logic a bit so that there are multiple helper functions that a C version can call back into so that we can optimise certain simple code paths to not call back into Python at all, and others to only do so selectively. Step 1: separate out the "fromlist" processing from __import__ into a separate helper function def _process_fromlist(module, fromlist): # Perform any required imports as per existing code: # http://hg.python.org/cpython/file/aba513307f78/Lib/importlib/_bootstrap.py#l... Step 2: separate out the relative import resolution from _gcd_import into a separate helper function. def _resolve_relative_name(name, package, level): assert hasattr(name, 'rpartition') assert hasattr(package, 'rpartition') assert level > 0 name = # Recalculate as per the existing code: # http://hg.python.org/cpython/file/aba513307f78/Lib/importlib/_bootstrap.py#l... return name Step 3: Implement builtin __import__ in C (pseudo-code below): def __import__(name, globals={}, locals={}, fromlist=[], level=0): if level > 0: name = importlib._resolve_relative_import(name) try: module = sys.modules[name] except KeyError: # Not cached yet, need to invoke the full import machinery # We already resolved any relative imports though, so # treat it as an absolute import return importlib.__import__(name, globals, locals, fromlist, 0) # Got a hit in the cache, see if there's any more work to do if not fromlist: # Duplicate relevant importlib.__import__ logic as C code # to find the right module to return from sys.modules elif hasattr(module, "__path__"): importlib._process_fromlist(module, fromlist) return module This would then be similar to the way main.c already works when it interacts with runpy - simple cases are handled directly in C, more complex cases get handed over to the Python module. Cheers, Nick. [1] http://hg.python.org/cpython/file/default/Lib/importlib/_bootstrap.py#l950 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Feb 8, 2012 at 20:26, Nick Coghlan <ncoghlan@gmail.com> wrote:
Fine by me.
I was actually already thinking of exposing this as importlib.resolve_name() so breaking it out makes sense. I also think it might be possible to expose a sort of importlib.find_module() that does nothing more than find the loader for a module (if available).
I suspect that if people want the case where you load from bytecode is fast then this will have to expand beyond this to include C functions and/or classes which can be used as accelerators; while this accelerates the common case of sys.modules, this (probably) won't make Antoine happy enough for importing a small module from bytecode (importing large modules like decimal are already fast enough).
On Fri, Feb 10, 2012 at 1:05 AM, Brett Cannon <brett@python.org> wrote:
No, my suggestion of keeping a de minimis C implementation for the builtin __import__ is purely about ensuring the case of repeated imports (especially those nested inside functions) remains as fast as it is today. To speed up *first time* imports (regardless of their origin), I think it makes a lot more sense to use better algorithms at the importlib level, and that's much easier in Python than it is in C. It's not like we've ever been philosophically *opposed* to smarter approaches, it's just that import.c was already hairy enough and we had grave doubts about messing with it too much (I still have immense respect for the effort that Victor put in to sorting out most of its problems with Unicode handling). Not having that millstone hanging around our necks should open up *lots* of avenues for improvement without breaking backwards compatibility (since we can really do what we like, so long as the PEP 302 APIs are still invoked in the right order and the various public APIs remain backwards compatible). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Brett Cannon <brett <at> python.org> writes:
IOW you want the sys.modules case fast, which I will never be able to match compared to C code since that is pure execution with no I/O.
Sure you can: have a really fast Python VM. Constructive: if you can run this code under PyPy it'd be easy to just: $ pypy -mtimeit "import struct" $ pypy -mtimeit -s "import importlib" "importlib.import_module('struct')" Or whatever the right API is. Alex
On Tue, Feb 7, 2012 at 18:26, Alex Gaynor <alex.gaynor@gmail.com> wrote:
I'm not worried about PyPy. =) I assume you will just flat-out use importlib regardless of what happens with CPython since it is/will be fully compatible and is already written for you.
On Tue, Feb 7, 2012 at 5:24 PM, Brett Cannon <brett@python.org> wrote:
Couldn't you just prefix the __import__ function with something like this: ... try: module = sys.modules[name] except KeyError: # slow code path (Admittedly, the import lock is still a problem; initially I thought you could just skip it for this case, but the problem is that another thread could be in the middle of executing the module.)
On Tue, Feb 7, 2012 at 21:27, PJ Eby <pje@telecommunity.com> wrote:
I practically do already. As of right now there are some 'if' checks that come ahead of it that I could shift around to fast path this even more (since who cares about types and such if the module name happens to be in sys.modules), but it isn't that far off as-is.
On 2/7/2012 4:51 PM, PJ Eby wrote:
importlib could provide a parameterized decorator for functions that are the only consumers of an import. It could operate much like this: def imps(mod): def makewrap(f): def wrapped(*args, **kwds): print('first/only call to wrapper') g = globals() g[mod] = __import__(mod) g[f.__name__] = f f(*args, **kwds) wrapped.__name__ = f.__name__ return wrapped return makewrap @imps('itertools') def ic(): print(itertools.count) ic() ic() # first/only call to wrapper <class 'itertools.count'> <class 'itertools.count'> -- Terry Jan Reedy
On Tue, Feb 7, 2012 at 6:40 PM, Terry Reedy <tjreedy@udel.edu> wrote:
If I were going to rewrite code, I'd just use lazy imports (see http://pypi.python.org/pypi/Importing ). They're even faster than this approach (or using plain import statements), as they have zero per-call function call overhead. It's just that not everything I write can depend on Importing. Throw an equivalent into the stdlib, though, and I guess I wouldn't have to worry about dependencies... (To be clearer; I'm talking about the http://peak.telecommunity.com/DevCenter/Importing#lazy-imports feature, which sticks a dummy module subclass instance into sys.modules, whose __gettattribute__ does a reload() of the module, forcing the normal import process to run, after first changing the dummy object's type to something that doesn't have the __getattribute__ any more. This ensures that all accesses after the first one are at normal module attribute access speed. That, and the "whenImported" decorator from Importing would probably be of general stdlib usefulness too.)
On 2/7/2012 9:35 PM, PJ Eby wrote:
My code above and Importing, as I understand it, both delay imports until needed by using a dummy object that gets replaced at first access. (Now that I am reminded, sys.modules is the better place for the dummy objects. I just wanted to show that there is a simple solution (though more specialized) even for existing code.) The cost of delay, which might mean never, is a bit of one-time extra overhead. Both have no extra overhead after the first call. Unless delayed importing is made standard, both require a bit of extra code somewhere.
And that is what I think (agree?) should be done to counteract the likely slowdown from using importlib.
-- Terry Jan Reedy
On Wed, Feb 8, 2012 at 12:54 PM, Terry Reedy <tjreedy@udel.edu> wrote:
Yeah, this is one frequently reinvented wheel that could definitely do with a standard implementation. Christian Heimes made an initial attempt at such a thing years ago with PEP 369, but an importlib based __import__ would let the implementation largely be pure Python (with all the increase in power and flexibility that implies). I'm not sure such an addition would help much with the base interpreter start up time though - most of the modules we bring in are because we're actually using them for some reason. The other thing that shouldn't be underrated here is the value in making the builtin import system PEP 302 compliant from a *documentation* perspective. I've made occasional attempts at fully documenting the import system over the years, and I always end up giving up because the combination of the pre-PEP 302 builtin mechanisms in import.c and the PEP 302 compliant mechanisms for things like zipimport just degenerate into a mess of special cases that are impossible to justify beyond "nobody got around to fixing this yet". The fact that we have an undocumented PEP 302 based reimplementation of imports squirrelled away in pkgutil to make pkgutil and runpy work is sheer insanity (replacing *that* with importlib might actually be a good first step towards full integration). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Feb 7, 2012 at 22:47, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'll see if I can come up with a pure Python way to handle setting attributes on the module since that is the one case that my importers project code can't handle.
It wouldn't. This would be for third-parties only.
I actually have never bothered to explain import as it is currently implemented in any of my PyCon import talks precisely because it is such a mess. It's just easier to explain from a PEP 302 perspective since you can actually comprehend that.
On 2/8/2012 11:13 AM, Brett Cannon wrote:
On Tue, Feb 7, 2012 at 22:47, Nick Coghlan <ncoghlan@gmail.com
It wouldn't. This would be for third-parties only.
such as hg. That is what I had in mind. Would the following work? Treat a function as a 'loop' in that it may be executed repeatedly. Treat 'import x' in a function as what it is, an __import__ call plus a local assignment. Apply a version of the usual optimization: put a sys.modules-based lazy import outside of the function (at the top of the module?) and leave the local assignment "x = sys.modules['x']" in the function. Change sys.modules.__delattr__ to replace a module with a dummy, so the function will still work after a deletion, as it does now. -- Terry Jan Reedy
On 2/8/2012 3:16 PM, Brett Cannon wrote:
The intent of what I proposed it to be transparent for imports within functions. It would be a minor optimization if anything, but it would mean that there is a lazy mechanism in place. For top-level imports, unless *all* are made lazy, then there *must* be some indication in the code of whether to make it lazy or not. -- Terry Jan Reedy
On Wed, Feb 8, 2012 at 4:08 PM, Brett Cannon <brett@python.org> wrote:
There's actually only a few things stopping all imports from being lazy. "from x import y" immediately de-lazies them, after all. ;-) The main two reasons you wouldn't want imports to *always* be lazy are: 1. Changing sys.path or other parameters between the import statement and the actual import 2. ImportErrors are likewise deferred until point-of-use, so conditional importing with try/except would break.
On Thu, Feb 9, 2012 at 11:28 AM, PJ Eby <pje@telecommunity.com> wrote:
3. Module level code may have non-local side effects (e.g. installing codecs, pickle handlers, atexit handlers) A white-listing based approach to lazy imports would let you manage all those issues without having to change all the code that actually *does* the imports. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Feb 8, 2012 at 20:28, PJ Eby <pje@telecommunity.com> wrote:
This actually depends on the type of ImportError. My current solution actually would trigger an ImportError at the import statement if no finder could locate the module. But if some ImportError was raised because of some other issue during load then that would come up at first use.
On Tue, Feb 7, 2012 at 22:47, Nick Coghlan <ncoghlan@gmail.com> wrote [SNIP]
It easily goes beyond runpy. You could ditch much of imp's C code (e.g. load_module()), you could write py_compile and compileall using importlib, you could rewrite zipimport, etc. Anything that touches import could be refactored to (a) use just Python code, and (b) reshare code so as to not re-invent the wheel constantly.
On Wed, Feb 8, 2012 at 11:15, Brett Cannon <brett@python.org> wrote:
And taking it even farther, all of the blackbox aspects of import go away. For instance, the implicit, hidden importers for built-in modules, frozen modules, extensions, and source could actually be set on sys.path_hooks. The Meta path importer that handles sys.path could actually exist on sys.meta_path.
participants (11)
-
Alex Gaynor -
Antoine Pitrou -
Barry Warsaw -
Brett Cannon -
Dirkjan Ochtman -
Eric Snow -
Nick Coghlan -
Paul Moore -
PJ Eby -
Terry Reedy -
Éric Araujo