Mailman 3 July 2011 - Python-Dev

Re: [Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"
by P.J. Eby July 21, 2011

July 21, 2011

At 12:59 PM 7/21/2011 -0700, Reliable Domains wrote: >I assume that the implicit extend_virtual_paths() would be smart >enough to only do real work if there are virtual packages to do it >in, so much of the performance costs (bunch of stats) are bounded by >the existence of and number of virtual packages that have actually >been imported, correct? Yes - this is true even for an explicit call. It only does this for imported virtual packages, and child virtual packages are … [View More]

1 0

Issue10271 - warnings.showwarning should allow any callable object - request commiter
by lekmalek July 21, 2011

July 21, 2011

Hello all, Can any of you core devs have a look at http://bugs.python.org/issue10271. It seems Brett is really busy right now and this uncontroversial (AFAICT) one liner only needs someone to review it and commit it. The pb is, it's holding me back a little bit, and I really would like to have it in the next 3.2 release if possible. Thanks for your help, lekma

3 4

Re: [Python-Dev] [Python-checkins] cpython: #665194: support roundtripping RFC2822 date stamps in the email.utils module
by Terry Reedy July 21, 2011

July 21, 2011

On 7/20/2011 11:41 AM, r.david.murray wrote: > diff --git a/Lib/email/utils.py b/Lib/email/utils.py > # We need wormarounds for bugs in these methods in older Pythons (see below) Is 'wormaround' (variation of workaround) an intentional play on the fact that some worms prey on other 'bugs' ;-? (rather then eating plant matter?) Terry

4 3

Re: [Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"
by P.J. Eby July 20, 2011

July 20, 2011

At 03:22 PM 7/20/2011 -0600, Eric Snow wrote: >On Wed, Jul 20, 2011 at 2:44 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > > > > So, yeah, actually, that's looking pretty sweet. Basically, we > just have to > > throw a virtual_package_paths dict into the sys module, and do the above > > along with the get_virtual_path() function and add get_subpath() to the > > importer objects, in order to get the PEP's core functionality working. > > >… [View More]

1 0

Re: [Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"
by P.J. Eby July 20, 2011

July 20, 2011

At 01:35 PM 7/20/2011 -0600, Eric Snow wrote: >This is a really nice solution. So a virtual package is not imported >until a submodule of the virtual package is successfully imported Correct... >(except for direct import of pure virtual packages). Not correct. ;-) What we do is avoid creating a parent module or altering its __path__ until a submodule/subpackage import is just about to be successfully completed. See the change I just pushed to the PEP: http://hg.python.org/… [View More]peps/rev/a6f02035c66c Or read the revised Specification section here (which is a bit easier to read than the diff): http://www.python.org/dev/peps/pep-0402/#specification The change is basically that we wait until a successful find_module() happens before creating or tweaking any parent modules. This way, the load_module() part will still see an initialized parent package in sys.modules, and if it does any relative imports, they'll still work. (It *does* mean that if an error happens during load_module(), then future imports of the virtual package will succeed, but I'm okay with that corner case.) >It seems like >sys.virtual_packages should be populated even during a failed >submodule import. Is that right? Yes. In the actual draft, btw, I dubbed it ``sys.virtual_package_paths`` and made it a dictionary. This actually makes the pkgutil.extend_path() code more general: it'll be able to fix the paths of things you haven't actually imported yet. ;-) >Also, it makes sense that the above applies to all virtual packages, >not just pure ones. Well, if the package isn't "pure" then what you've imported is really just an ordinary module, not a package at all. ;-) >When a pure virtual package is directly imported, a new [empty] module >is created and its __path__ is set to the matching value in >sys.virtual_packages. However, an "impure" virtual package is not >created upon direct import, and its __path__ is not updated until a >submodule import is attempted. Even the sys.virtual_packages entry is >not generated until the submodule attempt, since the virtual package >mechanism doesn't kick in until the point that an ImportError is >currently raised. > >This isn't that big a deal, but it would be the one behavioral >difference between the two kinds of virtual packages. So either leave >that one difference, disallow direct import of pure virtual packages, >or attempt to make virtual packages for all non-package imports. That >last one would impose the virtual package overhead on many more >imports so it is probably too impractical. I'm fine with leaving the >one difference. At this point, I've updated the PEP to disallow direct imports of pure virtual packages. AFAICT it's the only approach that ensures you can't get false positive imports by having unrelated-but-similarly-named directories floating around. So, really, there's not a difference, except that you can't import a useless empty module that you have no real business importing in the first place... and I'm fine with that. ;-) >FYI, last night I started on an importlib-based implementation for the >PEP and the above solution would be really easy to incorporate. Well, you might want to double-check that now that I've updated the spec. ;-) In the new approach, you cannot rely on parent modules existing before proceeding to the submodule import. However, I've just glanced at the importlib trunk, and I think I see what you mean. It's already using a recursive approach, rather than an iterative one, so the change should be a lot simpler there than in import.c. There probably just needs to be a pair of functions like: def _get_parent_path(parent): pmod = sys.modules.get(parent) if pmod is None: try: pmod = _gcd_import(parent) except ImportError: # Can't import parent, is it a virtual package? path = imp.get_virtual_path(parent) if not path: # no, allow the parent's import error to propagate raise return path if hasattr(pmod, '__path__'): return pmod.__path__ else: return imp.get_virtual_path(parent) def _get_parent_module(parent): pmod = sys.modules.get(parent) if pmod is None: pmod = sys.modules[parent] = imp.new_module(parent) if '.' in parent: head, _, tail = parent.rpartition('.') setattr(_get_parent_module(head), tail, pmod) if not hasattr(pmod, '__path__'): pmod.__path__ = imp.get_virtual_path(parent) return pmod And then instead of hanging on to parent_module during the import process, you'd just grab a path from _get_parent_path(), and initialize parent_module a little later, i.e.: if parent: path = _get_parent_path(parent) if not path: msg = (_ERR_MSG + '; {} is not a package').format(name, parent) raise ImportError(msg) meta_path = sys.meta_path + _IMPLICIT_META_PATH for finder in meta_path: loader = finder.find_module(name, path) if loader is not None: # ensure parent module exists and is a package before loading parent_module = _get_parent_module(parent) loader.load_module(name) break else: raise ImportError(_ERR_MSG.format(name)) So, yeah, actually, that's looking pretty sweet. Basically, we just have to throw a virtual_package_paths dict into the sys module, and do the above along with the get_virtual_path() function and add get_subpath() to the importer objects, in order to get the PEP's core functionality working. [View Less]

2 1

Re: [Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"
by P.J. Eby July 20, 2011

July 20, 2011

At 08:56 AM 7/20/2011 -0700, Jeff Hardy wrote: >On Tue, Jul 19, 2011 at 8:58 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > > The biggest likely exception to the above would be when a piece of > > code tries to check whether some package is installed by importing > > it. If this is done *only* by importing a top-level module (i.e., not > > checking for a ``__version__`` or some other attribute), *and* there > > is a directory of the same name as the … [View More]sought-for package on > > ``sys.path`` somewhere, *and* the package is not actually installed, > > then such code could *perhaps* be fooled into thinking a package is > > installed that really isn't. > >This part worries me slightly. Imagine a program as such: > >datagen.py >json/foo.js >json/bar.js > >datagen.py uses the files in json/ to generate sample data for a >database. In datagen.py is the following code: > >try: > import json >except ImportError: > import simplejson as json > >Currently, this works just fine, but if will break (as I understand >it) under the PEP because the json directory will become a virtual >package and no ImportError will be raised. Well, it won't fail as long if there actually *is* a json module or package on the path. ;-) But I do see your point. >Is there a mitigation for this in the PEP that I've missed? A possible mitigation would be to require that get_subpath() only return a directory name if that directory in fact contains importable modules somewhere. This is actually discussed a bit later as an open issue under "Implementation Notes", indicating that iter_modules() has this issue as well. The main open questions in doing this kind of checking have to do with recursion: it's perfectly valid to have say, a 'zc/' directory whose only content is a 'buildout/' subdirectory. Of course, it still wouldn't help if the 'json/' subdirectory in your example did contain .py files. There is another possibility, though: What if we change the logic for pure-virtual package creation so that the parent module is created *if and only if* a child module is found? In that case, trying to import a pure virtual 'zc' package would fail, but importing 'zc.buildout' would succeed as long as there was a zc/buildout.py or a zc/buildout/__init__.py somewhere. And in your example, 'import json' would fail -- which is to say, succeed. ;-) This is a minor change to the spec, though perhaps a bit hairier to implement in practice. The current import.c loop over the module name parts (iterating over say, 'zc', then 'buildout', and importing them in turn) would need to be reworked so that it could either roll back the virtual package creation in the event of sub-import failure or conversely delay creation of the parent package(s) until a sub-import finds a module. I certainly think it's *doable*, mind you, but I'd hate to have to do it in C. ;-) Hm. Here's another variant that might be easier to implement (even in C), and could offer some other advantages as well. Suppose we replace the sys.virtual_packages set() with a sys.virtual_paths dict(): a dictionary that maps from module names to __path__ lists, and that's populated by the __path__ creation algorithm described in the PEP. (An empty list would mean that __path__ creation failed for that module/package name.) Now, if a module doesn't have a __path__ (or doesn't exist), we look in sys.virtual_paths for the module name. If the retrieved list is empty, we fail the import. If it's not, we proceed... but *don't* create a module or set the existing module's __path__. Then, at the point where an import succeeds, and we're going to set an attribute on the parent module, we recursively construct parent modules and set their __path__ attributes from sys.virtual_paths, if a module doesn't exist in sys.path, or its __path__ isn't set. Voila. Now there are fewer introspection problems as well: trying to 'import json.foo' when there's no 'foo.py' in any json/ directory will *not* create an empty 'json' package in sys.modules as a side-effect. And it won't add a __path__ to the 'json' module if there were a json.py found, either. What's more, since importing a pure virtual package now fails unless you've successfully imported something from it before, it makes more sense for it to not have a __file__, or a __file__ of None. Actually, it's too bad that we have to have parent packages in sys.modules, or I'd suggest we just make pure virtual packages unimportable, period. Technically, we *could* always create dummy parent modules for virtual packages and *not* put them in sys.modules, but I'm not sure if that's a good idea. It would be more consistent in some ways with the idea that virtual packages are not directly importable, but an interesting side effect would be that if module A does: import foo.bar and module B does: import foo.baz Then module A's version of 'foo' has *only* a 'bar' attribute and B's version has *only* a 'baz' attribute. This could be considered a good thing, a bad thing, or a weird thing, depending on how you look at it. ;-) Probably, we should stick with the current shared 'foo' instance, even for pure virtual packages. It's just that 'foo' should not exist in sys.packages until one of the above imports succeeds. Anyway, thanks for bringing this issue up, because now we can fix the hole *entirely*. If pure virtual packages can never be imported directly, then they can *never* create false positive imports -- and the "Backward Compatibility" part of the PEP gets shorter. ;-) Hurray! (I'm tempted to run off and tweak the PEP for this right now, but I want to see if any of the folks who'd be doing the actual 3.x implementation of this want to weigh in on the details first.) [View Less]

3 2

Re: [Python-Dev] [Python-checkins] peps: Restore whitespace characters lost via email transmission.
by Éric Araujo July 20, 2011

July 20, 2011

Le 20/07/2011 16:40, P.J. Eby a écrit : > PEP 12 prescribes four-space indents for PEPs, actually, wherever it > mentions an specific indentation depth. Ah, thanks. I see in my docutils docs that David Goodger used three and four-space indents (three for indenting the body of a directive, four for other cases). > (I'm a bit surprised that three-space indents are mandated for > anything involving documenting Python in reST, though, since that > would mean you'd also have to … [View More]

2 1

Re: [Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"
by P.J. Eby July 20, 2011

July 20, 2011

At 12:37 PM 7/20/2011 -0400, Erik wrote: >The best solution I can think of would be to have a way for a module >to mark itself as "finalized" (I'm not sure if that's the best >term--just the first that popped into my head). This would prevent >its __path__ from being created or extended in any way. For example, >if the json module contains `__finalized__ = True` or something of the >like, any `import json.foo` would immediately fail. That wouldn't actually fix the problem … [View More]

1 0

Re: [Python-Dev] [Python-checkins] peps: Restore whitespace characters lost via email transmission.
by P.J. Eby July 20, 2011

July 20, 2011

At 04:21 PM 7/20/2011 +0200, Ãric Araujo wrote: >FYI, reST uses three-space indents, not four (so that blocks align >nicely under the leading two dots + one space), so I think the change >was intentional. The âDocumenting Pythonâ guide tells this (in the >standard docs), and I think it applies to PEPs too. PEP 12 prescribes four-space indents for PEPs, actually, wherever it mentions an specific indentation depth. Also, a formfeed character was lost, not just the leading … [View More]

1 0

Re: [Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"
by P.J. Eby July 20, 2011

July 20, 2011

At 06:46 PM 7/20/2011 +1000, Nick Coghlan wrote: >On Wed, Jul 20, 2011 at 1:58 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > > So, without further ado, here it is: > >I pushed this version up to the PEPs repo, so it now has a number >(402) and can be read in prettier HTML format: >http://www.python.org/dev/peps/pep-0402/ Technically, shouldn't this be a 3XXX series PEP? Or are we not doing those any more now that all PEPs would be 3XXX?

3 2