Mailman 3 May 2010 - Python-Dev

Re: [Python-Dev] Implementing PEP 382, Namespace Packages
by P.J. Eby May 31, 2010

May 31, 2010

At 06:18 PM 5/30/2010 -0700, Brett Cannon wrote: >On Sun, May 30, 2010 at 00:40, P.J. Eby <pje(a)telecommunity.com> wrote: > > > > Which would completely break one of the major use cases of the > PEP, which is > > precisely to ensure that you can install two pieces of code to the same > > namespace without either one overwriting the other's files. > >The PEP says the goal is to span packages across directories. The goal of namespace packages is to … [View More]allow separately-distributed pieces of code to live in the same package namespace. That this is sometimes achieved by installing them to different paths is an implementation detail. In the case of e.g. Linux distributions and other system packaging scenarios, the code will all be installed to the *same* directory -- so there cannot be any filename collisions among the separately-distributed modules. That's why we want to get rid of the need for an __init__.py to mark the directory as a package: it's a collision point for system package management tools. > > pkgutil doesn't have such a limitation, except in the case extend_path, and > > that limitation is one that PEP 382 intends to remove. > >It's because pkgutil.extend_path has that limitation I am asking as >that's what the PEP refers to. If the PEP wants to remove the >limitation it should clearly state how it is going to do that. I'm flexible on it either way. The only other importer I know of that does anything else is one that actually allows (unsafely) importing from URLs. If we allow for other things, then we need to extend the PEP 302 protocol to have a way to ask an importer for a subpath string. >As for adding to the PEP 302 protocols, it's a question of how much we >want importer implementors to have control over this versus us. I >personally would rather keep any protocol extensions simple and have >import handle as many of the details as possible. I lean the other way a bit, in that the more of the importer internals you expose, the harder you make it for an importer to be anything other than a mere virtual file system. (As it is, I think there is too much "file-ness" coupling in the protocol already, what with file extensions and the like.) Indeed, now that I'm thinking about it, it actually seems to make more sense to just require the importers to implement PEP 382, and provide some common machinery in imp or pkgutil for reading .pth strings, setting up __path__, and hunting down all the other directories. [View Less]

2 1

Re: [Python-Dev] Implementing PEP 382, Namespace Packages
by "Martin v. Löwis" May 31, 2010

May 31, 2010

Am 29.05.2010 21:06, schrieb P.J. Eby: > At 08:45 PM 5/29/2010 +0200, Martin v. Löwis wrote: >>> In it he says that PEP 382 is being deferred until it can address PEP >>> 302 loaders. I can't find any follow-up to this. I don't see any >>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was >>> never resolved. Does it need to be before PEP 382 is implemented? Are we >>> wasting our time by designing and (eventually) coding … [View More]

3 6

Re: [Python-Dev] Implementing PEP 382, Namespace Packages
by P.J. Eby May 31, 2010

May 31, 2010

At 09:24 AM 5/31/2010 +0200, Martin v. Löwis wrote: >Is this really how it works today? Shouldn't it abort here >if there is an ImportError? Oops. You're right - I was confusing find_module with the path_hooks protoocol. > > else: > > # errors here should propagate > > module = loader.load_module(fullname) > > if not hasattr(module, '__path__'): > > # found, but not a package > … [View More]

1 0

Bugfix releases should not change APIs
by Terry Reedy May 31, 2010

May 31, 2010

I had the strong impression that there was a policy that x.y.z bugfix releases should only fix bugs and not add new features and revise current ones. The rationale, as I understood it, is that x.y.z releases should be increasingly better implementations of a stable x.y feature set. Adding features is 'bad' because code using a new feature will not work in previous releases of the same x.y version. Changing features is even worse because it may also break working code going forward. … [View More]Because of this policy, an x.y.z Windows installer (I do not know about others) deletes any earlier release of the same version. Also, there is no What's New in Python x.y.z (relative to x.y.(z-1) since such should be empty. Consequently, violations of the policy are pretty much silent and well hidden. Yesterday, I spent two hours puzzling over the failure of my previously 'green' test sequence that tested a custom test function. I finally realized that the change was not due to anything I did (or undid), but a change in 3.1.2 in the interaction of StringIO.truncate, StringIO.getvalue, and print(x, StringIO()). (I should note the it is the usual stability and quality of Python that made me slow to blame Python rather than myself.) After filing http://bugs.python.org/issue8840 I was rather shocked to be told that the code-breaking and policy-violating change was intentional because being 'more consistent with other file-handling APIs out there' was somehow more important than consistency within version releases. It also seems to me that discussion of code-breaking API changes like this should involve more than one user and one developer. See http://bugs.python.org/issue6939 I have fixed my tests so they works in 3.1.2 and should work in other 3.1 releases, but that would be a nuisance to test. Of course, I should not have to worry about API changes within a version series. I think issue 8840 illustrates another reason for the bugfix-only policy. New x.y features and docs are (nearly always) added before the first beta. They can then be tested, discussed, and improved. This 'cooking' does not occur for bugfix releases. For reasons I give in my response on the tracker, I think the new behavior is buggy and the doc deficient. So, I think 1) the supposed bugfix-only policy should really be the policy; 2) the violation in 3.1.2 should be reverted in 3.1.3, and the API change reviewed in the normal 3.2 alpha/beta process. I am curious as to what others think on both points. Terry Jan Reedy [View Less]

7 9

Re: [Python-Dev] Implementing PEP 382, Namespace Packages
by P.J. Eby May 31, 2010

May 31, 2010

At 03:44 PM 5/29/2010 -0700, Brett Cannon wrote: >On Sat, May 29, 2010 at 12:29, "Martin v. LÃ¶wis" <martin(a)v.loewis.de> wrote: > > Am 29.05.2010 21:06, schrieb P.J. Eby: > >> > >> At 08:45 PM 5/29/2010 +0200, Martin v. LÃ¶wis wrote: > >>>> > >>>> In it he says that PEP 382 is being deferred until it can address PEP > >>>> 302 loaders. I can't find any follow-up to this. I don't see any > >>>> … [View More]discussion in PEP 382 about PEP 302 loaders, so I assume this issue was > >>>> never resolved. Does it need to be before PEP 382 is implemented? Are we > >>>> wasting our time by designing and (eventually) coding before this issue > >>>> is resolved? > >>> > >>> Yes, and yes. > >> > >> Is there anything we can do to help regarding that? > > > > You could comment on the proposal I made back then, or propose a different > > solution. > >[sorry for the fundamental PEP questions, but I think PEP 382 came >about while I was on my python-dev sabbatical last year] > >I have some questions about the PEP which might help clarify how to >handle the API changes. > >For finders, their search algorithm is changed in a couple of ways. >One is that modules are given priority over packages (is that >intentional, Martin, or just an oversight?). Two, the package search >requires checking for a .pth file on top of an __init__.py. This will >change finders that could before simply do an existence check on an >__init__ "file" (or whatever the storage back-end happened to be) and >make it into a list-and-search which one would hope wasn't costly, but >in same cases might be if the paths to files is not stored in a >hierarchical fashion (e.g. zip files list entire files paths in their >TOC or a sqlite3 DB which uses a path for keys will have to list >**all** keys, sort them to just the relevant directory, and then look >for .pth or some such approach). Are we worried about possible >performance implications of this search? No. First, an importer would not be required to implement it in a precisely analagous way; you could have database entries or a special consolidated index in a zipfile, if you wanted to do it like that. (In practice, Python's zipimporter has a memory cache of the TOC, and a simple database index on paths would make a search for .pth's in a subdirectory trivial for the database case.) > I say no, but I just want to >make sure people we are not and people are aware about the design >shift required in finders. This entire worry would be alleviated if >only .pth files named after the package were supported, much like >*.pkg files in pkgutil. Which would completely break one of the major use cases of the PEP, which is precisely to ensure that you can install two pieces of code to the same namespace without either one overwriting the other's files. >And then the search for the __init__.py begins on the newly modified >__path__, which I assume ends with the first __init__ found on >__path__, but if no file is found it's okay and essentially an empty >module with just module-specific attributes is used? In other words, >can a .pth file replace an __init__ file in delineating a package? Yes. >Or is it purely additive? I assume the latter for compatibility reasons, Nope. The idea is specifically to allow separately installed projects to create a package without overwriting any files (causing conflicts for system installers). >but the PEP says "a directory is considered a package if it **either** >contains a file named __init__.py, **or** a file whose name ends with >".pth"" (emphasis mine). Otherwise I assume that the search will be >done simply with ``os.path.isdir(os.path.join(sys_path_entry, >top_level_package_name)`` and all existing paths will be added to >__path__. Will they come before or after the directory where the *.pth >was found? And will any subsequent *.pth files found in other >directories also be executed? > >As for how "*" works, is this limited to top-level packages, or will >sub-packages participate as well? Sub-packages as well. > I assume the former, but it is not >directly stated in the PEP. If the latter, is a dotted package name >changed to ``os.sep.join(sy_path_entry, package_name.replace('".", >os.sep)``? > >For sys.path_hooks, I am assuming import will simply skip over passing >that as it is a marker that __path__ represents a namsepace package >and not in any way functional. Although with sys.namespace_packages, >is leaving the "*" in __path__ truly necessary? I'm going to leave these to Martin to answer. >For the search of paths to use to extend, are we limiting ourselves to >actual file system entries on sys.path (as pkgutil does), pkgutil doesn't have such a limitation, except in the case extend_path, and that limitation is one that PEP 382 intends to remove. >or do we >want to support other storage back-ends? To do the latter I would >suggest having a successful path discovery be when a finder can be >created for the hypothetical directory from sys.path_hooks. The downside to that is that NullImporter is the default importer, so you'd still have to special case it. It would make more sense to add to the PEP 302 protocols directly. >I'll shut up now and stop causing trouble. =) May I suggest you take a look at the implementation draft in my other email? I realize in retrospect it doesn't handle __init__ searching in precisely the order proposed by the PEP, but I'm not sure it would be that difficult to add. (It also needs to split the operation into find/load pieces, but that's also a straightforward mod: just defer the module loading until the end, and return a wrapper around the loader that finishes the process.) [View Less]

3 2

Re: [Python-Dev] Implementing PEP 382, Namespace Packages
by P.J. Eby May 31, 2010

May 31, 2010

At 05:59 PM 5/30/2010 -0700, Brett Cannon wrote: >Is it wise to modify __path__ post-import? Today people can make sure >that __path__ is set to what they want before potentially reading it >in their __init__ module by making the pkgutil.extend_path() call >first. This would actually defer to after the import and thus not >allow any __init__ code to rely on what __path__ eventually becomes. Well, that's what the other lines in the .pth files are for. Keep in mind that only *… [View More]

1 0

variable name resolution in exec is incorrect
by Colin H May 30, 2010

May 30, 2010

Hi, issue991196 was closed being described as intentional. I've added a comment in that issue which argues that this is a serious bug (also aserted by a previous commenter - Armin Rigo), because it creates a unique, undocumented, oddly behaving scope that doesn't apply closures correctly. At the very least I think this should be acknowledged as a plain old bug (rather than a feature), and then a discussion about whether it will be fixed or not. Appreciate your thoughts - cheers, Colin

10 31

PEP 3148 ready for pronouncement
by Brian Quinlan May 29, 2010

May 29, 2010

The PEP is here: http://www.python.org/dev/peps/pep-3148/ I think the PEP is ready for pronouncement, and the code is pretty much ready for submission into py3k (I will have to make some minor changes in the patch like changing the copyright assignment): http://code.google.com/p/pythonfutures/source/browse/#svn/branches/ feedback/python3/futures%3Fstate%3Dclosed The tests are here and pass on W2K, Mac OS X and Linux: http://code.google.com/p/pythonfutures/source/browse/branches/feedback/… [View More]

35 156

Implementing PEP 382, Namespace Packages
by Eric Smith May 29, 2010

May 29, 2010

Last night Barry Warsaw, Jason Coombs, and I met to work on implementing PEP 382. As part of my research, I came across this email from Martin: http://mail.python.org/pipermail/python-dev/2009-May/089316.html In it he says that PEP 382 is being deferred until it can address PEP 302 loaders. I can't find any follow-up to this. I don't see any discussion in PEP 382 about PEP 302 loaders, so I assume this issue was never resolved. Does it need to be before PEP 382 is implemented? Are we … [View More]

2 1

Summary of Python tracker Issues
by Python tracker May 28, 2010

May 28, 2010

ACTIVITY SUMMARY (2010-05-21 - 2010-05-28) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2705 open (+40) / 17957 closed (+21) / 20662 total (+61) Open issues with patches: 1095 Average duration of open issues: 720 days. Median duration of open issues: 500 days. Open Issues Breakdown open 2683 (+40) languishing 12 ( +0) pending 9 ( +0) Issues Created Or … [View More]

1 0