[Resent due to a python.org mail server problem] On 2009-04-03 22:07, Martin v. Löwis wrote:
I'd like to extend the proposal to Python 2.7 and later.
I don't object, but I also don't want to propose this, so I added it to the discussion.
My (and perhaps other people's) concern is that 2.7 might well be the last release of the 2.x series. If so, adding this feature to it would make 2.7 an odd special case for users and providers of third party tools.
I certainly hope that we'll see more useful features backported from 3.x to the 2.x series or forward ported from 2.x to 3.x (depending on what the core developer preferences are). Regarding this particular PEP, it is well possible to implement an importer that provides the functionality for Python 2.3-2.7 versions, so it doesn't have to be an odd special case.
That's going to slow down Python package detection a lot - you'd replace an O(1) test with an O(n) scan.
I question that claim. In traditional Unix systems, the file system driver performs a linear search of the directory, so it's rather O(n)-in-kernel vs. O(n)-in-Python. Even for advanced file systems, you need at least O(log n) to determine whether a specific file is in a directory. For all practical purposes, the package directory will fit in a single disk block (containing a single .pkg file, and one or few subpackages), making listdir complete as fast as stat.
On second thought, you're right, it won't be that costly. It requires an os.listdir() scan due to the wildcard approach and in some cases, such a scan may not be possible, e.g. when using frozen packages. Indeed, the freeze mechanism would not even add the .pkg files - it only handles .py file content. The same is true for distutils, MANIFEST generators and other installer mechanisms - it would have to learn to package the .pkg files along with the Python files. Another problem with the .pkg file approach is that the file extension is already in use for e.g. Mac OS X installers. You don't have those issues with the __pkg__.py file approach I suggested.
Wouldn't it be better to stick with a simpler approach and look for "__pkg__.py" files to detect namespace packages using that O(1) check ?
Again - this wouldn't be O(1). More importantly, it breaks system packages, which now again have to deal with the conflicting file names if they want to install all portions into a single location.
True, but since that means changing the package infrastructure, I think it's fair to ask distributors who want to use that approach to also take care of looking into the __pkg__.py files and merging them if necessary. Most of the time the __pkg__.py files will be empty, so that's not really much to ask for.
This would also avoid any issues you'd otherwise run into if you want to maintain this scheme in an importer that doesn't have access to a list of files in a package directory, but is well capable for the checking the existence of a file.
Do you have a specific mechanism in mind?
Yes: frozen modules and imports straight from a web resource. The .pkg file approach requires a directory scan and additional support from all importers. The __pkg__.py approach I suggested can use existing importers without modifications by checking for the existence of such a Python module in an importer managed resource. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 07 2009)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/