Mailman 3 May 2009 - Distutils-SIG

logging into distutils
by A. Cavallo May 6, 2009

May 6, 2009

Hi, I'm playing lately with distutils (on the 2.7a branch). This "sudden" interest has been triggered by the python bugs: http://bugs.python.org/issue5941 http://bugs.python.org/issue5940 What I've found (IMHO) is the lack of proper logging while stepping through all the loops inside the code. In fact distutils.log (the log module used internally) It is loosely written resembling the logging python module but missing the full elegance in the api: is there any reason it could not be made very similar to it? I think this is one of the reason is not more deeply spread into the distutils package (a useful tool in diagnostic, IMHO). Because I'm out of work at the moment (It might not last forever tough) I can do it if there is any interest and send a patch set to be reviewed. Beside this to the people interested I can provide a binary compiled python version with the patches hosted on the suse build server system: http://download.opensuse.org/repositories/home:/cavallo71:/python-opt/ This is totally system transparent so it can be installed and de-installed without impact on the host system. Let me know your toughts please, Antonio cavallo

2 1

Re: [Distutils] Setuptools, namespace packages, --single-version-externally-managed
by P.J. Eby May 6, 2009

May 6, 2009

At 04:35 PM 5/5/2009 -0500, Ian Bicking wrote: >Any ideas on this?Â Phillip? > >On Fri, May 1, 2009 at 5:07 PM, Ian Bicking ><<mailto:ianb@colorstudy.com>ianb(a)colorstudy.com> wrote: >So, a bit of a problem came up with pip and namespace >packages.Â Here's my understanding of what's going on: > >When you install a namespace package with pip, it uses install >--single-version-externally-managed, and generally the namespace >directory is empty and there's a *.nspkg.pth file that has this: > >import sys,new,os; p = >os.path.join(sys._getframe(1).f_locals['sitedir'], >*('<NAMESPACE>',)); ie = >os.path.exists(os.path.join(p,'__init__.py')); m = not ie and >sys.modules.setdefault('zope',new.module('zope')); mp = (m or []) >and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) > >So the lack of an __init__.py file doesn't really matter, because >it's created right there, and has its __path__ added to.Â But >there's a problem when there's another namespace package elsewhere >on the path, that wasn't installed with pip (or Setuptools) and uses >pkgutils.extend_path(__path__, __name__).Â This doesn't get >imported because of that .pth file, and the .pth file doesn't itself >use extend_path, so the path isn't searched.Â This is currently >happening with Zope packages installed with plain distutils, then >another package installed with the zope namespace elsewhere with >pip.Â (When using easy_install, I think >pkg_resource.declare_namespace comes into play somewhere, and this >seems to handle this case, but I'm not sure why the installation is >different with pip.) > >So... what should pip be doing differently to make this work? Heck if I know. IIUC, this should work as long as the namespace declaration gets invoked at some point. But I don't know of any way to ensure that, except for pip to write its own __init__.py (invoking declare_namespace()) and gets rid of the nspkg.pth file.

1 0

Setuptools, namespace packages, --single-version-externally-managed
by Ian Bicking May 5, 2009

May 5, 2009

So, a bit of a problem came up with pip and namespace packages. Here's my understanding of what's going on: When you install a namespace package with pip, it uses install --single-version-externally-managed, and generally the namespace directory is empty and there's a *.nspkg.pth file that has this: import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('<NAMESPACE>',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('zope',new.module('zope')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) So the lack of an __init__.py file doesn't really matter, because it's created right there, and has its __path__ added to. But there's a problem when there's another namespace package elsewhere on the path, that wasn't installed with pip (or Setuptools) and uses pkgutils.extend_path(__path__, __name__). This doesn't get imported because of that .pth file, and the .pth file doesn't itself use extend_path, so the path isn't searched. This is currently happening with Zope packages installed with plain distutils, then another package installed with the zope namespace elsewhere with pip. (When using easy_install, I think pkg_resource.declare_namespace comes into play somewhere, and this seems to handle this case, but I'm not sure why the installation is different with pip.) So... what should pip be doing differently to make this work? -- Ian Bicking | http://blog.ianbicking.org

1 1

Re: [Distutils] Adding entry points into Distutils ?
by P.J. Eby May 5, 2009

May 5, 2009

At 06:57 PM 5/4/2009 -0500, Ian Bicking wrote: >* I'm uncomfortable with the way entry points are scanned.Â I >haven't looked close enough to back it up with numbers, but I think >there's a noticeable performance degradation when the number of >installed packages becomes large.Â (Given the algorithm this would >be expected.) It's linear in the number of entry_points.txt files, yes, but in most apps it should occur at most once, since pkg_resources has a single WorkingSet object holding Distribution objects which cache their entry point data upon first access. There are all sorts of ways you could make different tradeoffs, but this particular set of tradeoffs was optimized for a single-application environment, rather than a massive global shared site-packages where there are plugins for every application on the system. It was also optimized for the zipimport case, because you can tell whether a project has entry points from its cached zip directory, that's needed at startup anyhow.

3 2

Re: [Distutils] Anyone stuck with easy_install / has .pth file issues -- this is for you!
by P.J. Eby May 5, 2009

May 5, 2009

At 01:11 PM 5/4/2009 -0700, Garrett Cooper wrote: >You're right -- it doesn't protect against the following (><): > >/full/path/to/package.egg >./package.egg By duplicates, I meant 'package-1.0.egg' and 'package-1.1.egg', not alternate paths to the same file. (As for the '.' replacement, you might want to look at what I said more closely.) >BTW, wasn't -m deprecated? No. > I mean, -m's `delivered' functionality >doesn't even work You'll need to be more specific there: I don't know what you're saying "doesn't work", let alone *how*, what behavior you expect, or what you're seeing instead. >because python stores all loaded package / module >entries in .pth files as keys in a dictionary _anyhow_ I also don't know what you mean here. .pth file contents are added to sys.path, not a dictionary. Unless you're talking about sys.path_importer_cache or something, in which case I still don't see the connection to anything else at hand. >I have no idea how -xN factor into this though The original docs said to use -m before uninstalling, but they really should say -mxN to exclude scripts and avoid installing (or uninstalling!) dependencies. The docs are broken (or at any rate, suboptimal) and should be fixed. >- The duplicates erasure was just gravy, and purely an exercise on my >part getting used to set's in python. The real point was the missing >package in .pth functionality, caused by easy_install / setuptools >lacking proper MP-safe logic. I'm going to guess that by "MP" you mean multiprocessing, in which case I would say, yes, easy_install assumes that you are not running multiple simultaneous installations to the same directory.

2 1

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby May 5, 2009

May 5, 2009

At 12:03 PM 5/3/2009 +0200, Tarek Ziadé wrote: >The name of each file will have to be normalized: all upper case with >no extensions. > >Any opinions ? I don't see any point to the normalization. However, being able to install arbitrary files in .egg-info is currently supported by setuptools, and that capability is used by e.g. the EggTranslations project. I would therefore suggest that perhaps the ability to specify file trees for egg-info inclusion would be better, if the goal is to eventually replace setuptools.

3 7

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby May 4, 2009

May 4, 2009

At 12:50 AM 5/5/2009 +0200, Tarek Ziadé wrote: >On Mon, May 4, 2009 at 7:51 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > > At 06:01 PM 5/4/2009 +0200, Tarek Ziadé wrote: > >> > >> On Mon, May 4, 2009 at 5:48 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > >> >> > I don't see any point to the normalization. > >> >> > >> >> To avoid different naming conventions like: > >> >> > >> >> PKG-INFO, requires.txt, SOURCES.txt > >> > > >> > And the problem with that is...? > >> > >> inconsistency, but right, it makes no sense if any file/dir can be added > >> there. > >> > >> What about SOURCES.txt btw ? What is the reason to add it ? > > > > It's for source distributions. It allows them to be able to rebuild an > > identical source distribution in the absence of source control metadata. > > > > It's not really necessary for the installation process, although it's used > > to figure out which files to install if you use include_package_data=True. > > > >Any particular reason to call it "SOURCES.txt" ? > >Or we can call it MANIFEST (with '/'-separated relative path) I called it SOURCES.txt because MANIFEST is ambiguous as to what it's a manifest *of*, and also to distinguish it from any user-generated MANIFEST file. (If you don't know about those, you're probably going to break backward compatibility w/somebody, btw. Distutils usage is a diverse collection of nightmares.) Anyway, the name of the file has no bearing on what distutils does, unless distutils is trying to implement the same feature as setuptools: i.e. round-trippable sdists whose original manifest was generated via revision control.

1 0

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby May 4, 2009

May 4, 2009

At 05:23 PM 5/4/2009 +0200, Tarek Ziadé wrote: >There's another point I was thinking about in PEP 376 > >What about dropping the 'egg' part in 'PROJECT.egg-info' ? and >replace it with > >'PROJECT.info' > >(and make the 2.7 version compatible with PROJECT.egg-info ) > >I know it's a minor change, Actually, it's a major change, since it means dropping backward compatibility with existing tools.

5 7

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby May 4, 2009

May 4, 2009

At 06:01 PM 5/4/2009 +0200, Tarek Ziadé wrote: >On Mon, May 4, 2009 at 5:48 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > >> > I don't see any point to the normalization. > >> > >> To avoid different naming conventions like: > >> > >> PKG-INFO, requires.txt, SOURCES.txt > > > > And the problem with that is...? > >inconsistency, but right, it makes no sense if any file/dir can be >added there. > >What about SOURCES.txt btw ? What is the reason to add it ? It's for source distributions. It allows them to be able to rebuild an identical source distribution in the absence of source control metadata. It's not really necessary for the installation process, although it's used to figure out which files to install if you use include_package_data=True.

2 1

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby May 4, 2009

May 4, 2009

At 06:31 PM 5/4/2009 +0200, Tarek Ziadé wrote: >Ok then, we will have to provide extra documentation to make people >understand that the '.egg-info' directory has absolutely nothing to do >with egg-the-format >but is rather a metadata container. On the contrary; .egg-info *is* an egg format; see the EggFormats documentation.

2 1

Re: [Distutils] Anyone stuck with easy_install / has .pth file issues -- this is for you!
by P.J. Eby May 4, 2009

May 4, 2009

At 02:43 AM 5/4/2009 -0700, Garrett Cooper wrote: >Hi guys, > Just thought I'd might provide this script to fellow developers >which fixes .pth files (easy-install.pth / .egg was the prime target >-- see the comments for more details): ><http://yaneurabeya.livejournal.com/3929.html>. > Comments are more than welcome. As far as I can tell, it doesn't do anything that "easy_install -mxN" doesn't, although it appears to also convert paths of this form: /foo/bar/baz/foo/bar/spam into: ./baz./spam if I'm reading the code correctly. It also seems to have no protection against adding multiple versions of the same project to a .pth file, and to ignore development eggs, whether or not their directories still exist. In contrast, easy_install already removes non-existent files/directories whenever it touches easy_install.pth, and if you gave it a command line globbing the same files as this tool (i.e., just "easy_install [list of eggs]"), you'd at least end up without any duplicates in the .pth file. In short, AFAICT, you could replace the entire tool with a short note on how to accomplish the same things using easy_install, or by simply having it invoke easy_install internally.

2 2

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby May 4, 2009

May 4, 2009

At 05:54 PM 5/4/2009 +0200, Tarek Ziadé wrote: >2009/5/4 P.J. Eby <pje(a)telecommunity.com>: > > At 05:23 PM 5/4/2009 +0200, Tarek Ziadé wrote: > >> > >> There's another point I was thinking about in PEP 376 > >> > >> What about dropping the 'egg' part in 'PROJECT.egg-info' ? and replace it > >> with > >> > >> 'PROJECT.info' > >> > >> (and make the 2.7 version compatible with PROJECT.egg-info ) > >> > >> I know it's a minor change, > > > > Actually, it's a major change, since it means dropping backward > > compatibility with existing tools. > >well if Distutils 2.7 / 3.1 keeps backward compatibility like I said, >that would give 3 years for existing >tools to change.. Actually, writing '.info' files means that existing tools that expect .egg-info files or directories would stop working correctly *right away*.

1 0

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby May 4, 2009

May 4, 2009

At 07:34 PM 5/3/2009 +0200, Tarek Ziadé wrote: >2009/5/3 P.J. Eby <pje(a)telecommunity.com>: > > At 12:03 PM 5/3/2009 +0200, Tarek Ziadé wrote: > >> > >> The name of each file will have to be normalized: all upper case with > >> no extensions. > >> > >> Any opinions ? > > > > I don't see any point to the normalization. > >To avoid different naming conventions like: > >PKG-INFO, requires.txt, SOURCES.txt And the problem with that is...? > > However, being able to install > > arbitrary files in .egg-info is currently supported by setuptools, and that > > capability is used by e.g. the EggTranslations project. > > > > I would therefore suggest that perhaps the ability to specify > file trees for > > egg-info inclusion would be better, > >Are you talking about the directory that is built by egg_info in setuptools ? >or something else ? > >I mean, do we want to have subdirectories in .egg-info ? They already exist. > > if the goal is to eventually replace > > setuptools. > >The goal is to provide an install_egg_info command that can be configured >without the need to replace/override it. > >I guess, this can be done by taking part of the code done in setuptools >install_egg_info and provide a way for setuptools or another project >to add more files >through configuration. setuptools simply copies a pre-built egg-info directory from the source tree, that is built by the "egg_info" command. But people can also put arbitrary files and directories there, besides the contents built by the command itself. >Maybe this straight-forward sentence would work better: > >"let's add entry points in Distutils" ? +1

2 1

RFC : PEP 376 - egg.info
by Tarek Ziadé May 4, 2009

May 4, 2009

Hi, I have reworked the PEP a little bit with people feedback. It needs more feedback : http://svn.python.org/projects/peps/trunk/pep-0376.txt - install/uninstall script I think the best solution is not to provide an install script since third-party tool do it. Furthermore, there's already the simplest install script available today: you can run "python setup.py install" on a given package so it gets installed. So what about adding just a global uninstall feature, that uninstalls the files installed for a package, using the record file, and let the third party tool have better features. - MANIFEST (SOURCES.txt) Ronald pointed out that it was not necessary to have a MANIFEST file included if we are going to have a RECORD file, as described in the PEP. The MANIFEST (which in a way is equivalent to SOURCES.txt pip or easy_install adds) is the list of source files, whereas the RECORD is the list if installed files (which might include more elements in case of compilations) What would be the interest of having the list of source files in egg-info ? Cheers Tarek -- Tarek Ziadé | http://ziade.org

4 20

Anyone stuck with easy_install / has .pth file issues -- this is for you!
by Garrett Cooper May 4, 2009

May 4, 2009

Hi guys, Just thought I'd might provide this script to fellow developers which fixes .pth files (easy-install.pth / .egg was the prime target -- see the comments for more details): <http://yaneurabeya.livejournal.com/3929.html>. Comments are more than welcome. If I get any, I'll try to get back in touch sometime before next weekend (my weekdays have been extremely busy lately thanks to a pending work release schedule -_-...). HTH and thanks! -Garrett

1 0

setup.py build_ext --rpath behaviour
by Floris Bruynooghe May 2, 2009

May 2, 2009

Hello Distutils allows you to use a handy --rpath option to the build_ext command to add an RPATH value into the linked object. However some of the semantics are not great IMHO. For the first issue some background first. On SysV-like systems (systems using the ELF binary format) the RPATH field was created in the .dynamic section to allow the shared object to specify extra locations to look for their required shared objects. However this did overwrite the use of the LD_LIBRARY_PATH and that was no good for administrators, so another new field was added: RUNPATH. This was only used *after* LD_LIBRARY_PATH instead of before it solving the problem. When both fields are present the runtime linker ignores RPATH (it is impossible to create a shared object with a RUNPATH but no RPATH). The Solaris linker decided to always encode RUNPATH into the shared object whenever an RPATH is specified (using the -R option). This is a very sensible option most likely the correct behaviour. The GNU linker however decided that backwards compatibility was more important and to make it add the RUNPATH field you have to pass --enable-new-dtags to the linker, if you use -R or -rpath on it's own you will only get an RPATH, no RUNPATH. Now when using the --rpath option to the build_ext command distutils.unixcompiler.UnixCCompiler.runtime_library_dir_option() will use some heuristics to figure out which option to pass to compiler to get the runpath in (i.e. "-R" or "-Wl,-R" etc). I'm going to argue that this needs to be extened pass in -Wl,--enable-new-dtags,-R if the GNU linker is used so that the newer and better RUNPATH gets put into the shared objects all the time. (I don't yet know how to detect the GNU linker but would like a consensus on the desired behaviour before looking into this). Note that this is not completely a disaster, thanks to distutils.sysconfig.customize_compiler() adding the contents of the LDFLAGS environment variable to the command line of the linker invocation you can work around this currently. The second issue with build_ext --rpath is on AIX. Again some background on AIX shared objects, AIX is not SysV-like and uses the XCOFF binary format instead of ELF. Therefore they don't have a RPATH or RUNPATH, but they do have a think called LIBPATH which does something similar. The difference between XCOFF's LIBPATH and ELF's RUNPATH is that AIX's runtime linker does not have a default search path, hence the full search path needs to be encoded into the LIBPATH of the shared object. Now I would propose for build_ext --rpath to encode the LIBPATH when used on AIX since that is the correct thing to do IMHO. (Implementation note: since this is done by passing -blibpath:/opt/example.com/lib:/usr/lib:/lib to the linker note that distutils.unixccompiler.UnixCCompiler.link() would have to be changed not to strip out /usr/lib and /lib from the runtime_library_dirs on AIX. And distutils.unixccompiler.UnixCCompiler.runtime_library_dir_option() would have to use -blibpath:... or -Wl,-blibpath as appropriate) Again, this can currently be circumvented by using the LDFLAGS environment variable. Do these improvements sound sensible? And if so should I create one patch for each (and two bug reports) or combine them into one patch? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

2 3

One Package Per Project
by Brandon Craig Rhodes May 1, 2009

May 1, 2009

I think that, going forward, Python packaging tools (not installation tools; they should remain as they are, for backwards compatibility) should move to supporting only One Package Per Project. And, each project should have the same name as the package inside. In the future, people should have to download an old copy of distutils deliberately if they want to build projects with several packages inside; we should stop releasing tools that support or encourage it. 1. It is easier on developers who want to "import escher" to know that they can simply list "escher" as a dependency instead of having to guess whether it's "Escher" or "EscherProject" or whether it's part of a larger "lithographers" project or whether, heaven forbid, the author decided to redundantly call the project "pyescher". 2. This practice would make PyPI's name make actual sense. It actually claims to be (you can check the site!) the "Python *Package* Index" whereas in fact it's currently nothing of the sort! It's really an index of "projects" that might have zero, one, or several packages inside of them. We should move all projects towards the good behavior of the ones that already name themselves after the single package that they contain. 3. I think the whole idea of putting several packages in a project was useful back when dependencies didn't exist. It made sense, in ancient days, for "ZODB" to include "transaction" because there was no other way to make sure they got installed together. But now that dependencies are possible, there is no longer a need for multiple- package project that outweights the costs involved. 4. The current scheme makes it impossible to choose a "safe" package name when creating and registering a new package. Just because there's no "escher" *project* when you look at PyPI doesn't mean that some project doesn't have an "escher" package hidden inside. You could choose a package name, distribute your product, and only find out later that your users cannot install both your product and another product simultaneously because the other product was, in fact, already using that package name but without your knowing it. -- Brandon Craig Rhodes brandon(a)rhodesmill.org http://rhodesmill.org/brandon

3 2

Re: [Distutils] One Package Per Project
by Ronald Oussoren May 1, 2009

May 1, 2009

I'm -1 on that, for three reasons. 1. I have a number of packages where the PyPI name is not the name of the toplevel package where the two names aren't the same on purpose. An example of this is "pyobjc-framework-WebKit", containing the python package "WebKit". The two don't have the same name because "import WebKit" is more natural during imports, while "pyobjc-framework-WebKit" is clearer in the PyPI listing (you don't have to wonder if this is some cross-platform web toolkit, it's obviously related to PyObjC). 2. For basicly the same reason a number of my PyPI packages have a number of toplevel Python packages. That's again because that makes sense for these packages, and it's furthermore needed for backward compatibility. 3. There actually a good reason for using "pysomelib" instead of "somelib" as the PyPI name for the python bindings for the C library "somelib". Naming the python bindings the same as the base project is confusion, while at the same time "import pysomelib" looks lame in Python code. That said, I agree that there should be a good reason for not using the python package/module name as the PyPI name, and I'm +1 on adding advice to the distutils documentation to keep the two the same. It's also not clear to me what your proposal would mean for namespace packages. Would packages like "zope.interface" be allowed with your proposal? Ronald On Friday, May 01, 2009, at 04:20PM, "Brandon Craig Rhodes" <brandon(a)rhodesmill.org> wrote: >I think that, going forward, Python packaging tools (not installation >tools; they should remain as they are, for backwards compatibility) >should move to supporting only One Package Per Project. And, each >project should have the same name as the package inside. In the future, >people should have to download an old copy of distutils deliberately if >they want to build projects with several packages inside; we should stop >releasing tools that support or encourage it. > > 1. It is easier on developers who want to "import escher" to know that > they can simply list "escher" as a dependency instead of having to > guess whether it's "Escher" or "EscherProject" or whether it's part > of a larger "lithographers" project or whether, heaven forbid, the > author decided to redundantly call the project "pyescher". > > 2. This practice would make PyPI's name make actual sense. It actually > claims to be (you can check the site!) the "Python *Package* Index" > whereas in fact it's currently nothing of the sort! It's really an > index of "projects" that might have zero, one, or several packages > inside of them. We should move all projects towards the good > behavior of the ones that already name themselves after the single > package that they contain. > > 3. I think the whole idea of putting several packages in a project was > useful back when dependencies didn't exist. It made sense, in > ancient days, for "ZODB" to include "transaction" because there was > no other way to make sure they got installed together. But now that > dependencies are possible, there is no longer a need for multiple- > package project that outweights the costs involved. > > 4. The current scheme makes it impossible to choose a "safe" package > name when creating and registering a new package. Just because > there's no "escher" *project* when you look at PyPI doesn't mean > that some project doesn't have an "escher" package hidden inside. > You could choose a package name, distribute your product, and only > find out later that your users cannot install both your product and > another product simultaneously because the other product was, in > fact, already using that package name but without your knowing it. > >-- >Brandon Craig Rhodes brandon(a)rhodesmill.org http://rhodesmill.org/brandon >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG(a)python.org >http://mail.python.org/mailman/listinfo/distutils-sig > >

1 0