Mailman 3 May 2009 - Distutils-SIG

ImportError running 'test' on namespace package - other package in namespace not found
by Jason R. Coombs 17 Apr '20

17 Apr '20

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

3 6

Upgrading dependencies
by Charlie Moad 29 Apr '18

29 Apr '18

Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie

2 1

Changes to --find-links and --upgrade in SVN
by Phillip J. Eby 29 Apr '18

29 Apr '18

Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file.

2 1

Problems with svn in windows
by Sharky On PTNet 29 Apr '18

29 Apr '18

Hi, I´m trying to do "easy_install setuptools==dev" in a windows box but have an error with svn command. """... 'svn' is not recognized as an internal or external command, ...""" I have installed TortoiseSVN but it seems that don't have command line, is all with the Explorer. Can you tell me what client I may use in windows? Thanks for all, Helio Pereira Sharky @ PTNet

3 2

local development scenarios
by tgreenwood 29 Apr '18

29 Apr '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to distutils/and the egg system. I have not researched the archives as much as I should, so please excuse me if this is familiar territory... Q: Is it possible for dependency checks to work for non pipy deployed eggs? - A.egg - B.egg (setup_requires and install_requires A.egg) - email A and B eggs to friend - tell friend to : easy_install B - but B.egg does not install A...in fact, in my test case, it fails b/c it cannot find A in pipy - in this example, what happens if friend does not have net connection on target machine? how can they install from the eggs alone and get the dep checking to work? Q: How does my friend run the unittests on the egg that they have just installed? As I am developing, I frequently run the unittests and after every bdist_egg build, I also run the unit tests that way, too. How can the consumer do the same? Q: One of my apps is a command line app, say foo.py is a command line app that takes arg1, arg2,... How can the user run this app when it is encased in an egg? Q: How do I define my own pipy? While I like the idea of deploying there for the overall community, I don't want to send my half baked code into the wild. But I'd like to try the general functionality of having a deploy site for my beta customers. How can I set this up? Thanks in advance. - -Todd -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDl9Hfz6uXX4lQc/URAnY4AJ47DFHL/cpEltmqoUyGqzhsK5FjRACgt01v akc3Wh3IOXVv2jiS8TszhuE= =Rv/x -----END PGP SIGNATURE-----

5 12

Choice of dependencies?
by John J Lee 29 Apr '18

29 Apr '18

Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. I know I could (ab)use extras_require, but in my case this is a real dependency rather than an optional extra. The other option I have is to just pick one of the dependencies (B or C in my example above), and thereby force people to install it even though it's not strictly required. Neither option seems particularly attractive. John

3 6

setuptools doc patch
by John J Lee 29 Apr '18

29 Apr '18

The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John

2 2

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann 29 Apr '18

29 Apr '18

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Sample setup.py for Numeric Python
by Greg Ward 29 Apr '18

29 Apr '18

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Can now compile simple extensions
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Re: [Distutils] how to easily consume just the parts of eggs that are good for you
by Stanley A. Klein 09 Oct '09

09 Oct '09

On Wed, April 9, 2008 10:05 pm, Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > Message: 4 > Date: Thu, 10 Apr 2008 12:59:39 +1200 > From: Greg Ewing <greg.ewing(a)canterbury.ac.nz> > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: distutils-sig(a)python.org > > Paul Moore wrote: > >> I believe that Mac OS X goes for an even simpler structure - >> applications store *everything* in the one directory, so that >> install/uninstall is simply a directory copy/remove. > > Yep, and thereby cuts the whole gordian knot, throws the > pieces on the fire and burns them. :-) > > Package managers have always seemed to me to be an > excessively complex solution to a problem that needn't > have existed in the first place. > > I keep hoping that someday Linux will support something > like MacOSX application bundles and frameworks, but I > haven't seen any sign of it yet. > I think this discussion arises because the operating systems we are dealing with are based on very different concepts. Windows and Mac are fundamentally single user systems that have added capabilities for multiple users and are intended to be used with proprietary software. Those considerations lead to minimal dependencies among packages (each proprietary provider needs to control its own package, except for the OS), individual users serving as their own sysadmins, and similar factors. Any dependencies in the proprietary software are hidden from the user because the provider has compiled the dependencies into the binary code they supply. Unix/Linux are fundamentally multi-user systems with a distinct role for a sysadmin. Linux and the BSD's are intended to be used with Free/Open Source Software (FOSS), and Unix originated the Software Tools concept in which individual, relatively simple, separately-developed tools are combined to perform complex tasks. Both FOSS and the Software Tools concept encourage dependencies. The need for package managers arises out of the Unix FHS. If you look at the FHS, it is clearly designed to simplify the job of the sysadmin in a multi-user system that uses FOSS and Software Tools. For example, deciding what to backup and how often to do it in a Unix/Linux system is relatively easy for the sysadmin. All the installed software is in certain top-level directories. All the config files are in /etc. All the logs. caches, spools, web pages, process locks, and certain other data are in /var. All the user data is in /home or its sysadmin-determined equivalents. If the sysadmin needs space, deleting files in /tmp will not cause a problem. In summary, Python is being used on systems that have very different underlying OS use cases. To some extent, the natural use case for Python is closest to that of Linux/Unix. Running Python on Windows/Mac requires adapting for those platforms some of the kinds of tools that simplify operations on Linux/Unix systems. This discussion is essentially about how far that goes, how to accomplish it, and how to remain compatible with the existing tools on Linux/Unix. Stan Klein

4 4

Low Level API for translating distutils/setuptools metatdata to Debian metadata
by Brian Sutherland 16 Aug '09

16 Aug '09

Hi, I've just published a very small library that does three things so far: * Provides a mapping between python project names and Debian binary/source package names * Converts setuptools versions to Debian versions while maintaining sort order * Can introspect an .egg-info directory to figure out the Debian dependencies for use in the debian/control file. It can also handle/understand extras (I Hope!) It's basically an attempt to find a common ground between all the projects doing automated Python->Debian packaging. I have a feeling everyone is re-implementing this code all the time. The code is here: http://pypi.python.org/pypi/van.pydeb/ And an example of direct use in a packaging situation with complex dependencies is here: http://svn.debian.org/viewsvn/pkg-zope/zope.component/trunk/debian/rules?vi… -- Brian Sutherland

5 10

Re: [Distutils] [Catalog-sig] Setuptools + Subversion 1.6 - new setuptools release necessary
by Chris Withers 18 Jun '09

18 Jun '09

Andreas Jung wrote: > it is known that the latest setuptools version produces broken > packages with SVN 1.6 checkouts. Could we get a fixed setuptools > version asap - fixing this issue is essential > (there is some patch floating around). I think you meant this to go to the distutils sig... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

4 4

add new compiler to distutils
by patrick flaherty 16 Jun '09

16 Jun '09

Hi, Relatively new to python (and very new to distutils) but experienced programmer. I'm trying to do a 64 bit build of http://sourceforge.net/projects/pyopenssl/ I'm on Windows Server 2008 R2. First had to acquaint myself with mingw32. That built but the DLLs (or rather PYDs) that are created won't load at runtime. I transferred the build to a 32 bit machine; the PYDs loaded; and the program ran successfully. Thus, as I said, I'm looking to rebuild in 64 bits. It's seems that there's a 64 bit version of mingw in development on sourceforge: http://sourceforge.net/projects/mingw-w64. You download a toolchain; add that to your Windows PATH; and I'm able to build a sample hello.c successfully. The problem is with Python and setup.py. When I try to specify compiler i86_64-mingw32 or some variant to setup.py, I get the following: > L:\pf\Python\pyOpenSSL>setup.py build_ext -Ic:\openssl\include -c > x86_64-pc-mingw32 > running build_ext > error: don't know how to compile C/C++ code on platform 'nt' with > 'x86_64-pc-mingw32' compiler So after fumbling around a bit, where I've arrived at is that in the distutils package ( c:\python26\lib\distutils ) there are various files such as msvccompiler.py, msvccompiler9.py (I have VS 2008 installed), ccompiler.py and cygwincompiler.py that would seem to indicate that 'new' compilers have to have support built into distutils. Is this correct, and if so, how does one go about the task of adding a new compiler? thanx - pat

2 1

Utility to mirror an egg index from a Debian distribution
by Brian Sutherland 16 Jun '09

16 Jun '09

Hi, It may seem like a backwards way of doing things, but I have a need for a utility that can maintain a python package index mirror of a Debian repository. The basic idea is to extract the tarballs of Python packages from the Debian repository and rename them to the original setuptools name. It should also create a buildout-compatible versions file of the versions in the repository. My current implementation idea is to unpack the tarball and use the egg-metadata to figure out what the "egg" name of the tarball should be. Does such a tool exists? If not, I'll probably start working on one on svn.zope.org Real Soon Now (TM). Comments, suggestions much appreciated! -- Brian Sutherland

4 14

Packaging: bdist_deb patches
by Gerry Reno 10 Jun '09

10 Jun '09

Tarek, Speaking of packaging patches. Would there be any chance that you could get the 'bdist_deb' patches integrated in Distutils prior to PyCon? Regards, Gerry

11 23

Version comparison - round 2
by Tarek Ziadé 09 Jun '09

09 Jun '09

Hello, I have gathered all the documentation for the versioning lib in a single document, http://bitbucket.org/tarek/distutilsversion/src/tip/README.txt It explains how the existing tools work and how "verlib", that we built during Pycon, works. It still needs to be completed by some people that were present, but a new feedback round is more than welcome. The goal here is to provide a version comparison standard to be included in Distutils. We do need such a standard to be able to include the "install_requires" metadata as planned (that will be a change in PEP 345) to be able to provide a standard for version comparisons. Cheers Tarek -- Tarek Ziadé | http://ziade.org

6 21

PEP 376 up-to-date
by Tarek Ziadé 09 Jun '09

09 Jun '09

Hello, I have updated PEP 376, mainly with Phillip's feedback, and changed the API section so it looks like the current prototype. - PEP : http://svn.python.org/view/peps/trunk/pep-0376.txt?view=markup - prototype code : http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py I'll wait for a new round of feedbacks, Thanks ! Tarek -- Tarek Ziadé | http://ziade.org

4 8

Mutual recursion between setuptools and distutils with Python 2.7
by Antoine Pitrou 01 Jun '09

01 Jun '09

Hello, I'm not sure where to post this (distutils or setuptools bug?), so I'm trying here first. Using the CPython trunk, there's an infinite recursion due to a mutual recursion between setuptools and distutils. Here is a snippet of a traceback (trimmed for obvious reasons): $ usr/bin/easy_install -Z -i http://www.turbogears.org/2.0/downloads/current/index tg.devtools Searching for tg.devtools Reading http://www.turbogears.org/2.0/downloads/current/index/tg.devtools/ Best match: tg.devtools 2.0 Downloading http://www.turbogears.org/2.0/downloads/current/tg.devtools-2.0.tar.gz Processing tg.devtools-2.0.tar.gz Running tg.devtools-2.0/setup.py -q bdist_egg --dist-dir /home/antoine/tmp/easy_install-AkgVEJ/tg.devtools-2.0/egg-dist-tmp-DQjVL4 Traceback (most recent call last): File "usr/bin/easy_install", line 8, in <module> load_entry_point('setuptools==0.6c9', 'console_scripts', 'easy_install')() File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1671, in main File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1659, in with_ei_usage File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1675, in <lambda> File "/home/antoine/cpython/__svn__/Lib/distutils/core.py", line 149, in setup dist.run_commands() File "/home/antoine/cpython/__svn__/Lib/distutils/dist.py", line 929, in run_commands self.run_command(cmd) File "/home/antoine/cpython/__svn__/Lib/distutils/dist.py", line 948, in run_command cmd_obj.run() File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 211, in run File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 446, in easy_install File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 476, in install_item File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 655, in install_eggs File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 930, in build_and_install File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 919, in run_setup File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 27, in run_setup File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 63, in run File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 29, in <lambda> File "setup.py", line 53, in <module> return None File "/home/antoine/cpython/__svn__/Lib/distutils/core.py", line 149, in setup dist.run_commands() File "/home/antoine/cpython/__svn__/Lib/distutils/dist.py", line 929, in run_commands self.run_command(cmd) File "/home/antoine/cpython/__svn__/Lib/distutils/dist.py", line 948, in run_command cmd_obj.run() File "build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py", line 167, in run File "/home/antoine/cpython/__svn__/Lib/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/antoine/cpython/__svn__/Lib/distutils/dist.py", line 948, in run_command cmd_obj.run() File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 177, in run File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 252, in find_sources File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 306, in run File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 330, in add_defaults File "/home/antoine/cpython/__svn__/Lib/distutils/command/sdist.py", line 288, in add_defaults for pkg, src_dir, build_dir, filenames in build_py.data_files: File "build/bdist.linux-x86_64/egg/setuptools/command/build_py.py", line 39, in __getattr__ File "build/bdist.linux-x86_64/egg/setuptools/command/build_py.py", line 44, in _get_data_files File "build/bdist.linux-x86_64/egg/setuptools/command/build_py.py", line 92, in analyze_manifest File "/home/antoine/cpython/__svn__/Lib/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/antoine/cpython/__svn__/Lib/distutils/dist.py", line 948, in run_command cmd_obj.run() [...] File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 177, in run File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 252, in find_sources File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 305, in run File "/home/antoine/cpython/__svn__/Lib/distutils/filelist.py", line 45, in findall self.allfiles = findall(dir) File "build/bdist.linux-x86_64/egg/setuptools/__init__.py", line 71, in findall File "/home/antoine/cpython/__svn__/Lib/os.py", line 294, in walk for x in walk(path, topdown, onerror, followlinks): File "/home/antoine/cpython/__svn__/Lib/os.py", line 294, in walk for x in walk(path, topdown, onerror, followlinks): File "/home/antoine/cpython/__svn__/Lib/os.py", line 294, in walk for x in walk(path, topdown, onerror, followlinks): File "/home/antoine/cpython/__svn__/Lib/os.py", line 294, in walk for x in walk(path, topdown, onerror, followlinks): File "/home/antoine/cpython/__svn__/Lib/os.py", line 294, in walk for x in walk(path, topdown, onerror, followlinks): File "/home/antoine/cpython/__svn__/Lib/os.py", line 284, in walk if isdir(join(top, name)): File "/home/antoine/cpython/__svn__/Lib/genericpath.py", line 41, in isdir st = os.stat(s) File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 87, in wrap RuntimeError: maximum recursion depth exceeded

2 5

Python svn builds
by A. Cavallo 01 Jun '09

01 Jun '09

Hi all, I've just published few python rpms for the trunk source. You can download and install the packages from (either by hand or using yum/yast): http://download.opensuse.org/repositories/home:/cavallo71:/opt-python- interpreters/ The interpreters will install under so it will leave the system python untouched: /opt/opt-python-svnXXXXX-2.7a0 XXXXX is the svn revision. Once installed you can use them just sourcing the file: $> . /opt/opt-python-svnXXXXX-2.7a0/opt-python-svnXXXXX-env.sh $> python Python 2.7a0 (trunk, May 24 2009, 21:18:39) [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2 Type "help", "copyright", "credits" or "license" for more information. There is a web page containing the build results (including a small simple smoke test): http://pyvm.sourceforge.net/ Module developer can use these build to test against the more up to date python source and anticipating bugs. Developer can use it to run regression/integration tests. I hope that will help, Regards, Antonio

2 2

Re: [Distutils] PEP 376 up-to-date
by P.J. Eby 01 Jun '09

01 Jun '09

At 12:29 PM 5/25/2009 +0200, Tarek Ziadé wrote: >Hello, > >I have updated PEP 376, mainly with Phillip's feedback, and changed >the API section so it looks like the current prototype. > >- PEP : http://svn.python.org/view/peps/trunk/pep-0376.txt?view=markup >- prototype code : http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py > >I'll wait for a new round of feedbacks, Thanks ! * There's no reason for anything shown in the module to be private; is_egg_info() and egg_info_dirs() would be useful API functions, and the additions to DistributionMetadata might as well go in distutils * Better yet, move DistributionMetadata to pkgutil and have distutils.dist import it; that way you won't get tons of distutils.* imports any time you import pkgutil. * get_egg_infos() should take a pathlist argument, which if None can default to sys.path. * Project name normalization and case insensitive comparison is still not implemented * File path normalization (absolutizing, case-normalization, de-cross-platforming, etc.) is not implemented * There is extensive coupling both to sys.path and to the global cache; in particular, the owns() operation properly belongs to an object representing a *directory*, rather than an object representing an individual project. Note too that such "directory" objects would then also be an appropriate cache target, and a suitable home for egg_info_dirs() and get_file_users() operations. This also simplifies get_egg_infos, since it would simply either retrieves directories from the cache or creates them and optionally caches them, without also meddling in the details of cache contents. Also, an application that wishes to do so can simply create directory objects of its own and manage their lifecycle accordingly. Personally, I would also make an object for a collection of directories, but you could possibly get away without it. (At this point, I have only briefly skimmed the updated PEP; I'll take a closer look at it once the API/implementation gets more settled.)

2 2

[issue73] setup.py --name prints warning messages to stdout, rather than stderr
by Sridhar Ratnakumar 31 May '09

31 May '09

New submission from Sridhar Ratnakumar <sridharr(a)activestate.com>: Warning messages must be printed to stderr. This allows for a programattic way to find setup.py metadata. > pwd ../dap.plugins.sql-0.2.1 > apy setup.py --name 2> /dev/null 'dap.plugins' is declared as a package namespace, but 'dap' is not: please correct this in setup.py dap.plugins.sql > echo $? 0 > ---------- messages: 295 nosy: srid priority: bug status: unread title: setup.py --name prints warning messages to stdout, rather than stderr _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue73> _______________________________________________

3 2

[issue74] pkg_resources.ImpWrapper documented but no longer exists
by Nicholas Riley 31 May '09

31 May '09

New submission from Nicholas Riley <com-pythonbugs(a)sabi.net>: pkg_resources.ImpWrapper was (and is) mentioned in the documentation (pkg_resources.txt), and as a result has been used by modules such as modulegraph (http://svn.pythonmac.org/modulegraph/modulegraph/trunk/modulegraph/modulegr… ). However, recent versions of pkg_resources remove this definition. Given this class appears in the documentation and still exists, just not with that name, it would be useful if backwards compatibility could be retained. ---------- messages: 296 nosy: nriley priority: bug status: unread title: pkg_resources.ImpWrapper documented but no longer exists _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue74> _______________________________________________

1 0

Patch for clean command
by James 29 May '09

29 May '09

Hi, I'm certaintly new to distutils and setuptools, however I figured I'd send in this patch, and either it will get merged because it's a great idea or someone will perhaps tell me why this doesn't exist already. (or maybe it does and i can't find it) In any case, it adds the pyc option to the clean command so that the .pyc can be removed on request. Personally i'll have a [clean] pyc=1 option in my setup.cfg, but that's for my convenience. cheers, _J [attached: patch and modified file]

2 2

eggs and Python Unicode variants (UCS2, UCS4)
by Martijn Faassen 29 May '09

29 May '09

Hi there, In the lxml project (http://codespeak.net/lxml), we've just noticed the following problem with lxml eggs: you can easy_install an egg that won't work for your Python. This is because Python can be compiled with either 2 or 4 bytes unicode as its internal representation. Any egg that contains compiled C code that uses unicode such as lxml will run into trouble: if it's compiled with a 4 bytes unicode Python, it won't work on a 2 bytes unicode Python, and vice versa. This problem is fairly common in Linux. Many distributions such as Ubuntu and Fedora compile their python with 4 bytes unicode internal representation. If you compile a Python interpreter by hand it defaults to 2 bytes unicode, however. Hand-building a Python interpreter is done fairly commonly by Linux sysadmins for various reasons. It would therefore be very nice if it became possible to make eggs for the different unicode compilation options of Python. This configuration dimension is a real world issue for any binary Python module that does anything with unicode text.. In an earlier mail to this list: http://mail.python.org/pipermail/distutils-sig/2005-October/005222.html M.-A. Lemburg and Phillip Eby had the following discussion: [MAL] >>Please make sure that your eggs catch all possible >>Python binary build dimensions: >> >>* Python version >>* Python Unicode variant (UCS2, UCS4) >>* OS name >>* OS version >>* Platform architecture (e.g. 32-bit vs. 64-bit) [PJE] >As far as I know, all of this except the Unicode variant is captured in >distutils' get_platform(). And if it's not, it should be, since it >affects any other kind of bdist mechanism. I'm not sure whether this means this needs to be escalated from setuptools to the Python interpreter level itself. With this mail, I've done the job escalating this lxml problem to what appears to be the right place, though. :) Thanks, Martijn

5 5

Buildout: caching of base configuration ('extends')
by Thomas Lotze 28 May '09

28 May '09

Hi, I've been implementing a download API for buildout and have reached the point where I'd like to try using it for downloading the base config files referenced from a buildout config by the 'extends' option. This would have the benefit of allowing to cache those config files and being able to run a buildout using that feature while being offline. However, I'd like to discuss two points about doing this: - I think buildout should not unconditionally use a cached copy of a base configuration file. While it is sensible in other use cases to never access the network if a file is found in the cache, base configs can be expected to change and should IMHO be taken from the cache only when offline mode is active or an attempt at downloading failed. - The download cache to be used is configured as one of the options that are read from either ~/.buildout/default.cfg or buildout.cfg and their respective bases. In order to keep things simple, I'd suggest to use a download cache only if specified directly inside any of these two files, and ignore download-cache options in any downloaded base configs. This would be an exception to how options read from config files are combined, but one I deem worthwhile. I'd allow for configuring and using different download caches in those two files, though. That way, bases of the user's default config can be cached centrally while allowing to keep project-related stuff in a different place. Opinions? -- Thomas

3 7

[Buildout] Include-dirs
by Svenn Helge Grindhaug 28 May '09

28 May '09

Hi, I'm trying to do something similar to the custom egg building in the tutorial, http://www.buildout.org/docs/tutorial.html?highlight=location#custom-egg- building I have the following files: setup.py ********* from setuptools import setup, find_packages setup( name='myproject', version='3.3.0', packages=find_packages('src'), package_dir = {'': 'src'}, include_package_data = True, install_requires=['setuptools', 'numpy', 'ScientificPython'], zip_safe = False, ) buildout.cfg ************* [buildout] parts = numpy ScientificPython mypy [numpy] recipe = zc.recipe.egg find-links = http://sunet.dl.sourceforge.net/sourceforge/numpy/numpy-1.3.0.tar.gz eggs= numpy [ScientificPython] recipe = zc.recipe.egg:custom egg=ScientificPython find-links = http://sourcesup.cru.fr/frs/download.php/2372/ScientificPython-2.9.0.tar.gz include-dirs = ${numpy:location}/numpy/core/include [mypy] recipe = zc.recipe.egg:script eggs = numpy ScientificPython interpreter = python ------------ The ScientificPython package need some header files located in the numpy package/egg. I get an error when using ${numpy:location} Error: Referenced option does not exist: numpy location When I use the absolute path to the include dir it works. What am I doing wrong? Best regards, Svenn.

1 0

Re: [Distutils] PEP 376 up-to-date
by P.J. Eby 27 May '09

27 May '09

At 11:01 AM 5/26/2009 +0200, Tarek Ziadé wrote: >Just one point though: > >- do we want to scan egg-info directories that are *directly* added in >sys.path ? No. Such a directory does not represent an importable egg on sys.path. If you mean an .egg file or directory that's directly on sys.path, then that's a bit different. Are you planning to support those? In that event you would need another EggInfo subclass anyway, and you could have the directory (or "PathEntry", perhaps?) constructor simply wrap a single such EggInfo in that event. Or, you can handle it the way pkg_resources does, which is to convert target paths into PEP 302 importer objects, and then use the importer's type to determine what sort of handler should be used for it.

2 1

[issue72] easy_install violates SELinux policies
by Jonathan Cervidae 26 May '09

26 May '09

New submission from Jonathan Cervidae <jonathan.cervidae(a)gmail.com>: I get AVC's aftef installing egg's with easy_install. The AVS are that the files are mislabeled as: unconfined_u:object_r:user_tmp_t:s0 When I we want them to be: system_u:object_r:lib_t:s0 It stops python from accessing the files. I haven't looked at the source code, but those contexts suggest that easy_install is downloading the egg file to a temp directory then moving (not copying) it from the temp dir to /usr/lib/python<VERSION>/site-packages. This is clearly a good thing as unnecessary copying should be avoid and it's easy to fix too. I notice there is a python module called selinux installed on my system. So you need to try and import that. If you succeed you can try and restore the context after moving it to the new location and just ignore any error on that too. ---------- files: fix.txt messages: 292 nosy: jcervidae priority: bug status: unread title: easy_install violates SELinux policies Added file: http://bugs.python.org/setuptools/file53/fix.txt _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue72> _______________________________________________

1 0

Re: [Distutils] Side by side debug/non-debug eggs
by P.J. Eby 22 May '09

22 May '09

At 12:00 PM 5/22/2009 -0400, Jean-Paul Calderone wrote: >On Fri, 22 May 2009 11:33:54 -0400, "P.J. Eby" <pje(a)telecommunity.com> wrote: >>At 09:45 AM 5/22/2009 -0400, Jean-Paul Calderone wrote: >>>Hello all, >>> >>>Prior to eggs, if I wanted to have debug and non-debug versions of an >>>extension module available, I could build and install the extension >>>twice and the modules would happily co-exist, for example: >>> >>> >>> $ python setup.py build install >>> ... >>> $ python-dbg setup.py build install >>> ... >>> $ .../site-packages/twisted/python$ ls *.so >>> _epoll_d.so _epoll.so >>> $ .../site-packages/twisted/python$ >>>If I then ran a debug build of Python, the debug extension would be loaded. >>>When I ran a normal build of Python, the non-debug extension would >>>be loaded. >>>However, with eggs, each new egg completely replaces the last egg. There >>>seems to be no possibility for side-by-side builds. Am I overlooking a >>>feature of eggs or of setuptools? >> >>Have you tried "setup.py develop"? This builds extensions in-place >>in the source tree and adds the source tree to sys.path. > >Aware of it, but I haven't tried it. In this use, it sounds very >similar to "setup.py build_ext -i", which also partially solves the >problem. Unfortunately, some packages have their source laid out >such that they cannot actually be used in-place, only after they've >been installed. The only other thing I can think of is installing in backward-compatibility mode, i.e. "setup.py install --single-version-externally-managed" (with a filename given to --record).

1 0

Re: [Distutils] [Trac] Re: Help packaging plugin : Source distribution
by P.J. Eby 22 May '09

22 May '09

At 12:43 PM 5/22/2009 -0500, Olemis Lang wrote: >On Thu, May 21, 2009 at 3:56 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > > At 11:15 AM 5/20/2009 -0500, Olemis Lang wrote: > >> > >> On Wed, May 20, 2009 at 10:09 AM, Jeff Hammel <jhammel(a)openplans.org> > >> wrote: > >> > > >> > You have to verbosely include them with package data. IMHO, this is a > >> > setuptools bug. E.g.: > >> > > >> > package_data={ 'geotrac': ['templates/*', 'htdocs/*'] }, > >> > include_package_data=True, > >> > > >> > This is probably not quite right :(, but you get the idea. > >> > >> I just tried this and doesnt include files for sdist ... :( > >> > >> Looking forward for further hints : maybe it's just something I'm > >> doing wrong, dont know, but eggs are ok so I'm not sure > > > > If you are not using a revision control system plugin, you will > have to list > > sdist files in MANIFEST.in; > >Thanks, I included files in MANIFEST.in now I can distribute source >packages for TracGViz [1]_ & TracPyTppTheme [2]_ so that EGGs can be >built for different Py versions or whatever ... > >So this is the way ! > >PS: BTW , I dont get the idea about specifying things twice (I mean >package_data & MANIFEST.in ;) Is there a «logical» reason for this ? I >mean why sdist doesnt look for the data specified in package_data ? Or >alternately, what is the reason for having MANIFEST.in too ? That's what distutils does. setuptools improves on it by adding revision control support. But if you don't use a revision control plugin or some other sort of file finder, you have to do it the old distutils way.

1 0

Re: [Distutils] Side by side debug/non-debug eggs
by P.J. Eby 22 May '09

22 May '09

At 09:45 AM 5/22/2009 -0400, Jean-Paul Calderone wrote: >Hello all, > >Prior to eggs, if I wanted to have debug and non-debug versions of an >extension module available, I could build and install the extension >twice and the modules would happily co-exist, for example: > > > $ python setup.py build install > ... > $ python-dbg setup.py build install > ... > $ .../site-packages/twisted/python$ ls *.so > _epoll_d.so _epoll.so > $ .../site-packages/twisted/python$ >If I then ran a debug build of Python, the debug extension would be loaded. >When I ran a normal build of Python, the non-debug extension would be loaded. >However, with eggs, each new egg completely replaces the last egg. There >seems to be no possibility for side-by-side builds. Am I overlooking a >feature of eggs or of setuptools? Have you tried "setup.py develop"? This builds extensions in-place in the source tree and adds the source tree to sys.path.

3 3

how to set cc correctly?
by rupert.thurner 22 May '09

22 May '09

how could one set the compiler correctly? http://opencsw has a setuptools package, which does not use the cc in the path, but in /opt/studio/SOS11/SUNWspro/bin/cc so the following happens: # CC=/opt/SUNWspro/bin/cc easy_install genshi Searching for genshi Reading http://pypi.python.org/simple/genshi/ Reading http://genshi.edgewall.org/ Reading http://genshi.edgewall.org/wiki/Download Best match: Genshi 0.5.1 Downloading http://ftp.edgewall.com/pub/genshi/Genshi-0.5.1.zip Processing Genshi-0.5.1.zip Running Genshi-0.5.1/setup.py -q bdist_egg --dist-dir /tmp/ easy_install-7yiT65/Genshi-0.5.1/egg-dist-tmp-Cp42JQ warning: no previously-included files found matching 'doc/ 2000ft.graffle' warning: no previously-included files matching '*' found under directory 'doc/logo.lineform' unable to execute /opt/studio/SOS11/SUNWspro/bin/cc: No such file or directory ********************************************************************** WARNING: An optional C extension could not be compiled, speedups will not be available. ********************************************************************** Adding Genshi 0.5.1 to easy-install.pth file Installed /opt/csw/lib/python2.6/site-packages/Genshi-0.5.1-py2.6- solaris-2.10-sun4v.egg Processing dependencies for genshi Finished processing dependencies for genshi

3 15

Side by side debug/non-debug eggs
by Jean-Paul Calderone 22 May '09

22 May '09

Hello all, Prior to eggs, if I wanted to have debug and non-debug versions of an extension module available, I could build and install the extension twice and the modules would happily co-exist, for example: $ python setup.py build install ... $ python-dbg setup.py build install ... $ .../site-packages/twisted/python$ ls *.so _epoll_d.so _epoll.so $ .../site-packages/twisted/python$ If I then ran a debug build of Python, the debug extension would be loaded. When I ran a normal build of Python, the non-debug extension would be loaded. However, with eggs, each new egg completely replaces the last egg. There seems to be no possibility for side-by-side builds. Am I overlooking a feature of eggs or of setuptools? Thanks, Jean-Paul

1 0

Re: [Distutils] easy_install picks windows egg on Linux
by P.J. Eby 21 May '09

21 May '09

At 01:49 PM 5/21/2009 -0400, Jean-Paul Calderone wrote: >Downloading >http://pypi.python.org/packages/2.5/p/pyOpenSSL/pyOpenSSL-0.9.py2.5-winxp32… What's bizarre is that that's not a valid egg filename; they must have renamed it at some point. The filename is invalid in two ways: first, it has a '.' between the version and the Python version (it should be a '-') and second, 'winxp32' is not a valid platform name. I would suggest contacting the PyOpenSSL maintainers to find out why they have renamed it in this fashion; .egg files should never be renamed under any circumstances. (I'm actually surprised that easy_install is recognizing it at all, but basically it appears to be treating it as if it's a non-platform-specific egg. That's arguably a bug.) >zipimport.ZipImportError: not a Zip file This part is an expected failure with a --dry-run. It perhaps could be fixed, but the problem is that with --dry-run on, the egg is not moved to the target directory, so when it tries to create a Distribution object for the installed egg, it fails. This has nothing to do with the version problem - you will probably see this occur at the end of any --dry-run install, unless an identically-named egg was was already present.

2 1

[issue71] type of renamed eggs misdetected by easy_install
by Jean-Paul Calderone 21 May '09

21 May '09

New submission from Jean-Paul Calderone <exarkun(a)twistedmatrix.com>: Due to user error, an egg which should have been a Python 2.5 Windows binary egg got renamed to "pyOpenSSL-0.9.py2.5-winxp32.egg" and uploaded to PyPI. easy_install will select this egg for installation on Linux, though it is not usable on that platform. ---------- messages: 290 nosy: exarkun priority: bug status: unread title: type of renamed eggs misdetected by easy_install _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue71> _______________________________________________

1 0

easy_install picks windows egg on Linux
by Jean-Paul Calderone 21 May '09

21 May '09

Hi all, I just noticed that easy_install selects a Windows binary egg when trying to install pyOpenSSL on Linux. Predictably, this fails: $ PYTHONPATH=/tmp/junk6 easy_install --install-dir /tmp/junk6 --dry-run pyOpenSSL Creating /tmp/junk6/site.py Searching for pyOpenSSL Reading http://pypi.python.org/simple/pyOpenSSL/ Reading http://pyopenssl.sourceforge.net/ Reading http://launchpad.net/pyopenssl Best match: pyOpenSSL 0.9.py2.5 Downloading http://pypi.python.org/packages/2.5/p/pyOpenSSL/pyOpenSSL-0.9.py2.5-winxp32… Processing pyOpenSSL-0.9.py2.5-winxp32.egg Moving pyOpenSSL-0.9.py2.5-winxp32.egg to /tmp/junk6 Traceback (most recent call last): File "/usr/bin/easy_install", line 8, in <module> load_entry_point('setuptools==0.6c8', 'console_scripts', 'easy_install')() File "/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py", line 1671, in main with_ei_usage(lambda: File "/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py", line 1659, in with_ei_usage return f() File "/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py", line 1675, in <lambda> distclass=DistributionWithoutHelpCommands, **kw File "/usr/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py", line 211, in run self.easy_install(spec, not self.no_deps) File "/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py", line 446, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py", line 476, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py", line 619, in install_eggs return [self.install_egg(dist_filename, tmpdir)] File "/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py", line 696, in install_egg return self.egg_distribution(destination) File "/usr/lib/python2.5/site-packages/setuptools/command/easy_install.py", line 661, in egg_distribution metadata = EggMetadata(zipimport.zipimporter(egg_path)) zipimport.ZipImportError: not a Zip file exarkun@charm:/tmp$ Is this a known issue? Jean-Paul

1 0

[issue70] Patch to remove call to /usr/bin/sw_vers on OSX
by Ronald Oussoren 21 May '09

21 May '09

New submission from Ronald Oussoren <ronaldoussoren(a)mac.com>: The attached patch for setuptools 0.6c9 replaces a shell callout to /usr/bin/sw_vers to using platform.mac_ver on OSX. The primary reason for writing the patch is that gdb tends to cause havoc with calls to os.popen (the debugged process gets an signal and debugging gets aborted). This patch makes it a lot more convenient to run "setup.py test" in a gdb session, without this patch I regularly have to re-run the tests several times until the callout to sw_vers succeeds. ---------- files: setuptools-no-sw_vers.patch messages: 289 nosy: ronaldoussoren priority: feature status: unread title: Patch to remove call to /usr/bin/sw_vers on OSX Added file: http://bugs.python.org/setuptools/file52/setuptools-no-sw_vers.patch _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue70> _______________________________________________

1 0

PEP 376 APIs
by Tarek Ziadé 21 May '09

21 May '09

Hello As discussed in python-dev, I have started a prototype for pep 376 to define the APIs. I have changed the APIs with Phillip feedback (eg using holders) It's located here : http://bitbucket.org/tarek/pep376 The idea would be to refine it until everyone finds it good, then update PEP 376 accordingly, then move forward with it It's on bitbucket, so feel free to create a branch to demonstrate a change. Regards Tarek -- Tarek Ziadé | http://ziade.org

1 0

Specifying version information once
by Jean-Paul Calderone 21 May '09

21 May '09

Hello, For a long time, the idiom I've used to specify version information in my distutils-based packages has been this simple one: put version information into the Python package somewhere, structure the source layout so that the package can be imported directly from an unpacked source tarball (or vcs wc), and import the version into setup.py. This worked great. And I only had to define my version once. Increasingly, it seems that new distutils-related tools don't support this idiom and break in various ways. (eg http://divmod.org/trac/ticket/2629 http://divmod.org/trac/ticket/2831 ) What is the recommendation for specifying version information in a way which is compatible with all these new tools? Jean-Paul

7 11

buildout/setuptools setup_requires and test_requires
by Chris Withers 20 May '09

20 May '09

Wichert Akkerman wrote: > Previously Chris Withers wrote: >> Wichert Akkerman wrote: >>>> BUT, one thing I have noticed so far is that after running buildout, I >>>> end up with eggs for Paste, PasteDeploy and PasteScript in >>>> pylons_buildout/myproject, which is wrong. >>> Everything in setup_requires and test_requires is installed there, >> Why? >> >>> and >>> buildout can't prevent that. >> Again, why? > > setuptools internals do that, and buildout has no influence over that as > far as I know. If you want to discuss that distutils-sig is the right > list. Sorry, still not sure I follow you... Are you saying that any packages specified in setup_requires and test_requires will be installed in the path of the package that specified them rather than the same location as all other eggs? If so, that would seem to be a bug, no? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

2 2

Re: [Distutils] Specifying version information once
by P.J. Eby 20 May '09

20 May '09

At 09:05 PM 5/19/2009 -0400, Jean-Paul Calderone wrote: >On Tue, 19 May 2009 20:49:37 -0400, "P.J. Eby" <pje(a)telecommunity.com> wrote: >>At 04:28 PM 5/19/2009 -0400, Jean-Paul Calderone wrote: >>>On Tue, 19 May 2009 16:23:16 -0400, "P.J. Eby" >>><pje(a)telecommunity.com> wrote: >>>>At 02:21 PM 5/19/2009 -0400, Jean-Paul Calderone wrote: >>>>>What is the recommendation for specifying version information in >>>>>a way which >>>>>is compatible with all these new tools? >>>> >>>>Don't import the file with your version info; execfile() it instead. >>>> >>>>That is, you can have a somepackage.version module for importing, >>>>but from within your setup.py, do >>>>execfile('somepackage/version.py') instead, so as to avoid >>>>actually importing your package (and anything it depends on). >>> >>>execfile() doesn't set __name__ properly; setting __name__ in a namespace >>>passed to execfile() produces a CPython SystemError. At the moment my >>>version code depends on __name__ being set. I'm sure I could work around >>>this, but it'd be better if I didn't have to worry about code in the >>>version module being executed in two unrelated contexts. Any other >>>suggestions? >> >>Well, if you're okay with depending on setuptools, you could have >>your version code do this: >> >> version = pkg_resources.require('ThisProject')[0].version >> >>And *only* define the version in setup.py, but don't import or run that code. > >I don't think I'm ready to make setuptools a mandatory dependency yet. > >>Of, if you can't depend on setuptools (no pun intended), the only >>other thing I can think of would be to put the version in a text >>file by itself, and have both the version code and setup.py read that file. > >That's an idea that's come up before. I'm curious to here if anyone has >tried it and what their experiences have been. Oh, now that I think of it, there is one other thing that I've done. For some of my packages (including setuptools itself), I use a script that automatically searches and replaces version numbers, and stores the current version number in a configuration file. Another configuration file specifies what files to edit and what format to insert the numbers in. So, I just do a quick command line like './version incr build' to increment the most-minor version number in the docs, the setup.py, and even to edit my release script. And I suppose one last idea that you could use would be to define your version string in the module in such a way that setup.py could read it with a regular expression match, so it wouldn't be actually executing the code.

1 0

Re: [Distutils] Specifying version information once
by P.J. Eby 20 May '09

20 May '09

At 04:28 PM 5/19/2009 -0400, Jean-Paul Calderone wrote: >On Tue, 19 May 2009 16:23:16 -0400, "P.J. Eby" <pje(a)telecommunity.com> wrote: >>At 02:21 PM 5/19/2009 -0400, Jean-Paul Calderone wrote: >>>What is the recommendation for specifying version information in a way which >>>is compatible with all these new tools? >> >>Don't import the file with your version info; execfile() it instead. >> >>That is, you can have a somepackage.version module for importing, >>but from within your setup.py, do >>execfile('somepackage/version.py') instead, so as to avoid actually >>importing your package (and anything it depends on). > >execfile() doesn't set __name__ properly; setting __name__ in a namespace >passed to execfile() produces a CPython SystemError. At the moment my >version code depends on __name__ being set. I'm sure I could work around >this, but it'd be better if I didn't have to worry about code in the >version module being executed in two unrelated contexts. Any other >suggestions? Well, if you're okay with depending on setuptools, you could have your version code do this: version = pkg_resources.require('ThisProject')[0].version And *only* define the version in setup.py, but don't import or run that code. Of, if you can't depend on setuptools (no pun intended), the only other thing I can think of would be to put the version in a text file by itself, and have both the version code and setup.py read that file.

2 1

Re: [Distutils] Specifying version information once
by P.J. Eby 19 May '09

19 May '09

At 02:21 PM 5/19/2009 -0400, Jean-Paul Calderone wrote: >Hello, > >For a long time, the idiom I've used to specify version information in my >distutils-based packages has been this simple one: put version information >into the Python package somewhere, structure the source layout so that the >package can be imported directly from an unpacked source tarball (or vcs wc), >and import the version into setup.py. This worked great. And I only had to >define my version once. > >Increasingly, it seems that new distutils-related tools don't support this >idiom and break in various ways. (eg > > http://divmod.org/trac/ticket/2629 > http://divmod.org/trac/ticket/2831 >) > >What is the recommendation for specifying version information in a way which >is compatible with all these new tools? Don't import the file with your version info; execfile() it instead. That is, you can have a somepackage.version module for importing, but from within your setup.py, do execfile('somepackage/version.py') instead, so as to avoid actually importing your package (and anything it depends on).

2 1

Changes in r72585
by Tim Golden 19 May '09

19 May '09

[coming from a thread on python-win32: http://mail.python.org/pipermail/python-win32/2009-May/009130.html ] In short: changes to build_ext.py in r72585 caused the pywin32 packages to stop building correctly. The changes in question are those which call get_ext_filename with the short form of the ext.name (without the package prefix). The pywin32 setup.py expects the fully-qualified name in the get_ext_filename in its subclass. Now I don't know whether this is an incautious change to build_ext or a false -- altho' historically valid -- assumption on the part of the pywin32 developers. You can see their code here at line 1119: http://pywin32.cvs.sourceforge.net/viewvc/pywin32/pywin32/setup.py?revision… I'm afraid I'm fairly ignorant of the design rationale behind distutils; I'm hoping someone here has a better idea :) Opinions? TJG

3 11

Re: [Distutils] PEP 376 - critique - why more sub-directories within python aren't needed...
by P.J. Eby 19 May '09

19 May '09

At 12:59 PM 5/19/2009 +1200, Noah Gift wrote: >On Tue, May 19, 2009 at 12:59 PM, P.J. Eby ><<mailto:pje@telecommunity.com>pje(a)telecommunity.com> wrote: >At 08:34 PM 5/18/2009 -0400, David Lyon wrote: >Why not keep the .EGG_INFO files in the site-packages directory? > > >That's where they go. Each installed project has its own .egg-info >subdirectory containing the listed files. See the EggFormats >documentation for details. PEP 376 is just adding stdlib support >for the .egg-info format defined by setuptools, and adding a new >RECORD file to it. > > >But if this implementation is the same as eggs, then each egg >directory is then scanned and imported into sys.path, unlike normal >packages which are called via simple namespaces. You are confused. Please refer to the EggFormats documentation for the layout difference between .egg and .egg-info: http://peak.telecommunity.com/DevCenter/EggFormats .egg-info eggs look just like "normal" distutils installs, except for the addition of a "project-version-pyversion.egg-info" subdirectory alongside the code. In Python 2.5 and up, distutils also installs packages this way, except the "project-version-pyversion.egg-info" is a file instead of a directory - as documented in the above documentation. PEP 376 is simply proposing stdlib support for querying the existing formats, and adding a new standard RECORD file to go in that directory.

1 0

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by Tarek Ziadé 19 May '09

19 May '09

On Thu, May 14, 2009 at 12:33 PM, Noah Gift <noah.gift(a)gmail.com> wrote: > One thing that isn't clear to me is whether this could then lead to a > potentially new third party packaging system, or library, that will just do > the same thing again, and create a linear scanning algorithm. Is there a > way to architect things in a way that will never increase the number of > stats to the filesystem or limit them? I'd like to make a difference between what is "importable" and what is "installed" in Python. what's "installed" for me is what is in a site-packages directory that contains egg-info directories. At startup, you have to visit at least once every site-packages directory, that's linear and that won't go away. But it's OK eventually, with a cache or a memory index that takes care of making the lookup code non-linear after Python has started *as long as sys.path is not involved* So, by restricting this API to site-packages (e.g. "installed"), rather than sys.path (eg "importable"), is just a way of limiting the scope to some specific places where you are supposed to install packages (and where egg-info files are located). When a program is launched, the site-packages places should be limited to: - the python site-packages (wether is central, wether it's local, using virtualenv) - the per user site-packages (PEP 370) - maybe a third place added manually using site.addsitedir() > One other thing that hopefully isn't off topic is the idea of scanning for > imports in general. Does it at all seem logical to have an option for > python to have zero scanning to load a module? After all if you really know > the path to a module, why do you need to scan anything first to find it? well if it's not loaded already, you have to find it don't you ? And you don't know where it is, you have to locate it (see imporlib and pep 302) But for me that's another story, than locating the metadata info of an installed package. Tarek

7 13

setup tools maximum recursion depth exceeded
by A. Cavallo 18 May '09

18 May '09

There's been a previous post (Tarek, "Fixing the mess in sdist/egg_info") about the fact that setuptools fails due to a circular dependency: distutils.sdist.run() -> setuptools.build_py.data_files -> setuptools.egg_info.run() -> distutils.sdist.add_defaults() -> setuptools.build_py.data_files -> etc Actually I'm working packaging sphinx and it fails for this reason. Has anything moved to fix this problem? Regards, Antonio

1 0

Re: [Distutils] Version string normalization
by P.J. Eby 18 May '09

18 May '09

At 04:00 PM 5/18/2009 -0400, Jean-Paul Calderone wrote: >Hello, > >I've just noticed that when using setuptools, the name of the file created >by the sdist command changes. The version passed to setup is "0.9.33+r17283" >but the file written is "Nevow-0.9.33-r17283.tar.gz". > >This change causes various problems for me. How can I avoid it? You would need to subclass the sdist command, or use a post-processing step to rename it.

1 0

Version string normalization
by Jean-Paul Calderone 18 May '09

18 May '09

Hello, I've just noticed that when using setuptools, the name of the file created by the sdist command changes. The version passed to setup is "0.9.33+r17283" but the file written is "Nevow-0.9.33-r17283.tar.gz". This change causes various problems for me. How can I avoid it? Thanks, Jean-Paul

1 0

Strict/Loose Version and toolchains checks
by David Cournapeau 16 May '09

16 May '09

Hi, Since we are talking about versioning, here is one more complaint which may fall into the bugreport category or not, I am not sure. There are some version checks in some codepaths which do not make much sense IMHO, in particular: - distutils checks the versions of the toolchain. This is broken. If I want to use gcc built from svn to build my extensions, distutils should just do what I ask it to do, and not barf as it currently does. - bdist_msi also fails to run when the project version is not a released version. I can't see the rationale for that behavior, and it is not consistent with others bdist_ anyway. thanks, David

2 2

Adding entry points into Distutils ?
by Tarek Ziadé 15 May '09

15 May '09

Hello I am making a separate email for this topic to make sure no one misses it. There are *many* benefits of adding entry points into Distutils. In fact, adding a plugin system in there, will help make the commands more extendable and therefore will help us in the long term to remove things out of Distutils. So any strong opinion against them ? If not, I'll add a section in PEP 376 - and the first use case I can see is having a pluggable way to extend the list of files contained in .egg-info. The work to be done would be to extract entry points from pkg_resources (the discovery work that consists of looking for entry_points.txt files will be covered by the APIs already described in that PEP) This is not hard. Cheers Tarek -- Tarek Ziadé | http://ziade.org

16 52

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by P.J. Eby 15 May '09

15 May '09

At 11:38 PM 5/14/2009 +0200, Tarek Ziadé wrote: >That happened because I got a bit lost in the way finders were working >in setuptools, which seemed >a bit over-engineered at that time to me (but I might end up changing >my mind at sime point when I understand it better). The finders mechanism is there to allow support for arbitrary PEP 302 importers -- i.e., sys.path strings that don't go to the filesystem. The main reason why there are two finders currently (find_on_path and find_in_zip) is that find_in_zip supports egg directory bundling. That is, an .egg zipfile may contain arbitrarily nested .egg directories, and an .egg directory may contain arbitrarily nested .egg directories or .egg zipfiles. However, if you're not implementing dependency resolution, you don't need that support. If such a nested item were in use, it'd be on sys.path already. (In other words, the nesting lookups are only useful for alternate version discovery, not simple metadata lookups.)

2 1

including a third party egg without "installing" it
by Randall Smith 14 May '09

14 May '09

I'd like to bundle some third party eggs with my application without installing them. Typically, I'd read __file__ and alter sys.path at runtime to include the third party packages, but I understand that will not work with zipped eggs. I don't know the best way to go about this, but I at least want to know a clean way to do it without requiring the third party eggs to be downloaded and installed separately. Consider this layout. theapp/ docs/ setup.py theapp/ __init__.py app.py ext/ thirdparty1.egg thirdparty2.egg thirdparty3.egg tests/ From my current understanding about how setuptools works, I think I could treat the third party eggs as data files and and runtime (somehow?) insert them into the path and import them. I'm trying to work from this example. http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resour… Is this a workable approach? Is there a better way? --Randall

2 2

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by P.J. Eby 14 May '09

14 May '09

At 11:10 PM 5/14/2009 +0200, Tarek Ziadé wrote: >2009/5/14 P.J. Eby <pje(a)telecommunity.com>: > > IOW, this approach keeps simple things simple, and leaves complex things > > possible. It also does less I/O than what you're proposing, since in the > > normal case the directories are only ever searched once, and the actual > > metadata reads are both lazy and cached. > >Makes a lot of sense yes. I think I'll just start a prototype for that >code, propose 376 on python-dev and see. Notice, by the way, that this is a good example of pkg_resources' code only looking complex if you haven't gone through all the requirements and design issues yourself. The further you go with this, the more likely you're going to end up with something very much resembling pkg_resources itself. The bad news is that you'll be reinventing a lot of wheels. The good news is, you should also be able to understand and maintain pkg_resources internals by the time you're done. ;-) The relevant classes in pkg_resources, btw, are WorkingSet, Distribution, and EntryPoint. You might want to just copy them and strip out the parts you don't need. (i.e. sys.path manipulation and requirements-resolving stuff)

2 1

Re: [Distutils] PEP 376 - site-directories and site.addsitedir
by P.J. Eby 14 May '09

14 May '09

At 12:03 PM 5/14/2009 +0200, Tarek Ziadé wrote: >Hello > >for PEP 376, I have one last fuzzy point. > >http://svn.python.org/view/peps/trunk/pep-0376.txt?view=markup > >The "get_egg_info" api is currently based on scanning the whole >sys.path. And since sys.path can be modified by people, >so the algorithm is linear and can slow down when there are a lot of paths. > >I have a proposal: let's restrict the search for this API to >site-package directories only. (directories added with >site.addsitedir) > >People will be able to mark add any directory (like the per-user >site-package directory - http://www.python.org/dev/peps/pep-0370) > >This requires to add in site.py a registry to keep track of all >directories added through site.addsitedir > >Any thoughts ? What tradeoffs are you optimizing for? Note that a single scan of every directory on sys.path is exactly what happens when an import doesn't find its target until the *last* directory on sys.path. So this is not really a big deal if you're only doing it *once*. If you want to optimize for repeated searches, the best way to do this is with a structure like pkg_resources' WorkingSet object - it simply reads the directories once and makes an object for each installed package. These objects don't do any further I/O, so really we're just talking about caching a list of .egg-info filenames. Each object in the set can be queried for its metadata -- in which case it reads it exactly once, and caches it. With this setup, the full directory scan is only ever done once -- and it's basically equivalent to adding an extra import at the time you first import the metadata management module. Yes, it does mean a global, unless you want to hand off cache management to the application. But the way pkg_resources does it, with WorkingSet and Distribution objects, allows an app with special needs to do its own path management and search operations. IOW, this approach keeps simple things simple, and leaves complex things possible. It also does less I/O than what you're proposing, since in the normal case the directories are only ever searched once, and the actual metadata reads are both lazy and cached. Note, too, that site-packages dirs are likely to have more packages on them than other directories, which means you're not necessarily saving much I/O to start with, and even that small savings evaporates as soon as you do more than one lookup for plugins.

2 1

Re: [Distutils] RFC : Version comparison
by P.J. Eby 14 May '09

14 May '09

At 10:19 AM 5/14/2009 +0200, Tarek Ziadé wrote: >from a setuptools user point of view, the benefit I can see is that >they will have better >version numbers Better how? That was my question. I personally find the common version patterns in use (e.g. '-' and '-r' for post-releases) less clunky and easier to read than 'post'. I also don't care for '.' in front of dev and post. So, I don't see the changes as all "better" from a readability POV. That's not to say it's a bad thing, if it gives other benefits. But it has not been stated what the benefits are supposed to be. >that will comply with a documented standard, that is This isn't actually a change; setuptools also has a documented standard. >usable and understandable by os packagers for example. I agree that a canonical form is useful. However, if that canonical form can be generated from their existing version numbers, then why should they bother changing? That's the open question, IOW. >Now I suppose setuptools will have to propose both for some time, I don't see why. AFAICT, RationalVersion is a strict subset of the syntax supported by setuptools, and most setuptools versions in use should be convertible. >But what I am scared of is : who will work on setuptools side ? can >you bless someone to do the >work when we agree on a common roadmap ? I don't see that there is any work *to* do in setuptools core, since RationalVersion is a strict subset, and we have setup-argument validation providers already. I.e., anyone can make a plugin that validates or converts a setup(version="...") string, put it on sys.path, and force canonical versions.

2 1

Re: [Distutils] RFC : Version comparison
by P.J. Eby 14 May '09

14 May '09

At 10:59 AM 5/14/2009 +0200, Tarek Ziadé wrote: >2009/5/14 P.J. Eby <pje(a)telecommunity.com>: > > At 02:34 PM 5/13/2009 +0200, Tarek Ziadé wrote: > >> > >> Any feedback on this "dev version of post release" use case > > > > It looks like the 'suggest' function doesn't handle svn revisions properly > > for conversion from setuptools versions. Setuptools '-r###' is a > > post-release tag, as is any alpha string alphabetically after > 'final'. That > > is, '0.4a1.r10' is actually '0.4a1.post10'. (Notice, for example, that > > setuptools' trunk version is '0.7a1dev-r66608'.) > > > > See > > > http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-… > > for the full syntax; I'm a bit concerned that there may be other incorrect > > interpretations taking place in this function. (Notice, for > example, that a > > '-' in a version number indicates a post-release, but your code is treating > > '-' as identical to '.'.) > >Ok, I have changed the suggest function accordingly, > > > def test_suggest_rational_version(self): > > self.assertEquals(suggest('1.0'), '1.0') > self.assertEquals(suggest('1.0-alpha1'), '1.0a1') > self.assertEquals(suggest('1.0rc2'), '1.0c2') > self.assertEquals(suggest('walla walla washington'), None) > > # from setuptools > self.assertEquals(suggest('0.4a1.r10'), '0.4a1.post10') > self.assertEquals(suggest('0.7a1dev-r66608'), '0.7a1.dev66608') > self.assertEquals(suggest('0.6a9.dev-r41475'), '0.6a9.dev41475') > self.assertEquals(suggest('2.4preview1'), '2.4c1') > self.assertEquals(suggest('2.4rc1'), '2.4c1') > self.assertEquals(suggest('2.4pre1') , '2.4c1') All looking pretty good, except: > self.assertEquals(suggest('2.1-rc2'), None) # no suggestion This should be equivalent to '2.1c2': >>> from pkg_resources import parse_version as pv >>> pv('2.1-rc2')==pv('2.1c2') True

1 0

PEP 376 - site-directories and site.addsitedir
by Tarek Ziadé 14 May '09

14 May '09

Hello for PEP 376, I have one last fuzzy point. http://svn.python.org/view/peps/trunk/pep-0376.txt?view=markup The "get_egg_info" api is currently based on scanning the whole sys.path. And since sys.path can be modified by people, so the algorithm is linear and can slow down when there are a lot of paths. I have a proposal: let's restrict the search for this API to site-package directories only. (directories added with site.addsitedir) People will be able to mark add any directory (like the per-user site-package directory - http://www.python.org/dev/peps/pep-0370) This requires to add in site.py a registry to keep track of all directories added through site.addsitedir Any thoughts ? (In the meantime, I'll propose the inclusion of a sitedir list in site, in Python-ideas) ++ Tarek -- Tarek Ziadé | http://ziade.org

2 1

[issue69] No 2.6 install setup
by mkesper 14 May '09

14 May '09

New submission from mkesper <mkesper(a)schokokeks.org>: If you want to install 0.6c9 on windows, you face a hen-and-egg-Problem: An .eeg exists, but no installer. ---------- messages: 287 nosy: mkesper priority: feature status: unread title: No 2.6 install setup _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue69> _______________________________________________

1 0

Re: [Distutils] RFC : Version comparison
by P.J. Eby 14 May '09

14 May '09

At 02:34 PM 5/13/2009 +0200, Tarek Ziadé wrote: >Any feedback on this "dev version of post release" use case It looks like the 'suggest' function doesn't handle svn revisions properly for conversion from setuptools versions. Setuptools '-r###' is a post-release tag, as is any alpha string alphabetically after 'final'. That is, '0.4a1.r10' is actually '0.4a1.post10'. (Notice, for example, that setuptools' trunk version is '0.7a1dev-r66608'.) See http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-… for the full syntax; I'm a bit concerned that there may be other incorrect interpretations taking place in this function. (Notice, for example, that a '-' in a version number indicates a post-release, but your code is treating '-' as identical to '.'.) >or I move >forward and integrate it ? Into the stdlib? I'm a bit surprised that something with such global consequences is having its bleeding edge development done in the stdlib, rather than as an external package. I sort of assumed at first the idea was to provide a normalized or canonical form for setuptools versions, but of course it's a bit more restrictive than that, and at the moment the only mapping between the two seems broken -- and in a way that suggests the designers neither read the setuptools version documentation nor examined what people were actually doing with their versioning schemes. To be clear, I'm not opposed to blessing a normalized version scheme -- there are solid benefits to doing such a thing. I'm opposed to putting one in the stdlib without a clear rationale for the choices, and an upgrade path that either lets setuptools users ignore the new scheme, or gives them some benefit for switching. Meanwhile, I haven't seen any indication that this new format is easier for non-Python tools to parse. Is the plan for other tools to use the stdlib parsing code, because I don't see how existing tools like RPM et al could use the proposed scheme.

2 2

Eggs, zc.buildout, and Distributed Version Control, oh my!
by Ross Patterson 14 May '09

14 May '09

...and of course the elephant in the living room, svn:externals. :) I've been using DVC (git mostly) more and more these days and enjoying it very much. It works particularly well when I have a self-contained software package. I tend to start a project in SVN with the standard trunk/branches/tags layout and then use git in front of the SVN repo. I make the egg root itself a buildout for a test environment with my favorite but not required development tools. (Of course I test with "python setup.py test" before release when the egg doesn't require any additional deployment details provided by the buildout. I don't want to get into that discussion.) It all works much better than trying to work directly with SVN far beyond the benefits of just being able to work offline. What I want to ask about and still haven't worked out an answer to is how to best manage framework or deployment projects where I am developing a number of eggs in parallel under a buildout that needs to be versioned as well. In this case, I'm finding moving away from svn:externals more painful than I expected. Having an overview of all version control status for all code being developed is very valuable. Now some of this comes down to eggs being overused and I've certainly been guilty of that myself. For example, I used to make separate eggs for all my client specific code (policy, theme, etc.). This is clearly unnecessary and I've switched to using one egg that contians multiple sub-packages for factoring. Even with that, however, I still find the lack of a good "what changes have I made to all sub projects" overview to be rather painful. Maybe this is outside the scope of the individual DVC tool and should be handled by an independent tool that aggregates status across projects. How do others handle this kind of situation? Ross

2 2

Re: [Distutils] RFC : Version comparison
by P.J. Eby 13 May '09

13 May '09

At 08:58 PM 5/6/2009 +0200, Tarek Ziadé wrote: >Sorry to post again that mail, but it seems that this is the last remaining >issue for the versionning proposal. > >Phillip, could you check my proposal for your development versions of >postreleases >use case ? Is the code up-to-date with the below? >thx > >On Thu, Apr 23, 2009 at 11:19 AM, Tarek Ziadé <ziade.tarek(a)gmail.com> wrote: > > 2009/4/22 P.J. Eby <pje(a)telecommunity.com>: > >> I don't see how it can manage, e.g. a development version of a > postrelease, > >> with an SVN rev or date stamp on it. Such versions might not be found on > >> PyPI or on RPMs, but would be needed in development. > > > > So, instead of having 'dev' and 'post', we would require a third case > > for "pos+dev" version > > > > so (dev|post)N+ could become, ((dev|post)N+)|(postN+devN+) > > > > example: > > > > 1.0.dev459 < 1.0 < 1.0.post456dev463 < 1.0.post456 < 1.0.post489 > > > > > >> > >> (Btw, the wiki page pseudo-regex doesn't match what the code actually > >> parses, either.) > >> > > > > i'll check it thx > > > >> > > > > > > > > -- > > Tarek Ziadé | http://ziade.org > > > > > >-- >Tarek Ziadé | http://ziade.org

2 3

Distutils roadmap : 8th may
by Tarek Ziadé 13 May '09

13 May '09

Hello, I would like to make some decisions and move forward on three topics for Distutils : 1/ version comparison : http://wiki.python.org/moin/Distutils/VersionComparison 2/ standardization of egg-info : PEP 376 + http://wiki.python.org/moin/Distutils/StandardizeEggInfo 3/ changes in PKG-INFO : PEP 345 + http://svn.python.org/projects/peps/branches/jim-update-345/pep-0345.txt + http://wiki.python.org/moin/Distutils/Metadata I propose to settle on these three topics and set next friday (8th may) as a deadline. Cheers Tarek -- Tarek Ziadé | http://ziade.org

2 1

Buildout: adding a download API?
by Thomas Lotze 12 May '09

12 May '09

Buildout as well as a number of buildout recipes need to download files from the net. They need to take care of doing the actual download, using the download cache and honouring the offline option. A couple of years ago, we also wrote the gocept.download recipe that does all that as well as MD5 checks and some other stuff. In order to simplify this situatin, we propose adding a download API to buildout itself. A function like this: def download(url, use_cache=True, md5sum=None) ... return filename might be sufficient to remove the download logic from recipes and make optional checksum testing available without needing to depend on a separate download recipe. As a consequence, zc.recipe.cmmi would be able to do MD5 checks, which would basically make gocept.cmmi redundant. Using gocept.download's capabilities is the one feature of it which still currently keeps us from dropping it in favour of zc.recipe.cmmi. If there's no objection against a download API being added to buildout, we'd like to implement one soon. Otherwise, we'd propose implementing MD5 testing in zc.recipe.cmmi since we consider it a good thing to reduce the zoo of cmmi-related recipes. We do think that a reusable API would be the better choice, though. Viele Grüße, Thomas -- Thomas Lotze · tl(a)gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development

2 1

Setuptools release (svn 1.6)
by Tarek Ziadé 12 May '09

12 May '09

Phillip, What about blessing someone to review and commit the patch for svn 1.6 support (and maybe other patches) and do a setuptools release ? Regards Tarek -- Tarek Ziadé | http://ziade.org

2 1

Possible bug in setuptools - test command?
by Iwan Vosloo 11 May '09

11 May '09

Hi there, We've been having some trouble with a script which invokes setuptools.setup(script_args=['test']) more than once in a single process. During each invocation, python imports different, new instances of the same module. We've narrowed it down to setuptools/command/test.py, in the method with_project_on_sys_path: It saves sys.modules like this: old_modules = sys.modules.copy() Then does its thing and restores sys.modules like this: sys.modules.clear() sys.modules.update(old_modules) But, as the code below illustrates, this results in multiple imports of the same modules: import sys def i(): old_modules = sys.modules.copy() from decimal import Decimal sys.modules.clear() sys.modules.update(old_modules) return Decimal assert i() is i(), 'Two different Decimal instances returned' # Fails Is this a bug? (We are on version 0.6c8 of setuptools, python 2.5) There is an easy solution: Save sys.modules like before: old_modules = sys.modules.copy() But restore it like this: sys.modules = old_modules That seems to fix our problem (not sure of other effects though). Regards - Iwan Vosloo

1 1

Re: [Distutils] Adding entry points into Distutils ?
by P.J. Eby 09 May '09

09 May '09

At 10:21 AM 5/8/2009 +1200, Noah Gift wrote: >1. Different versions of Python conflict with previous versions of >console scripts. Take paste for example. I don't understand what you mean. >2. The entry point mechanism IIRC recursively scans the site-packages >directory and loads up the system path with eggs. This is too >expensive of an operation for the current environment I work in. The scan is not recursive; only files that are actually *on* sys.path are scanned: i.e., either an .egg that is directly on sys.path, or an .egg-info in a directory that is directly on sys.path. Now, it's possible that some application you are using does such a scan explicitly; I'm just noting that merely querying or loading entry points doesn't cause any recursive scans, and it most definitely does not add anything new to sys.path, unless the entry point to be loaded has declared an additional dependency that's *not* on sys.path yet. >3. There doesn't seem to be a clean way to inject user specific >environment details to the console script. >I often need the ability to alter the sys.path in a user specific way >for the entry point without needing to mess up the global sys.path >permanently. I don't understand what you mean here, either.

4 3

Organize e diminua riscos - NS
by Mariana Costa 08 May '09

08 May '09

1 0

Re: [Distutils] Adding entry points into Distutils ?
by P.J. Eby 07 May '09

07 May '09

At 01:14 PM 5/7/2009 -0700, Wheat wrote: >But then having both 'scripts' and 'entry_points/console_scripts' is >less than perfect since it introduces (mostly) unneccessary TIMTOWTDI. I interpret 'scripts' as meaning "non-Python scripts", so there is still only one obvious way to do each thing. ;-)

1 0

Distutils global status
by Tarek Ziadé 07 May '09

07 May '09

Hi, Here's a quick summary of the ongoing work in Distutils, on the issues we seem to reach a consensus. The goal would be to finish the discussions on those and to push them in some kind of "Distutils official roadmap" then look at implementation details (for instance, the pkgutil APIs will be linear unless we use some kind of index) - task : - link - remaining work --- - Introduction the new version comparison algorithm in Distutils - http://bitbucket.org/tarek/distutilsversion - Phillip needs to check the tarek-postdev branch for his use case, then Trent and al. - We need to add the "bits" constructor - Standardize the .egg-info directory, provide APIs, uninstall command - http://svn.python.org/view/peps/trunk/pep-0376.txt?revision=72420&view=mark… - we need to peer-review it here again - Change PKG-INFO content (PEP 345 changes) - http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?revisi… - Tres needs to push the latest changes - we need to peer-review it here again - Finish the massive cleanup / test coverage - not much to say - work in progress by Tarek Cheers Tarek -- Tarek Ziadé | http://ziade.org

1 0

Re: [Distutils] RFC : Version comparison
by P.J. Eby 06 May '09

06 May '09

At 11:12 AM 4/22/2009 +0200, Tarek Ziadé wrote: >Hi, > >We worked during Pycon on version comparisons. Distutils has one but >it is a bit strict, setuptools has another one, but it's a bit >heuristic. > >We would like to propose the inclusion for Python 2.7 of a new version >comparison algorithm, based on the discussion >Fedora, Ubuntu and Python people had. The plan would be to deprecate >the current one (which is not really used anyway) >and provide, promote this one. > >Trent Mick took the lead on this work at the end of Pycon, and worked >on a prototype. > >It's explained here, and there's an implementation (I've put it at >the top of the page for conveniency): > >http://wiki.python.org/moin/Distutils/VersionComparison > >Please comment, I don't see how it can manage, e.g. a development version of a postrelease, with an SVN rev or date stamp on it. Such versions might not be found on PyPI or on RPMs, but would be needed in development. (Btw, the wiki page pseudo-regex doesn't match what the code actually parses, either.)

3 4

logging into distutils
by A. Cavallo 06 May '09

06 May '09

Hi, I'm playing lately with distutils (on the 2.7a branch). This "sudden" interest has been triggered by the python bugs: http://bugs.python.org/issue5941 http://bugs.python.org/issue5940 What I've found (IMHO) is the lack of proper logging while stepping through all the loops inside the code. In fact distutils.log (the log module used internally) It is loosely written resembling the logging python module but missing the full elegance in the api: is there any reason it could not be made very similar to it? I think this is one of the reason is not more deeply spread into the distutils package (a useful tool in diagnostic, IMHO). Because I'm out of work at the moment (It might not last forever tough) I can do it if there is any interest and send a patch set to be reviewed. Beside this to the people interested I can provide a binary compiled python version with the patches hosted on the suse build server system: http://download.opensuse.org/repositories/home:/cavallo71:/python-opt/ This is totally system transparent so it can be installed and de-installed without impact on the host system. Let me know your toughts please, Antonio cavallo

2 1

Re: [Distutils] Setuptools, namespace packages, --single-version-externally-managed
by P.J. Eby 06 May '09

06 May '09

At 04:35 PM 5/5/2009 -0500, Ian Bicking wrote: >Any ideas on this?Â Phillip? > >On Fri, May 1, 2009 at 5:07 PM, Ian Bicking ><<mailto:ianb@colorstudy.com>ianb(a)colorstudy.com> wrote: >So, a bit of a problem came up with pip and namespace >packages.Â Here's my understanding of what's going on: > >When you install a namespace package with pip, it uses install >--single-version-externally-managed, and generally the namespace >directory is empty and there's a *.nspkg.pth file that has this: > >import sys,new,os; p = >os.path.join(sys._getframe(1).f_locals['sitedir'], >*('<NAMESPACE>',)); ie = >os.path.exists(os.path.join(p,'__init__.py')); m = not ie and >sys.modules.setdefault('zope',new.module('zope')); mp = (m or []) >and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) > >So the lack of an __init__.py file doesn't really matter, because >it's created right there, and has its __path__ added to.Â But >there's a problem when there's another namespace package elsewhere >on the path, that wasn't installed with pip (or Setuptools) and uses >pkgutils.extend_path(__path__, __name__).Â This doesn't get >imported because of that .pth file, and the .pth file doesn't itself >use extend_path, so the path isn't searched.Â This is currently >happening with Zope packages installed with plain distutils, then >another package installed with the zope namespace elsewhere with >pip.Â (When using easy_install, I think >pkg_resource.declare_namespace comes into play somewhere, and this >seems to handle this case, but I'm not sure why the installation is >different with pip.) > >So... what should pip be doing differently to make this work? Heck if I know. IIUC, this should work as long as the namespace declaration gets invoked at some point. But I don't know of any way to ensure that, except for pip to write its own __init__.py (invoking declare_namespace()) and gets rid of the nspkg.pth file.

1 0

Setuptools, namespace packages, --single-version-externally-managed
by Ian Bicking 05 May '09

05 May '09

So, a bit of a problem came up with pip and namespace packages. Here's my understanding of what's going on: When you install a namespace package with pip, it uses install --single-version-externally-managed, and generally the namespace directory is empty and there's a *.nspkg.pth file that has this: import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('<NAMESPACE>',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('zope',new.module('zope')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) So the lack of an __init__.py file doesn't really matter, because it's created right there, and has its __path__ added to. But there's a problem when there's another namespace package elsewhere on the path, that wasn't installed with pip (or Setuptools) and uses pkgutils.extend_path(__path__, __name__). This doesn't get imported because of that .pth file, and the .pth file doesn't itself use extend_path, so the path isn't searched. This is currently happening with Zope packages installed with plain distutils, then another package installed with the zope namespace elsewhere with pip. (When using easy_install, I think pkg_resource.declare_namespace comes into play somewhere, and this seems to handle this case, but I'm not sure why the installation is different with pip.) So... what should pip be doing differently to make this work? -- Ian Bicking | http://blog.ianbicking.org

1 1

Re: [Distutils] Adding entry points into Distutils ?
by P.J. Eby 05 May '09

05 May '09

At 06:57 PM 5/4/2009 -0500, Ian Bicking wrote: >* I'm uncomfortable with the way entry points are scanned.Â I >haven't looked close enough to back it up with numbers, but I think >there's a noticeable performance degradation when the number of >installed packages becomes large.Â (Given the algorithm this would >be expected.) It's linear in the number of entry_points.txt files, yes, but in most apps it should occur at most once, since pkg_resources has a single WorkingSet object holding Distribution objects which cache their entry point data upon first access. There are all sorts of ways you could make different tradeoffs, but this particular set of tradeoffs was optimized for a single-application environment, rather than a massive global shared site-packages where there are plugins for every application on the system. It was also optimized for the zipimport case, because you can tell whether a project has entry points from its cached zip directory, that's needed at startup anyhow.

3 2

Re: [Distutils] Anyone stuck with easy_install / has .pth file issues -- this is for you!
by P.J. Eby 05 May '09

05 May '09

At 01:11 PM 5/4/2009 -0700, Garrett Cooper wrote: >You're right -- it doesn't protect against the following (><): > >/full/path/to/package.egg >./package.egg By duplicates, I meant 'package-1.0.egg' and 'package-1.1.egg', not alternate paths to the same file. (As for the '.' replacement, you might want to look at what I said more closely.) >BTW, wasn't -m deprecated? No. > I mean, -m's `delivered' functionality >doesn't even work You'll need to be more specific there: I don't know what you're saying "doesn't work", let alone *how*, what behavior you expect, or what you're seeing instead. >because python stores all loaded package / module >entries in .pth files as keys in a dictionary _anyhow_ I also don't know what you mean here. .pth file contents are added to sys.path, not a dictionary. Unless you're talking about sys.path_importer_cache or something, in which case I still don't see the connection to anything else at hand. >I have no idea how -xN factor into this though The original docs said to use -m before uninstalling, but they really should say -mxN to exclude scripts and avoid installing (or uninstalling!) dependencies. The docs are broken (or at any rate, suboptimal) and should be fixed. >- The duplicates erasure was just gravy, and purely an exercise on my >part getting used to set's in python. The real point was the missing >package in .pth functionality, caused by easy_install / setuptools >lacking proper MP-safe logic. I'm going to guess that by "MP" you mean multiprocessing, in which case I would say, yes, easy_install assumes that you are not running multiple simultaneous installations to the same directory.

2 1

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby 05 May '09

05 May '09

At 12:03 PM 5/3/2009 +0200, Tarek Ziadé wrote: >The name of each file will have to be normalized: all upper case with >no extensions. > >Any opinions ? I don't see any point to the normalization. However, being able to install arbitrary files in .egg-info is currently supported by setuptools, and that capability is used by e.g. the EggTranslations project. I would therefore suggest that perhaps the ability to specify file trees for egg-info inclusion would be better, if the goal is to eventually replace setuptools.

3 7

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby 04 May '09

04 May '09

At 12:50 AM 5/5/2009 +0200, Tarek Ziadé wrote: >On Mon, May 4, 2009 at 7:51 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > > At 06:01 PM 5/4/2009 +0200, Tarek Ziadé wrote: > >> > >> On Mon, May 4, 2009 at 5:48 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > >> >> > I don't see any point to the normalization. > >> >> > >> >> To avoid different naming conventions like: > >> >> > >> >> PKG-INFO, requires.txt, SOURCES.txt > >> > > >> > And the problem with that is...? > >> > >> inconsistency, but right, it makes no sense if any file/dir can be added > >> there. > >> > >> What about SOURCES.txt btw ? What is the reason to add it ? > > > > It's for source distributions. It allows them to be able to rebuild an > > identical source distribution in the absence of source control metadata. > > > > It's not really necessary for the installation process, although it's used > > to figure out which files to install if you use include_package_data=True. > > > >Any particular reason to call it "SOURCES.txt" ? > >Or we can call it MANIFEST (with '/'-separated relative path) I called it SOURCES.txt because MANIFEST is ambiguous as to what it's a manifest *of*, and also to distinguish it from any user-generated MANIFEST file. (If you don't know about those, you're probably going to break backward compatibility w/somebody, btw. Distutils usage is a diverse collection of nightmares.) Anyway, the name of the file has no bearing on what distutils does, unless distutils is trying to implement the same feature as setuptools: i.e. round-trippable sdists whose original manifest was generated via revision control.

1 0

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby 04 May '09

04 May '09

At 05:23 PM 5/4/2009 +0200, Tarek Ziadé wrote: >There's another point I was thinking about in PEP 376 > >What about dropping the 'egg' part in 'PROJECT.egg-info' ? and >replace it with > >'PROJECT.info' > >(and make the 2.7 version compatible with PROJECT.egg-info ) > >I know it's a minor change, Actually, it's a major change, since it means dropping backward compatibility with existing tools.

5 7

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby 04 May '09

04 May '09

At 06:01 PM 5/4/2009 +0200, Tarek Ziadé wrote: >On Mon, May 4, 2009 at 5:48 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > >> > I don't see any point to the normalization. > >> > >> To avoid different naming conventions like: > >> > >> PKG-INFO, requires.txt, SOURCES.txt > > > > And the problem with that is...? > >inconsistency, but right, it makes no sense if any file/dir can be >added there. > >What about SOURCES.txt btw ? What is the reason to add it ? It's for source distributions. It allows them to be able to rebuild an identical source distribution in the absence of source control metadata. It's not really necessary for the installation process, although it's used to figure out which files to install if you use include_package_data=True.

2 1

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby 04 May '09

04 May '09

At 06:31 PM 5/4/2009 +0200, Tarek Ziadé wrote: >Ok then, we will have to provide extra documentation to make people >understand that the '.egg-info' directory has absolutely nothing to do >with egg-the-format >but is rather a metadata container. On the contrary; .egg-info *is* an egg format; see the EggFormats documentation.

2 1

Re: [Distutils] Anyone stuck with easy_install / has .pth file issues -- this is for you!
by P.J. Eby 04 May '09

04 May '09

At 02:43 AM 5/4/2009 -0700, Garrett Cooper wrote: >Hi guys, > Just thought I'd might provide this script to fellow developers >which fixes .pth files (easy-install.pth / .egg was the prime target >-- see the comments for more details): ><http://yaneurabeya.livejournal.com/3929.html>. > Comments are more than welcome. As far as I can tell, it doesn't do anything that "easy_install -mxN" doesn't, although it appears to also convert paths of this form: /foo/bar/baz/foo/bar/spam into: ./baz./spam if I'm reading the code correctly. It also seems to have no protection against adding multiple versions of the same project to a .pth file, and to ignore development eggs, whether or not their directories still exist. In contrast, easy_install already removes non-existent files/directories whenever it touches easy_install.pth, and if you gave it a command line globbing the same files as this tool (i.e., just "easy_install [list of eggs]"), you'd at least end up without any duplicates in the .pth file. In short, AFAICT, you could replace the entire tool with a short note on how to accomplish the same things using easy_install, or by simply having it invoke easy_install internally.

2 2

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby 04 May '09

04 May '09

At 05:54 PM 5/4/2009 +0200, Tarek Ziadé wrote: >2009/5/4 P.J. Eby <pje(a)telecommunity.com>: > > At 05:23 PM 5/4/2009 +0200, Tarek Ziadé wrote: > >> > >> There's another point I was thinking about in PEP 376 > >> > >> What about dropping the 'egg' part in 'PROJECT.egg-info' ? and replace it > >> with > >> > >> 'PROJECT.info' > >> > >> (and make the 2.7 version compatible with PROJECT.egg-info ) > >> > >> I know it's a minor change, > > > > Actually, it's a major change, since it means dropping backward > > compatibility with existing tools. > >well if Distutils 2.7 / 3.1 keeps backward compatibility like I said, >that would give 3 years for existing >tools to change.. Actually, writing '.info' files means that existing tools that expect .egg-info files or directories would stop working correctly *right away*.

1 0

Re: [Distutils] RFC : PEP 376 - egg.info
by P.J. Eby 04 May '09

04 May '09

At 07:34 PM 5/3/2009 +0200, Tarek Ziadé wrote: >2009/5/3 P.J. Eby <pje(a)telecommunity.com>: > > At 12:03 PM 5/3/2009 +0200, Tarek Ziadé wrote: > >> > >> The name of each file will have to be normalized: all upper case with > >> no extensions. > >> > >> Any opinions ? > > > > I don't see any point to the normalization. > >To avoid different naming conventions like: > >PKG-INFO, requires.txt, SOURCES.txt And the problem with that is...? > > However, being able to install > > arbitrary files in .egg-info is currently supported by setuptools, and that > > capability is used by e.g. the EggTranslations project. > > > > I would therefore suggest that perhaps the ability to specify > file trees for > > egg-info inclusion would be better, > >Are you talking about the directory that is built by egg_info in setuptools ? >or something else ? > >I mean, do we want to have subdirectories in .egg-info ? They already exist. > > if the goal is to eventually replace > > setuptools. > >The goal is to provide an install_egg_info command that can be configured >without the need to replace/override it. > >I guess, this can be done by taking part of the code done in setuptools >install_egg_info and provide a way for setuptools or another project >to add more files >through configuration. setuptools simply copies a pre-built egg-info directory from the source tree, that is built by the "egg_info" command. But people can also put arbitrary files and directories there, besides the contents built by the command itself. >Maybe this straight-forward sentence would work better: > >"let's add entry points in Distutils" ? +1

2 1

RFC : PEP 376 - egg.info
by Tarek Ziadé 04 May '09

04 May '09

Hi, I have reworked the PEP a little bit with people feedback. It needs more feedback : http://svn.python.org/projects/peps/trunk/pep-0376.txt - install/uninstall script I think the best solution is not to provide an install script since third-party tool do it. Furthermore, there's already the simplest install script available today: you can run "python setup.py install" on a given package so it gets installed. So what about adding just a global uninstall feature, that uninstalls the files installed for a package, using the record file, and let the third party tool have better features. - MANIFEST (SOURCES.txt) Ronald pointed out that it was not necessary to have a MANIFEST file included if we are going to have a RECORD file, as described in the PEP. The MANIFEST (which in a way is equivalent to SOURCES.txt pip or easy_install adds) is the list of source files, whereas the RECORD is the list if installed files (which might include more elements in case of compilations) What would be the interest of having the list of source files in egg-info ? Cheers Tarek -- Tarek Ziadé | http://ziade.org

4 20

Anyone stuck with easy_install / has .pth file issues -- this is for you!
by Garrett Cooper 04 May '09

04 May '09

Hi guys, Just thought I'd might provide this script to fellow developers which fixes .pth files (easy-install.pth / .egg was the prime target -- see the comments for more details): <http://yaneurabeya.livejournal.com/3929.html>. Comments are more than welcome. If I get any, I'll try to get back in touch sometime before next weekend (my weekdays have been extremely busy lately thanks to a pending work release schedule -_-...). HTH and thanks! -Garrett

1 0

setup.py build_ext --rpath behaviour
by Floris Bruynooghe 02 May '09

02 May '09

Hello Distutils allows you to use a handy --rpath option to the build_ext command to add an RPATH value into the linked object. However some of the semantics are not great IMHO. For the first issue some background first. On SysV-like systems (systems using the ELF binary format) the RPATH field was created in the .dynamic section to allow the shared object to specify extra locations to look for their required shared objects. However this did overwrite the use of the LD_LIBRARY_PATH and that was no good for administrators, so another new field was added: RUNPATH. This was only used *after* LD_LIBRARY_PATH instead of before it solving the problem. When both fields are present the runtime linker ignores RPATH (it is impossible to create a shared object with a RUNPATH but no RPATH). The Solaris linker decided to always encode RUNPATH into the shared object whenever an RPATH is specified (using the -R option). This is a very sensible option most likely the correct behaviour. The GNU linker however decided that backwards compatibility was more important and to make it add the RUNPATH field you have to pass --enable-new-dtags to the linker, if you use -R or -rpath on it's own you will only get an RPATH, no RUNPATH. Now when using the --rpath option to the build_ext command distutils.unixcompiler.UnixCCompiler.runtime_library_dir_option() will use some heuristics to figure out which option to pass to compiler to get the runpath in (i.e. "-R" or "-Wl,-R" etc). I'm going to argue that this needs to be extened pass in -Wl,--enable-new-dtags,-R if the GNU linker is used so that the newer and better RUNPATH gets put into the shared objects all the time. (I don't yet know how to detect the GNU linker but would like a consensus on the desired behaviour before looking into this). Note that this is not completely a disaster, thanks to distutils.sysconfig.customize_compiler() adding the contents of the LDFLAGS environment variable to the command line of the linker invocation you can work around this currently. The second issue with build_ext --rpath is on AIX. Again some background on AIX shared objects, AIX is not SysV-like and uses the XCOFF binary format instead of ELF. Therefore they don't have a RPATH or RUNPATH, but they do have a think called LIBPATH which does something similar. The difference between XCOFF's LIBPATH and ELF's RUNPATH is that AIX's runtime linker does not have a default search path, hence the full search path needs to be encoded into the LIBPATH of the shared object. Now I would propose for build_ext --rpath to encode the LIBPATH when used on AIX since that is the correct thing to do IMHO. (Implementation note: since this is done by passing -blibpath:/opt/example.com/lib:/usr/lib:/lib to the linker note that distutils.unixccompiler.UnixCCompiler.link() would have to be changed not to strip out /usr/lib and /lib from the runtime_library_dirs on AIX. And distutils.unixccompiler.UnixCCompiler.runtime_library_dir_option() would have to use -blibpath:... or -Wl,-blibpath as appropriate) Again, this can currently be circumvented by using the LDFLAGS environment variable. Do these improvements sound sensible? And if so should I create one patch for each (and two bug reports) or combine them into one patch? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

2 3

One Package Per Project
by Brandon Craig Rhodes 01 May '09

01 May '09

I think that, going forward, Python packaging tools (not installation tools; they should remain as they are, for backwards compatibility) should move to supporting only One Package Per Project. And, each project should have the same name as the package inside. In the future, people should have to download an old copy of distutils deliberately if they want to build projects with several packages inside; we should stop releasing tools that support or encourage it. 1. It is easier on developers who want to "import escher" to know that they can simply list "escher" as a dependency instead of having to guess whether it's "Escher" or "EscherProject" or whether it's part of a larger "lithographers" project or whether, heaven forbid, the author decided to redundantly call the project "pyescher". 2. This practice would make PyPI's name make actual sense. It actually claims to be (you can check the site!) the "Python *Package* Index" whereas in fact it's currently nothing of the sort! It's really an index of "projects" that might have zero, one, or several packages inside of them. We should move all projects towards the good behavior of the ones that already name themselves after the single package that they contain. 3. I think the whole idea of putting several packages in a project was useful back when dependencies didn't exist. It made sense, in ancient days, for "ZODB" to include "transaction" because there was no other way to make sure they got installed together. But now that dependencies are possible, there is no longer a need for multiple- package project that outweights the costs involved. 4. The current scheme makes it impossible to choose a "safe" package name when creating and registering a new package. Just because there's no "escher" *project* when you look at PyPI doesn't mean that some project doesn't have an "escher" package hidden inside. You could choose a package name, distribute your product, and only find out later that your users cannot install both your product and another product simultaneously because the other product was, in fact, already using that package name but without your knowing it. -- Brandon Craig Rhodes brandon(a)rhodesmill.org http://rhodesmill.org/brandon

3 2

Re: [Distutils] One Package Per Project
by Ronald Oussoren 01 May '09

01 May '09

I'm -1 on that, for three reasons. 1. I have a number of packages where the PyPI name is not the name of the toplevel package where the two names aren't the same on purpose. An example of this is "pyobjc-framework-WebKit", containing the python package "WebKit". The two don't have the same name because "import WebKit" is more natural during imports, while "pyobjc-framework-WebKit" is clearer in the PyPI listing (you don't have to wonder if this is some cross-platform web toolkit, it's obviously related to PyObjC). 2. For basicly the same reason a number of my PyPI packages have a number of toplevel Python packages. That's again because that makes sense for these packages, and it's furthermore needed for backward compatibility. 3. There actually a good reason for using "pysomelib" instead of "somelib" as the PyPI name for the python bindings for the C library "somelib". Naming the python bindings the same as the base project is confusion, while at the same time "import pysomelib" looks lame in Python code. That said, I agree that there should be a good reason for not using the python package/module name as the PyPI name, and I'm +1 on adding advice to the distutils documentation to keep the two the same. It's also not clear to me what your proposal would mean for namespace packages. Would packages like "zope.interface" be allowed with your proposal? Ronald On Friday, May 01, 2009, at 04:20PM, "Brandon Craig Rhodes" <brandon(a)rhodesmill.org> wrote: >I think that, going forward, Python packaging tools (not installation >tools; they should remain as they are, for backwards compatibility) >should move to supporting only One Package Per Project. And, each >project should have the same name as the package inside. In the future, >people should have to download an old copy of distutils deliberately if >they want to build projects with several packages inside; we should stop >releasing tools that support or encourage it. > > 1. It is easier on developers who want to "import escher" to know that > they can simply list "escher" as a dependency instead of having to > guess whether it's "Escher" or "EscherProject" or whether it's part > of a larger "lithographers" project or whether, heaven forbid, the > author decided to redundantly call the project "pyescher". > > 2. This practice would make PyPI's name make actual sense. It actually > claims to be (you can check the site!) the "Python *Package* Index" > whereas in fact it's currently nothing of the sort! It's really an > index of "projects" that might have zero, one, or several packages > inside of them. We should move all projects towards the good > behavior of the ones that already name themselves after the single > package that they contain. > > 3. I think the whole idea of putting several packages in a project was > useful back when dependencies didn't exist. It made sense, in > ancient days, for "ZODB" to include "transaction" because there was > no other way to make sure they got installed together. But now that > dependencies are possible, there is no longer a need for multiple- > package project that outweights the costs involved. > > 4. The current scheme makes it impossible to choose a "safe" package > name when creating and registering a new package. Just because > there's no "escher" *project* when you look at PyPI doesn't mean > that some project doesn't have an "escher" package hidden inside. > You could choose a package name, distribute your product, and only > find out later that your users cannot install both your product and > another product simultaneously because the other product was, in > fact, already using that package name but without your knowing it. > >-- >Brandon Craig Rhodes brandon(a)rhodesmill.org http://rhodesmill.org/brandon >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG(a)python.org >http://mail.python.org/mailman/listinfo/distutils-sig > >

1 0