Mailman 3 April 2008 - Distutils-SIG

Re: [Distutils] shebang line modified by setuptools
by Stephen Waterbury April 14, 2008

April 14, 2008

Gael Varoquaux wrote: > On Sat, Apr 12, 2008 at 09:41:08PM -0400, Stephen Waterbury wrote: >>> These are all broken and you should report bugs on them. I have >>> reported many for Ubuntu. A system application should only ever >>> depend on the system Python (or interpreter), never on the whims of >>> your $PATH. > >> I agree with Barry here -- I would even go further and say that >> the Python used by system apps (i.e., apps that the OS depends on) >> should be separate and should not be on the default PATH (so the user >> can't use it accidentally), and no non-system apps should use the >> system Python unless specifically installed by the user or sysadmin >> into the system Python's area. > > I agree with you, and not with Barry that all these are broken. If you > look at the apps that use "#!/usr/bin/env", they are all pretty > non-system apps, like for instance "f2py". I think it is good that these > apps use the default python: I don't want to have f2py to be running for > another python than my default one. I am actually quite surprised to see > that ipython is installed with a hardcoded path to /usr/bin/python. > > Anyway, I am not going to bug report anything, as my system works just > fine, thank you. Yes, your luck holds -- as does mine, for now ... but since Barry works for Canonical (besides being Python's new Release Manager), he has a civic responsibility to report Ubuntu Python bugs even if his own system works ... and since that can be quite time-consuming, I can't blame him for trying to recruit some help ... ;) Steve

2 1

Re: [Distutils] how to easily consume just the parts of eggs that are good for you
by Stanley A. Klein April 12, 2008

April 12, 2008

On Fri, April 11, 2008 6:00 am, Dave Peterson <dpeterson(a)enthought.com> wrote: > > Message: 5 > Date: Thu, 10 Apr 2008 18:16:03 -0500 > From: Dave Peterson <dpeterson(a)enthought.com> > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: "Phillip J. Eby" <pje(a)telecommunity.com> > Cc: distutils-sig <distutils-sig(a)python.org> > > Phillip J. Eby wrote: >> At 03:48 PM 4/10/2008 -0500, Dave Peterson wrote: >>> Stanley A. Klein wrote: >>>> On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote: >>>>> I think I can sum up any further points by simply asking: "Should it >>>>> be safe to assume I can distribute my application via eggs / >>>>> easy_install just because it is written in Python?" >>>> >>>> I think that based on this discussion the bottom line answer to this >>>> question is "No". >>> >>> I agree that it seems like that's where things are headed in the >>> discussion. But the discussion doesn't always coincide with the >>> reality, right? I guess I'm trolling more for a statement from the >>> setuptools maintainer here. >>> >>> Particularly since I'm looking for an answer to my question about >>> should Enthought continue to invest time into a setuptools patch that >>> lets developers include docs, config files, etc. in eggs for >>> installation in a FHS-approved location at install time? >> >> I think it's more than reasonable to define a standard for including >> such data. I'm somewhat more skeptical about doing that installation >> automatically within easy_install. Likewise, I'm skeptical about >> doing other sorts of non-package, non-script installation. I'd like >> to see proposals that show due care to cross-platformness, >> uninstallability, etc. >> >> In other words, when it comes to a "patch" -- the documentation is >> going to count for a lot more than the code, and I'd rather see a >> concrete proposal well in advance of the patch. >> >> Sooner would be better than later, too, because it's likely that the >> plan for "non-egg installs" is going to be affected by the plan as well. > > I believe I understand, and agree, with your concerns. Let me be clear > on the status of where we are in our work: we've internally talked > through a number of design possibilities, and are now trying out (via > hacking on setuptools) how the one we thought was "best" worked out. In > particular, we're concerned about the difficulty of use in terms of even > just the use-cases we have for ETS projects. Also, since we do a bit > of cross-platform deployment, we're also investigating those effects on > the design as well. That being said, I don't think we're ready to put > forward a proposal that would withstand too much public scrutiny quite > yet - at least if the resulting discussion implied a significant time or > effort commitment on our part to carry the conversation forward. If it > sounded like we'd already figured it all out, I apologize for getting > people's hopes up! I just wanted to make sure further pursuit in this > direction on our part wasn't completely wasted. > > The above not withstanding, if anyone is interested in talking about it > / helping us, we certainly wouldn't ignore you. I just can't promise > immediate responses due to pretty pressing customer commitments on our > part. > > > Regarding the mention of 'uninstallability' above, I assume it would be > sufficient if the installed files were simply included in the generated > list of files provided by the "--record" option since there is currently > no uninstall command at all? Or is there something else you'd like to > see as an intermediate measure? I'd love to add an uninstall option to > easy_install as well (it's something we get hit about alot by our user > community) but there's only so many hours in a day. ;-) > Here are a few principles I think I've gleaned from this discussion: 1. Eggs are not intended to serve as substitutes for rpms, debs, and other platform package distribution systems. They are primarily intended as a convenience for developers during development. They can be used in distribution packaging in a manner analogous to .jar files in java. 2. Every platform wants to put the files somewhere different. Accordingly, Setuptools and Distutils need to support mapping the files to the preferred locations for the various platforms. 3. Setuptools and Distutils support some platform packaging systems, such as rpm. Packagers should distribute using those systems. If eggs are included in a package, they should be treated as any other file for that packaging system and not as a form of distribution. For example, an rpm should reasonably be able to contain egg files along with other distribution files. (Note that java-related rpms often contain .jar files.) Consideration should be given to adding features to existing support for platform packaging systems. Also, if egg files are to be included in a distribution package, it needs to be a two-step process (first make the eggs, then make the package). 4. It would probably help to have a style guide or automated converter for naming and placement of files in a source distribution to facilitate file location mappings and platform naming issues. For example, Windows accepts file names with spaces in them but Linux generally doesn't (and bdist_rpm produces errors if it sees them). For a multiplatform distribution either a style guide or a lint-type tool needs to address issues like this. Stan Klein

1 0

Re: [Distutils] [PATCH][setuptools] Add test_runner support to setup
by Phillip J. Eby April 12, 2008

April 12, 2008

At 06:07 PM 4/11/2008 +0200, Tarek Ziadé wrote: >On Fri, Apr 11, 2008 at 6:01 PM, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > > > > >was the patch satisfactory or is it still lacking? > > >Or do I need to do anything more for it to be applied? > > > > It's sitting in my folder of patches to eventually be applied. :) > >What about this one ? > >http://mail.python.org/pipermail/distutils-sig/2008-April/009187.html Yes. >Do you have a public tracker where we can submit patches ? No -- but I heard it's being worked on. My first preference would be a python.org hosted Roundup instance, especially if I can use my existing login. If that can't be arranged, I'll probably look into throwing up a Trac or Roundup myself -- eventually.

1 0

Batch installation of python extentions
by Elazar Leibovich April 11, 2008

April 11, 2008

Hi, I need to install a big bunch of python extentions in a couple of offline windows XPcomputer. Is there any standard way to do that on the win32 platform? The best options I thought of is installing ez_install manually on every computer, and then install all the egg-files in a certain directory. Is there any better way to do that? Is there a standard way to distribute python extentions. Example extentions: scipy, pyserial, wxpython

1 0

Re: [Distutils] setuptools and pkg registration
by Phillip J. Eby April 11, 2008

April 11, 2008

At 11:37 PM 4/10/2008 +0200, Jeroen Ruigrok van der Werven wrote: >Phillip, > >I am not sure (haven't verified) if my email ever got through to the list, >but with FreeBSD's Perl installation an additional module gets installed >that registers every CPAN installed module as a pkg in FreeBSD's pkg system. > >Given the recent bruhaha about the setuptools and eggs (hey, from a >programmer and sysadmin point of view I like 'm) I wonder if this would be >easier for the discussion if you could dump a prototype on the table. > >If it would help I can see if I can cook up something. At this point, it has become clear to me that the package database isn't really necessary for other systems. easy_install could simply not uninstall things that it doesn't find listed in its *own* metadata. Really, the only technical hurdle remaining to solving egg-free installation is setting up a namespace package's __path__ correctly, in a situation where: 1. The user has system packages contributing to the namespace in a "site" directory, 2. There is no __init__.py in the namespace under that site directory, 3. There are non-egg packages on PYTHONPATH that contribute to the same directory, and 4. Two versions of the same module within that namespace have been installed, one in site-packages and the other on PYTHONPATH Actually, there are various other criteria involved, such as whether pkg_resources is imported or not, and whether the setuptools site.py hack is employed in the PYTHONPATH directory. For example, if pkg_resources were imported (which normally happens when setuptools-installed scripts are run), it would be possible to have it do the sorting, and thus ensure that there's never a problem like that. What it would not fix, is the circumstance where you just pop into the Python interpreter with your PYTHONPATH set, if you have the above situation. And the above situation is going to occur anytime you're using PYTHONPATH to get a later version of something that uses a namespace package. So, the other alternative is to keep the 'site.py' hack in effect, but either change the contents of that site.py or add extra information to a .pth file so that the namespaces get sorted at that point. There would still be a hacked site module at that point, but at least upgrading the site.py would let us get rid of the need to load any .pth files from the directory, put eggs ahead of other things, etc. For that matter, the site.py wouldn't be needed unless you were installing things with namespace packages in it. (Unfortunately, looking at site-packages to see if the hack is needed won't help, since you might install a conflicting system package *later*.)

1 0

Re: [Distutils] [Enthought-dev] Installation woes - again..
by Dave Peterson April 11, 2008

April 11, 2008

NOTE: cc'ing distutils-sig for help debugging potential problem with setuptools on gentoo. Hans Meine wrote: > Am Donnerstag, 10. April 2008 09:19:31 schrieb Hans Meine: > >> The same errors occur without virtualenv, so you're obviously right that >> this is setuptools-related. I could try again and really deinstall >> Gentoo's setuptools (instead of just locally installing it the official >> way), but I really doubt now that this would make a difference. >> > > So, I have even tried this - deinstalling setuptools altogether, and > installing everything from scratch. I used "script" to create a log file of > the process with all input and resulting output + errors: > http://kogs-www.informatik.uni-hamburg.de/~meine/clean-ets-install.log > Thanks for providing that full log! From looking at that, I'm really starting to suspect a bug in setuptools 0.6c8 as I can't make any sense of the following sequence of events. While it's possible we've done something wrong in our setup.py scripts, I have my doubts as this release appears to install fine for users on Windows, OSX, and RH. (For distutils-sig readers, Hans is using Gentoo.) 1) During the first install attempt of "ets", the log shows two lines reporting success with installing enthought.tvtk-2.0.2-py2.4-linux-i686.egg. One right after downloading and building enthought.tvtk from the source tarball, and one in the middle of the enthought.kiva build. 2) Regarding that enthought.kiva build, the log indicates that a dependency caused a search to be made for "enthought.kiva>=2.0.3.dev,<=2.0.3" and a match was found (enthought.kiva-2.0.3.tar.gz) The log then shows this building, apparently a successful build (no error messages, just warnings) up until the normally reported "Installed /path/to/egg" line where it mentions success with installing enthought.tvtk rather enthought.kiva. After that, it reports being unable to find the enthought.kiva distribution. 3) You then re-run the "easy_install -f ... ets" command which has no problem building enthought.kiva this time using the exact same source tarball. I have no idea what might be causing this sequence of events. Has anyone seen anything similar with setuptools? -- Dave

1 0

Re: [Distutils] Proposal: egg -> bdist_wininst converter
by Phillip J. Eby April 10, 2008

April 10, 2008

At 09:10 AM 4/10/2008 +0100, Paul Moore wrote: >For me, on Windows, the biggest negative aspect of eggs is when people >provide binary installations (ones with compiled C extensions) in eggs >only. I prefer bdist_wininst/bdist_msi installers, and with compiled >code, I can't always build from source (complex library dependencies, >etc). > >So, given that an egg already has everything that is needed at >runtime, how easy would it be to create a tool which converted an egg >into a bdist_wininst installer? With that, I could happily forget >about eggs. http://mail.python.org/pipermail/python-dev/2008-March/077959.html Basically what you need to do is rewrite the zipfile index such that everything is prefixed with PURELIB/ or PLATLIB/, except for the EGG-INFO directory which must be renamed to match the original egg's filename, plus '-info' -- i.e., foo-1.2-py2.3-win32.egg should have its EGG-INFO renamed to PURELIB/foo-1.2-py2.3-win32.egg-info or PLATLIB/foo-1.2-py2.3-win32.egg-info. Once the egg is reformatted in this way, it's a perfectly valid zipfile that the existing bdist_wininst code could be reused to turn into an .exe.

2 1

Proposal: egg -> bdist_wininst converter
by Paul Moore April 10, 2008

April 10, 2008

For me, on Windows, the biggest negative aspect of eggs is when people provide binary installations (ones with compiled C extensions) in eggs only. I prefer bdist_wininst/bdist_msi installers, and with compiled code, I can't always build from source (complex library dependencies, etc). So, given that an egg already has everything that is needed at runtime, how easy would it be to create a tool which converted an egg into a bdist_wininst installer? With that, I could happily forget about eggs. Paul. PS I am assuming that "eggs are only a packaging format" - and that all of the setuptools goodies like namespace support, entry points etc, that some projects benefit from, would still work whether the package was an egg or a conventionally-installed bdist_wininst...

1 0

Re: [Distutils] A Modest Proposal for "A Database of Installed Packages"
by Phillip J. Eby April 10, 2008

April 10, 2008

At 05:05 PM 4/7/2008 -0400, Alexander Michael wrote: >a. I believe that having side-car files that sit alongside >packages because they have the same base name makes the database more >transparent to the uninitiated. I'm not aware that this was ever a stated design goal, nor why it should have any priority. OTOH, files named by distribution would be at least as, if not even *more* transparent than package names, so I don't see any particular benefit to this. >Just browsing a directory of python >packages will allow you to see what's going on. Moving like-names >files around manually maintains the integrity and availability of the >data. Moving anything manually, other than the *entire* directory, will be unlikely to retain any form of integrity, so it's best not to give the false impression that it would. >I think that having magic entries in an essentially "hidden" >directory somewhere will cause all sorts of trouble that could be >avoiding at the cost of a small bit of duplication. > b. I assume, perhaps incorrectly, that most distributions contain >only a single package. Very incorrectly, unless you mean a single top-level package. Odds are fairly good that if there's a package, there's probably at least a subpackage, too, like perhaps a tests subpackage. >That said, I do agree that if you are primarily interested in a >database of *distributions* (as opposed to *packages*) then something >like is proposed in PEP 262 makes more sense (but it would have to be >per directory and not site-wide due to the dynamic nature of the >python path). That's exactly what I want. The only reason I didn't just implement easy_install using a per-directory form of PEP 262 is that I wanted something done rather more immediately. That was years ago, so I can afford to be more patient now. :) >This is a trade-off between putting the metadata up >front in an obvious and easy to understand way so that it will >hopefully have a better chance of being noticed and maintained, versus >tucking it away hidden someplace so that even though it is broken, it >doesn't bother anyone until they care enough to fix it. *It is this >trade-off that I am exploring with this strawman "counter" proposal to >PEP 262.* Someone would have to be crazy to maintain this information by hand. So I'd actually consider it an advantage if the file format made this fact plain, by using something that's difficult for a human being to maintain, like say a pickle. ;-) OTOH, it's possible that some system packagers will not wish to use Python to generate the files, so using something a bit less complex would be a good idea. The format proposed by PEP 262 isn't really that bad of a trade-off in those terms. >2. The strawman proposal did not explicitly address how optional >add-on tools (like setuptools) might manage namespace packages. I think there's some mistunderstanding here about the proposal's goals. If the proposal doesn't work for setuptools, it doesn't work, period. The entire point is to allow setuptools to do its work without annoying the people who don't want to use it. >I agree with Floris that the best way to avoid magic is to actually >have the sub-packages in a namespace share the same parent directory >on disk. I agree with this also. The issue is that an __init__.py must exist for this to happen, but most system packaging tools (e.g. RPM) require that a given file be owned by at most one system package (i.e., distribution), whereas the contents of a namespace package are assembled from multiple distributions. That's the problem that needs solving, not runtime support for the namespace itself. >4. Concerns were raised about the performance penalty for using the >side-car style files without version numbers possibly not all of which >were located at the top-most level of the directory listed in the >python path. > >Any add-on tool that actually used the data would very likely need to >build a cache of the data using a more efficient representation, This is a misunderstanding of the point I raised. Floris merely asked why there were version numbers in .egg-info files, and I answered him. That doesn't actually have much, if anything, to do with the package database proposal. It's merely how installed distributions' versions can be recognized quickly at runtime, not anything to do with how potential installation conflicts are handled at installation time. easy_install uses eggs for installation simply because it need never worry about file ownership conflicts. There is a direct mapping from a distribution to its files: the contents of a zipfile or subdirectory. This also allows for (relatively) straightforward uninstallation. The goal of the proposal, then, is to have a way for easy_install to have another way to map from a distribution to its owned files (and vice versa), so that eggs are not necessary for normal, single-version installations.

2 1

Re: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you
by Phillip J. Eby April 10, 2008

April 10, 2008

At 11:48 PM 4/9/2008 +0100, Paul Moore wrote: >On 09/04/2008, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > It would be, if .eggs were a packaging format, rather than a binary > > distribution/runtime format. > > > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > > doesn't contain documentation either, unless it's needed at > > runtime. Same for configuration files. > >And yet, Java doesn't have an equivalent of easy_install for jar >files, to my knowledge. Actually, OSGi and Eclipse plugins and "feature sites" come quite close, and setuptools rips off many of its features from them. OSGi is basically a standard for additional .jar metadata to encompass dependencies and other info.

1 0