On Jun 17, 2009, at 6:07 PM, Trent Mick wrote:

...

I've changed the sdist command some time ago, so it uses the sdtilb tarfile module, instead of the "tar" command, meaning that we could switch to the tar format under windows by default as well now, not requiring the "tar" program anymore

But some people may prefer the zip format.

What are your opinion on this ?

My preferences is for .zip (that is what I use for my packages), the main reason is that Windows users can always unpack a .zip file. Often that is not true for .tar.gz or .tar.bz2 files.

This reference is ancient, but the argument is valid, IMO: http://www.37signals.com/svn/archives2/some_notes_on_the_building_of_codezoo...

...

... Where there were multiple versions available, we chose the most recent, stable version and made that the default. Where there were multiple packages, we chose the Zip format (which is widely available on any platform) and got rid or tar.gz, tar.bz, .exe, and everything else we could. If there were many parts of the component, we defaulted to the one file that gave the user the most value in a single download. Other files, versions, and packages fell to the bottom of the page -- available for the users that absolutely need it, but far from distracting for the users that just want to get their work done. Reducing choices makes the site more useful, not less. ...

The tradeoff is that .zip files are compressed as well. But I don't believe that difference in compression size between .zip/.tar.gz/.tar.bz2 is that big of a concern in *most* cases because most Python sdist's are small, are they not?

Zip files don't support symbolic links(and maybe unix permissions/ ownership). Maybe tarek could include a python script in tools to uncompress tar files so you can always be sure that there is support for it even on windows. Is support for symbolic links(unix permissions/ownership) important? -- Leonardo Santagada santagada at gmail.com

On Thu, 18 Jun 2009 11:57:57 +1000, Ben Finney wrote:

The reverse is true on Unix. The tar-and-gzip format is extensively used so it is pretty much guaranteed to be readable on any Unix system; the Zip format does not have anywhere near the same level of guarantee.

I would think the best solution is to use the tar format (since, as pointed out elsewhere, it has better support for symlinks and permission flags that are often important when distributing works), and ensure that distutils will include a tool for unpacking them with pure Python on those platforms without external support for the format.

Symlinks shouldn't be used.. within python packages... they are totally unneccessary.... As soon as the package gets to a windows system with symlinks it will break and we are back to where we started - a divided community. If you can give one example within a cross-platform package where a symlink would be required (outside of what can be created by distutils from the metadata) I would be interested... but I don't believe there are any... I already thought we had eggs.... With zip, the unzipper code is within python (I believe). So anything where the code for the format is already buried within python is going to be the best for python developers. If tar code is built in within python... then it should be a flick of the coin for people to decide.... There's nothing bad about tar... as long as symlinks stay well out of them.... David

On Jun 17, 2009, at 11:29 PM, David Lyon wrote:

With zip, the unzipper code is within python (I believe). So anything where the code for the format is already buried within python is going to be the best for python developers.

If tar code is built in within python... then it should be a flick of the coin for people to decide....

The tar code is built in within python (there have been problems with early versions of the tar module though). What about unix permissions are those supported on zip, and are they important in a source package? I would think they are importante (and not supported in zip) but I'm not sure. -- Leonardo Santagada santagada at gmail.com

Ben Finney wrote:

Leonardo Santagada writes:

...

What about unix permissions are those supported on zip, and are they important in a source package? I would think they are importante (and not supported in zip) but I'm not sure.

The classic one is the “execute” permission on files. It's very important to preserve that (whether on or off) in distribution of works for use on Unix; and the Zip format doesn't support it AFAIK.

I think the problem of execute permission should be handled on its own (where the format is part of the problem, of course). There are problems with e.g. setuptools which do things differently than distutils with respect to the execute permission on some platforms, and that causes some distributions problems (like nose not finding tests at some locations), not just for sdist command. Whatever the solution, it should be as consistent as possible on every platform. cheers, David

On Thu, Jun 18, 2009 at 10:17 AM, David Lyon wrote:

This is kind of typical of what I get all the time...

Platform: XP-Python 2.5

----------------------------------------------------------------------------- Running installer ... C:\Python25\scripts\easy_install.exe asciiporn Searching for asciiporn ...

Exception: asciiporn requires Python 2.6 or higher

Processing asciiporn-2009.05.01.tar.gz Running asciiporn-2009.05.01\setup.py -q bdist_egg --dist-dir

c:\docume~1\david\locals~1\temp\easy_install-pnlek8\asciiporn-2009.05.01\egg-dist-tmp-8p44m6

What's wrong with that ? If you look at ascii porn setup.py file, it has an explicit test on the python version: ... if sys.version_info[1] < 6: raise Exception("asciiporn requires Python 2.6 or higher") ... So you have to easy_install it on Python 2.6, not Python2.5, or to ask asciiporn maintainer to backport it to 2.5

On Thu, Jun 18, 2009 at 11:42 AM, David Lyon wrote:

Why can't Distutils check that a package will work on multiple python versions before uploading it?

you mean before uploading a distribution at pypi ? how do you automate that ?

On Thu, Jun 18, 2009 at 11:59 AM, David Lyon wrote:

...

On Thu, Jun 18, 2009 at 11:42 AM, David Lyon> wrote:

...

Why can't Distutils check that a package will work on multiple python versions before uploading it?

you mean before uploading a distribution at pypi ? how do you automate

On Thu, 18 Jun 2009 11:52:02 +0200, Tarek Ziadé wrote: that

...

?

Easily(*).....

in a script...

for pyver in installed_python_versions:    run_all_tests_on(pyver)

eg.. A 2.6 python can fire off a 2.2 python script.... which can run lots of tests on the 2.2 system....

I'm lost now :) What do you want to do exaclty ? Let's suppose you have all your code 100% covered in tests (wich is not the case in 99.99% of the packages). So you run the test using all Python versions (which supposes you have *all* Python versions installed, which is not the case on 99% of the people systems out ther) Then what happens ? Do you store the result in the metadata ? Like, by setting up the Requires-Python field with the output ? Frankly, you could just explicitely mark the Python version your code is suppose to run with,

Please have a look to my pet project: http://pyvm.sourceforge.net It solves all the problem you could possibly imagine (famine included!): * provides binary build - ADMINISTRATOR FRIENDLY * could guarantee a reproducible build * integrates smoke tests (just the begin of a continuous integration system) Regards, Antonio

On Thu, 18 Jun 2009 12:19:13 +0200, Tarek Ziadé

wrote:

...

Let's suppose you have all your code 100% covered in tests (wich is not the case in 99.99% of the packages).

Sure. Ok, developers don't write enough tests...

They don't have time... it's not fun.... can't be bothered...

But lets be realistic, there's too many python versions now...

too many platforms...

why can't we do automated builds/tests to cut down the work for developers...

...

So you run the test using all Python versions (which supposes you have *all* Python versions installed, which is not the case on 99% of the people systems out ther)

Think of it like this....

We give python 3.1 the crown....

and write the code for that, that can regressively test any code backwards a package on all the old python versions..

thus, our latest python version... has way more power than any previous version...

...

Then what happens ? Do you store the result in the metadata ? Like, by setting up the Requires-Python field with the output ?

I didn't think of that... it's an excellent idea...

Think of how much work that could save...

...

Frankly, you could just explicitely mark the Python version your code is suppose to run with,

but it would be more fun as you say if it was determined programmatically by distutils based on running some analysis of the actual code....

David _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On Thu, Jun 18, 2009 at 10:26 AM, David Lyon wrote:

Hi All,

Here's a package installation error message that's just plain scary for the average user.

imho Distutils should have checks to stock packages requiring a C compiler from being uploaded. It isn't "right" to expect that from users. ie They should get the .pyd files.

That's not distutils fault, neither easy_install one. "appscript" should provide a binary distribution for Windows (it does for Mac OS if you look at the pypi repo) Since it doesn't, easy_install grabs the source distro, which is the best behavior I can think of, then tries to compile it. The error message is pretty explicit by the way Tarek -- Tarek Ziadé | http://ziade.org

On Thu, Jun 18, 2009 at 11:05 AM, Ronald Oussoren wrote:

The only way to achieve that at the moment is to add some code to setup.py that explicitly raises an exception when you try to build it on an unsupported platform.

Or to provide binary releases because python version and os info are set in the archive name. But it's hackish.

The same is true for the python version, pyobjc currently explicitly tests for python >= 2.3,<3.0 because other versions aren't supported and cause compile errors (at best, IIRC it will compile with python 2.2 but cause interpreter crashes when you try to use it).

Some Trove classifiers have been added for Python 3 for example, Martin uses it to provide a "Python 3 packages" page at Pypi it's "Programming Language :: Python :: 3" (You have other Python versions as well) There are also an "Operating System ::" classifier. I don't like this very much though, I find it cryptic, and I don't think we can force people+installers to use those classifiers. The other solution is to use the "Requires-Python" Metadadata in PEP 345 and add a "Requires-Platform" when someone wants to mark a source distribution compatibility. And the RPC Interface at pypi provides a method to retrieve them without downloading the package it's : release_data(package_name, version) It would be up to easy_install and such tools, to test those values, prior downloading the package. -- Tarek Ziadé | http://ziade.org

On Thu, Jun 18, 2009 at 11:25 AM, Ronald Oussoren wrote:

On 18 Jun, 2009, at 11:22, Tarek Ziadé wrote:

...

On Thu, Jun 18, 2009 at 11:05 AM, Ronald Oussoren wrote:

...

The only way to achieve that at the moment is to add some code to setup.py that explicitly raises an exception when you try to build it on an unsupported platform.

Or to provide binary releases because python version and os info are set in the archive name. But it's hackish.

A binary release that does nothing would indeed be hackish, and would be worse than the existing error because a binary package gives the impression that the platform is actually supported.

IIRC if you provide only binary releases for supported platforms, easy_install will end up saying it didn't find the appropriate distribution. I'm not saying it's the way to go, but I know some people distribute only bdist_egg to make sure their packages are used with the right Python version.

Ronald

-- Tarek Ziadé | http://ziade.org

On Thu, 18 Jun 2009 02:32:10 -0300, Leonardo Santagada wrote:

The tar code is built in within python (there have been problems with early versions of the tar module though).

Ok - well maybe it is fixed now...

What about unix permissions are those supported on zip, and are they important in a source package? I would think they are importante (and not supported in zip) but I'm not sure.

True. But we are back to coin tossing... :-) If you think about it, the "scripts" get specified in the metadata. Distutils itself, if running on *nix, can chmod the scripts with +x as required. (As in, when "setup.py install" is run... That way, nothing is being done that is "naughty"...... It means people can have their scripts and have them +x'ed too.... (but only after they arrive... ie no sneaky +x scripts can be shipped.) Then the issue should come back to which archive format has the best support within python (core).. ie fastest.. most robust.. Whatever is the best black box... should be the winner... David

Trent Mick writes:

...

The reverse is true on Unix. The tar-and-gzip format is extensively used so it is pretty much guaranteed to be readable on any Unix system; the Zip format does not have anywhere near the same level of guarantee.

On what modern Linux and Mac OSes is "/usr/bin/unzip" not installed by default?

You ask as though there were a single default for such systems. For GNU and BSD, at least, there is no such thing. To answer the question you may have instead been wanting answered: Any Unix-like system where “get a minimal installation first and then I'll install just what's needed to support the task this machine should perform” will not have an ‘unzip’ utility except by mere chance coincidence with the desired task. It's certainly not a valid assumption that “I want to set up this machine for a purpose that needs Python packages” and “I want the ability to read Zip format files” will necessarily co-incide. On the other hand, it *is* a valid assumption that any such system will have the ability to read tar-and-gzip archives. -- \ “Friendship is born at that moment when one person says to | `\ another, ‘What! You too? I thought I was the only one!’” —C.S. | _o__) Lewis | Ben Finney

On Jun 17, 2009, at 15:07 PM, Trent Mick wrote:

The tradeoff is that .zip files are compressed as well. But I don't believe that difference in compression size between .zip/.tar.gz/.tar.bz2 is that big of a concern in *most* cases because most Python sdist's are small, are they not?

My project is 2.5 MB in .zip, 2.25 MB in .tar.bz2, and 1.8 MB in .tar.xz. I would be happy to save that 700 KB by moving from .zip to .tar.xz (once there is LZMA-tar in Python), but I'm more concerned about the permissions issues, which are still an unsolved problem in setuptools, e.g.: http://bugs.python.org/setuptools/issue6 Regards, Zooko $ ls -lS -rw-rw-r-- 1 zooko zooko 8355840 2009-06-17 09:48 allmydata- tahoe-1.4.1-r3916.tar -rw-rw-r-- 1 zooko zooko 2562835 2009-06-17 09:51 allmydata- tahoe-1.4.1-r3916.zip -rw-rw-r-- 1 zooko zooko 2383653 2009-06-17 09:48 allmydata- tahoe-1.4.1-r3916.tar.gz -rw-rw-r-- 1 zooko zooko 2250149 2009-06-17 09:49 allmydata- tahoe-1.4.1-r3916.tar.bz2 -rw-rw-r-- 1 zooko zooko 2223425 2009-06-17 09:49 allmydata- tahoe-1.4.1-r3916.tar.rz -rw-rw-r-- 1 zooko zooko 1818466 2009-06-17 09:52 allmydata- tahoe-1.4.1-r3916.7z -rw-rw-r-- 1 zooko zooko 1811698 2009-06-17 09:50 allmydata- tahoe-1.4.1-r3916.tar.xz

On Jun 18, 2009, at 17:18 PM, David Lyon wrote:

But setup tools looks like it is trying to do the "right" thing and getting it wrong...

It isn't so easy for setuptools to know which things ought to have +x and which things ought not just based on their pathnames or other metadata. There was a bug in setuptools on cygwin years ago where something didn't have +x that needed it, and that was fixed in setuptools-0.6c6 by adding +x to a variety of files. That led to this new bug where something has +x that shouldn't: http://bugs.python.org/setuptools/issue6 I can imagine that in general there might be a package which has two files named "foo/bar" and "foo/quux" where the package really needs bar to have +x and quux not to have +x. Having an archive format that preserves such bits would probably be a good way to solve all such problems -- by making it the packagers problems to set the bits before packaging rather than setuptools's problem to figure out which bits ought to be set after installation.

10% difference in library size is nothing.

1.8 / 2.5 == 72%, so 28% of the .zip file size is saved by switching to .tar.xz. Of course, some packages may gain more or less than 28% (compared to ZIP compression) from LZMA compression. Regards, Zooko

On Jun 19, 2009, at 13:54 PM, Eric Smith wrote:

...

It isn't so easy for setuptools to know which things ought to have +x and which things ought not just based on their pathnames or other metadata.

Unless you specify explicitly it in a metadata file.

My argument is that using a metadata-preserving archive format allows a programmer to control that stuff and also allows setuptools (or distutils, I guess) to ignore that stuff. Inventing a way to pass that information through the metadata requires the programmer to learn and use an extra configuration on top of what they already do, and requires setuptools (distutils) to pay attention to that stuff.

...

Having an archive format that preserves such bits would probably be a good way to solve all such problems -- by making it the packagers problems to set the bits before packaging rather than setuptools's problem to figure out which bits ought to be set after installation.

How would you create such an archive on Windows? (A serious question, not trying to be argumentative.)

You mean a bdist? You can't "cross-compile" and build a bdist for Linux on Windows (can you?), or vice versa. I think we're only talking about "non-cross-compiled" bdists here. A metadata-preserving archive format wouldn't make this harder, and it might make it easier. Regards, Zooko

Zooko Wilcox-O'Hearn wrote:

...

...

Having an archive format that preserves such bits would probably be a good way to solve all such problems -- by making it the packagers problems to set the bits before packaging rather than setuptools's problem to figure out which bits ought to be set after installation.

How would you create such an archive on Windows? (A serious question, not trying to be argumentative.)

You mean a bdist? You can't "cross-compile" and build a bdist for Linux on Windows (can you?), or vice versa. I think we're only talking about "non-cross-compiled" bdists here.

I mean an sdist, which I hope is what we're still talking about, in the context of using a .zip or .tar file. How do I specify +x on Windows? I use cygwin, but what about those who don't? (Again, this is a serious question. I don't use straight Windows enough to know the answer.)

A metadata-preserving archive format wouldn't make this harder, and it might make it easier.

As long as the metadata is cross-platform, which I'm not sure it is. We're trying to create a cross-platform source distribution, after all. Eric.

On Fri, 19 Jun 2009 10:38:39 -0600, Zooko Wilcox-O'Hearn wrote:

It isn't so easy for setuptools to know which things ought to have +x and which things ought not just based on their pathnames or other metadata.

It is very easy.... it already knows... http://docs.python.org/distutils/setupscript.html#installing-scripts The scripts option simply is a list of files to be handled in this way.

From the PyXML setup script:

setup(..., scripts=['scripts/xmlproc_parse', 'scripts/xmlproc_val'] )

Having an archive format that preserves such bits would probably be a good way to solve all such problems -- by making it the packagers problems to set the bits before packaging rather than setuptools's problem to figure out which bits ought to be set after installation.

It's totally impossible on windows... Because you are talking about a unix specific problem... Why not just solve the problem rather than spreading it to more platforms?

1.8 / 2.5 == 72%, so 28% of the .zip file size is saved by switching to .tar.xz. Of course, some packages may gain more or less than 28% (compared to ZIP compression) from LZMA compression.

Well maybe its good. Its a new one to me though... David

At 07:29 PM 6/17/2009 -0400, David Lyon wrote:

(try - easyinstall buildbot... or pydbf to see what i mean)

I don't see any errors here. What errors are you seeing?