I've changed the sdist command some time ago, so it uses the sdtilb tarfile module, instead of the "tar" command, meaning that we could switch to the tar format under windows by default as well now, not requiring the "tar" program anymore

But some people may prefer the zip format.

What are your opinion on this ?

My preferences is for .zip (that is what I use for my packages), the main reason is that Windows users can always unpack a .zip file. Often that is not true for .tar.gz or .tar.bz2 files.

This reference is ancient, but the argument is valid, IMO: http://www.37signals.com/svn/archives2/some_notes_on_the_building_of_codezoo...

... Where there were multiple versions available, we chose the most recent, stable version and made that the default. Where there were multiple packages, we chose the Zip format (which is widely available on any platform) and got rid or tar.gz, tar.bz, .exe, and everything else we could. If there were many parts of the component, we defaulted to the one file that gave the user the most value in a single download. Other files, versions, and packages fell to the bottom of the page -- available for the users that absolutely need it, but far from distracting for the users that just want to get their work done. Reducing choices makes the site more useful, not less. ...

The tradeoff is that .zip files are compressed as well. But I don't believe that difference in compression size between .zip/.tar.gz/.tar.bz2 is that big of a concern in *most* cases because most Python sdist's are small, are they not?

Cheers, Trent

On Jun 17, 2009, at 6:07 PM, Trent Mick wrote:

...

I've changed the sdist command some time ago, so it uses the sdtilb tarfile module, instead of the "tar" command, meaning that we could switch to the tar format under windows by default as well now, not requiring the "tar" program anymore

But some people may prefer the zip format.

What are your opinion on this ?

My preferences is for .zip (that is what I use for my packages), the main reason is that Windows users can always unpack a .zip file. Often that is not true for .tar.gz or .tar.bz2 files.

This reference is ancient, but the argument is valid, IMO: http://www.37signals.com/svn/archives2/some_notes_on_the_building_of_codezoo...

...

... Where there were multiple versions available, we chose the most recent, stable version and made that the default. Where there were multiple packages, we chose the Zip format (which is widely available on any platform) and got rid or tar.gz, tar.bz, .exe, and everything else we could. If there were many parts of the component, we defaulted to the one file that gave the user the most value in a single download. Other files, versions, and packages fell to the bottom of the page -- available for the users that absolutely need it, but far from distracting for the users that just want to get their work done. Reducing choices makes the site more useful, not less. ...

The tradeoff is that .zip files are compressed as well. But I don't believe that difference in compression size between .zip/.tar.gz/.tar.bz2 is that big of a concern in *most* cases because most Python sdist's are small, are they not?

Zip files don't support symbolic links(and maybe unix permissions/ ownership). Maybe tarek could include a python script in tools to uncompress tar files so you can always be sure that there is support for it even on windows.

Is support for symbolic links(unix permissions/ownership) important?

-- Leonardo Santagada santagada at gmail.com

Trent Mick trentm@gmail.com writes:

My preferences is for .zip (that is what I use for my packages), the main reason is that Windows users can always unpack a .zip file. Often that is not true for .tar.gz or .tar.bz2 files.

The reverse is true on Unix. The tar-and-gzip format is extensively used so it is pretty much guaranteed to be readable on any Unix system; the Zip format does not have anywhere near the same level of guarantee.

I would think the best solution is to use the tar format (since, as pointed out elsewhere, it has better support for symlinks and permission flags that are often important when distributing works), and ensure that distutils will include a tool for unpacking them with pure Python on those platforms without external support for the format.

On Wed, 17 Jun 2009 21:39:35 -0400, "P.J. Eby" pje@telecommunity.com wrote:

At 07:29 PM 6/17/2009 -0400, David Lyon wrote:

...

(try - easyinstall buildbot... or pydbf to see what i mean)

I don't see any errors here. What errors are you seeing?

Python 2.5, Windows XP

Running installer ... C:\Python25\scripts\easy_install.exe buildbot Searching for buildbot Reading http://pypi.python.org/simple/buildbot/

Reading http://buildbot.sourceforge.net/

Reading http://buildbot.net/

Reading https://sourceforge.net/project/showfiles.php?group_id=73177&package_id=...

Best match: buildbot 0.7.10p1 Downloading http://pypi.python.org/packages/source/b/buildbot/buildbot-0.7.10p1.zip#md5=...

Processing buildbot-0.7.10p1.zip

Running buildbot-0.7.10p1\setup.py -q bdist_egg --dist-dir c:\docume~1\david\locals~1\temp\easy_install-xnvlp8\buildbot-0.7.10p1\egg-dist-tmp-cxmna9

zip_safe flag not set; analyzing archive contents...

buildbot.clients.debug: module references __file__ buildbot.scripts.runner: module references __file__

buildbot.status.web.baseweb: module references __file__ buildbot.status.web.grid: module references __file__ buildbot.test.runutils: module references __file__ buildbot.test.test_maildir: module references __file__ buildbot.test.test_mailparse: module references __file__ buildbot.test.test_shell: module references __file__ buildbot.test.test_slavecommand: module references __file__

No eggs found in c:\docume~1\david\locals~1\temp\easy_install-xnvlp8\buildbot-0.7.10p1\egg-dist-tmp-cxmna9 (setup script problem?)

There's nothing bad about tar... as long as symlinks stay well out of them....

Tar on solaris is often broken by design. :) For compat reasons Solaris' default 'tar' is one that has file length problems (http://www.mikehan.com/rant/solaris-tools.html).

Trent

Trent Mick trentm@gmail.com writes:

...

The reverse is true on Unix. The tar-and-gzip format is extensively used so it is pretty much guaranteed to be readable on any Unix system; the Zip format does not have anywhere near the same level of guarantee.

On what modern Linux and Mac OSes is "/usr/bin/unzip" not installed by default?

You ask as though there were a single default for such systems. For GNU and BSD, at least, there is no such thing.

To answer the question you may have instead been wanting answered: Any Unix-like system where “get a minimal installation first and then I'll install just what's needed to support the task this machine should perform” will not have an ‘unzip’ utility except by mere chance coincidence with the desired task.

It's certainly not a valid assumption that “I want to set up this machine for a purpose that needs Python packages” and “I want the ability to read Zip format files” will necessarily co-incide. On the other hand, it *is* a valid assumption that any such system will have the ability to read tar-and-gzip archives.

On Thu, 18 Jun 2009 02:32:10 -0300, Leonardo Santagada santagada@gmail.com wrote:

The tar code is built in within python (there have been problems with early versions of the tar module though).

Ok - well maybe it is fixed now...

What about unix permissions are those supported on zip, and are they important in a source package? I would think they are importante (and not supported in zip) but I'm not sure.

True.

But we are back to coin tossing... :-)

If you think about it, the "scripts" get specified in the metadata.

Distutils itself, if running on *nix, can chmod the scripts with +x as required. (As in, when "setup.py install" is run...

That way, nothing is being done that is "naughty"......

It means people can have their scripts and have them +x'ed too....

(but only after they arrive... ie no sneaky +x scripts can be shipped.)

Then the issue should come back to which archive format has the best support within python (core).. ie fastest.. most robust..

Whatever is the best black box... should be the winner...

David

This is kind of typical of what I get all the time...

Platform: XP-Python 2.5

----------------------------------------------------------------------------- Running installer ... C:\Python25\scripts\easy_install.exe asciiporn Searching for asciiporn Reading http://pypi.python.org/simple/asciiporn/

Reading http://www-rcf.usc.edu/~kaizhu/work/asciiporn

Best match: asciiporn 2009.05.01 Downloading http://pypi.python.org/packages/source/a/asciiporn/asciiporn-2009.05.01.tar....

ERRORS: Traceback (most recent call last): File "C:\Python25\scripts\easy_install-script.py", line 8, in <module> load_entry_point('setuptools==0.6c9', 'console_scripts', 'easy_install')() File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\command\easy_install.py", line 1671, in main File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\command\easy_install.py", line 1659, in with_ei_usage File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\command\easy_install.py", line 1675, in <lambda> File "C:\Python25\lib\distutils\core.py", line 151, in setup dist.run_commands() File "C:\Python25\lib\distutils\dist.py", line 974, in run_commands self.run_command(cmd) File "C:\Python25\lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\command\easy_install.py", line 211, in run File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\command\easy_install.py", line 446, in easy_install File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\command\easy_install.py", line 476, in install_item File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\command\easy_install.py", line 655, in install_eggs File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\command\easy_install.py", line 930, in build_and_install File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\command\easy_install.py", line 919, in run_setup File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\sandbox.py", line 27, in run_setup File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\sandbox.py", line 63, in run File "C:\Python25\lib\site-packages\setuptools-0.6c9-py2.5.egg\setuptools\sandbox.py", line 29, in <lambda> File "setup.py", line 440, in <module>

Exception: asciiporn requires Python 2.6 or higher

Processing asciiporn-2009.05.01.tar.gz Running asciiporn-2009.05.01\setup.py -q bdist_egg --dist-dir c:\docume~1\david\locals~1\temp\easy_install-pnlek8\asciiporn-2009.05.01\egg-dist-tmp-8p44m6

On Thu, Jun 18, 2009 at 10:17 AM, David Lyon david.lyon@preisshare.netwrote:

This is kind of typical of what I get all the time...

Platform: XP-Python 2.5

---

Running installer ... C:\Python25\scripts\easy_install.exe asciiporn Searching for asciiporn ...

Exception: asciiporn requires Python 2.6 or higher

Processing asciiporn-2009.05.01.tar.gz Running asciiporn-2009.05.01\setup.py -q bdist_egg --dist-dir

c:\docume~1\david\locals~1\temp\easy_install-pnlek8\asciiporn-2009.05.01\egg-dist-tmp-8p44m6

What's wrong with that ?

If you look at ascii porn setup.py file, it has an explicit test on the python version:

... if sys.version_info[1] < 6: raise Exception("asciiporn requires Python 2.6 or higher") ...

So you have to easy_install it on Python 2.6, not Python2.5, or to ask asciiporn maintainer to backport it to 2.5

On Thu, Jun 18, 2009 at 3:57 AM, Ben Finneyben+python@benfinney.id.au wrote:

Trent Mick trentm@gmail.com writes:

...

My preferences is for .zip (that is what I use for my packages), the main reason is that Windows users can always unpack a .zip file. Often that is not true for .tar.gz or .tar.bz2 files.

The reverse is true on Unix. The tar-and-gzip format is extensively used so it is pretty much guaranteed to be readable on any Unix system; the Zip format does not have anywhere near the same level of guarantee.

I would think the best solution is to use the tar format (since, as pointed out elsewhere, it has better support for symlinks and permission flags that are often important when distributing works), and ensure that distutils will include a tool for unpacking them with pure Python on those platforms without external support for the format.

+1

A end-user that uses easy_install ends up calling the stdlib tarfile module to unpack the archives, so it works out of the box. I guess pip is also using that code (unpack_archive in setuptools.archive_util)

Although, I would rather see a new high-level function in tarfile itself, callable from the __main__ section, to pack/unpack an archive provided in the arguments. (like the doctest modules has).

Cheers Tarek

On Thu, Jun 18, 2009 at 11:05 AM, Ronald Oussorenronaldoussoren@mac.com wrote:

The only way to achieve that at the moment is to add some code to setup.py that explicitly raises an exception when you try to build it on an unsupported platform.

Or to provide binary releases because python version and os info are set in the archive name. But it's hackish.

The same is true for the python version, pyobjc currently explicitly tests for python >= 2.3,<3.0 because other versions aren't supported and cause compile errors (at best, IIRC it will compile with python 2.2 but cause interpreter crashes when you try to use it).

Some Trove classifiers have been added for Python 3 for example,

Martin uses it to provide a "Python 3 packages" page at Pypi

it's "Programming Language :: Python :: 3" (You have other Python versions as well)

There are also an "Operating System ::" classifier.

I don't like this very much though, I find it cryptic, and I don't think we can force people+installers to use those classifiers.

The other solution is to use the "Requires-Python" Metadadata in PEP 345 and add a "Requires-Platform" when someone wants to mark a source distribution compatibility. And the RPC Interface at pypi provides a method to retrieve them without downloading the package

it's : release_data(package_name, version)

It would be up to easy_install and such tools, to test those values, prior downloading the package.

On Thu, Jun 18, 2009 at 11:25 AM, Ronald Oussorenronaldoussoren@mac.com wrote:

On 18 Jun, 2009, at 11:22, Tarek Ziadé wrote:

...

On Thu, Jun 18, 2009 at 11:05 AM, Ronald Oussorenronaldoussoren@mac.com wrote:

...

The only way to achieve that at the moment is to add some code to setup.py that explicitly raises an exception when you try to build it on an unsupported platform.

Or to provide binary releases because python version and os info are set in the archive name. But it's hackish.

A binary release that does nothing would indeed be hackish, and would be worse than the existing error because a binary package gives the impression that the platform is actually supported.

IIRC if you provide only binary releases for supported platforms, easy_install will end up saying it didn't find the appropriate distribution.

I'm not saying it's the way to go, but I know some people distribute only bdist_egg to make sure their packages are used with the right Python version.

Ronald

On Thu, 18 Jun 2009 10:45:58 +0200, Tarek Ziadé ziade.tarek@gmail.com wrote:

That's not distutils fault, neither easy_install one.

"appscript" should provide a binary distribution for Windows (it does for Mac OS if you look at the pypi repo) Since it doesn't, easy_install grabs the source distro, which is the best behavior I can think of, then tries to compile it.

The error message is pretty explicit by the way

It's good that you are seeing the challenges from a user perspective.

These are only tests I am doing on random packages and it doesn't worry me too much if they don't work.

But I think there is much to be done to smooth things out...

Assigning all blame to the developers isn't what I'm doing. As I am finding it hard to build and test on all platforms myself. Needless to say it is something that distutils imho should be aiming for.

David

On Thu, 18 Jun 2009 11:52:02 +0200, Tarek Ziadé ziade.tarek@gmail.com wrote:

On Thu, Jun 18, 2009 at 11:42 AM, David Lyon> wrote:

...

Why can't Distutils check that a package will work on multiple python versions before uploading it?

you mean before uploading a distribution at pypi ? how do you automate

that

?

Easily(*).....

in a script...

for pyver in installed_python_versions: run_all_tests_on(pyver)

eg.. A 2.6 python can fire off a 2.2 python script.... which can run lots of tests on the 2.2 system....

David

On Thu, 18 Jun 2009 12:19:13 +0200, Tarek Ziadé ziade.tarek@gmail.com wrote:

Let's suppose you have all your code 100% covered in tests (wich is not the case in 99.99% of the packages).

Sure. Ok, developers don't write enough tests...

They don't have time... it's not fun.... can't be bothered...

But lets be realistic, there's too many python versions now...

too many platforms...

why can't we do automated builds/tests to cut down the work for developers...

So you run the test using all Python versions (which supposes you have *all* Python versions installed, which is not the case on 99% of the people systems out ther)

Think of it like this....

We give python 3.1 the crown....

and write the code for that, that can regressively test any code backwards a package on all the old python versions..

thus, our latest python version... has way more power than any previous version...

Then what happens ? Do you store the result in the metadata ? Like, by setting up the Requires-Python field with the output ?

I didn't think of that... it's an excellent idea...

Think of how much work that could save...

Frankly, you could just explicitely mark the Python version your code is suppose to run with,

but it would be more fun as you say if it was determined programmatically by distutils based on running some analysis of the actual code....

David

On Jun 17, 2009, at 15:07 PM, Trent Mick wrote:

The tradeoff is that .zip files are compressed as well. But I don't believe that difference in compression size between .zip/.tar.gz/.tar.bz2 is that big of a concern in *most* cases because most Python sdist's are small, are they not?

My project is 2.5 MB in .zip, 2.25 MB in .tar.bz2, and 1.8 MB in .tar.xz. I would be happy to save that 700 KB by moving from .zip to .tar.xz (once there is LZMA-tar in Python), but I'm more concerned about the permissions issues, which are still an unsolved problem in setuptools, e.g.: http://bugs.python.org/setuptools/issue6

Regards,

Zooko

$ ls -lS -rw-rw-r-- 1 zooko zooko 8355840 2009-06-17 09:48 allmydata- tahoe-1.4.1-r3916.tar -rw-rw-r-- 1 zooko zooko 2562835 2009-06-17 09:51 allmydata- tahoe-1.4.1-r3916.zip -rw-rw-r-- 1 zooko zooko 2383653 2009-06-17 09:48 allmydata- tahoe-1.4.1-r3916.tar.gz -rw-rw-r-- 1 zooko zooko 2250149 2009-06-17 09:49 allmydata- tahoe-1.4.1-r3916.tar.bz2 -rw-rw-r-- 1 zooko zooko 2223425 2009-06-17 09:49 allmydata- tahoe-1.4.1-r3916.tar.rz -rw-rw-r-- 1 zooko zooko 1818466 2009-06-17 09:52 allmydata- tahoe-1.4.1-r3916.7z -rw-rw-r-- 1 zooko zooko 1811698 2009-06-17 09:50 allmydata- tahoe-1.4.1-r3916.tar.xz

On Jun 18, 2009, at 17:18 PM, David Lyon wrote:

But setup tools looks like it is trying to do the "right" thing and getting it wrong...

It isn't so easy for setuptools to know which things ought to have +x and which things ought not just based on their pathnames or other metadata. There was a bug in setuptools on cygwin years ago where something didn't have +x that needed it, and that was fixed in setuptools-0.6c6 by adding +x to a variety of files. That led to this new bug where something has +x that shouldn't:

http://bugs.python.org/setuptools/issue6

I can imagine that in general there might be a package which has two files named "foo/bar" and "foo/quux" where the package really needs bar to have +x and quux not to have +x.

Having an archive format that preserves such bits would probably be a good way to solve all such problems -- by making it the packagers problems to set the bits before packaging rather than setuptools's problem to figure out which bits ought to be set after installation.

10% difference in library size is nothing.

1.8 / 2.5 == 72%, so 28% of the .zip file size is saved by switching to .tar.xz. Of course, some packages may gain more or less than 28% (compared to ZIP compression) from LZMA compression.

Regards,

Zooko

On Jun 19, 2009, at 13:54 PM, Eric Smith wrote:

...

It isn't so easy for setuptools to know which things ought to have +x and which things ought not just based on their pathnames or other metadata.

Unless you specify explicitly it in a metadata file.

My argument is that using a metadata-preserving archive format allows a programmer to control that stuff and also allows setuptools (or distutils, I guess) to ignore that stuff. Inventing a way to pass that information through the metadata requires the programmer to learn and use an extra configuration on top of what they already do, and requires setuptools (distutils) to pay attention to that stuff.

...

Having an archive format that preserves such bits would probably be a good way to solve all such problems -- by making it the packagers problems to set the bits before packaging rather than setuptools's problem to figure out which bits ought to be set after installation.

How would you create such an archive on Windows? (A serious question, not trying to be argumentative.)

You mean a bdist? You can't "cross-compile" and build a bdist for Linux on Windows (can you?), or vice versa. I think we're only talking about "non-cross-compiled" bdists here.

A metadata-preserving archive format wouldn't make this harder, and it might make it easier.

Regards,

Zooko

Zooko Wilcox-O'Hearn wrote:

...

...

Having an archive format that preserves such bits would probably be a good way to solve all such problems -- by making it the packagers problems to set the bits before packaging rather than setuptools's problem to figure out which bits ought to be set after installation.

How would you create such an archive on Windows? (A serious question, not trying to be argumentative.)

You mean a bdist? You can't "cross-compile" and build a bdist for Linux on Windows (can you?), or vice versa. I think we're only talking about "non-cross-compiled" bdists here.

I mean an sdist, which I hope is what we're still talking about, in the context of using a .zip or .tar file. How do I specify +x on Windows? I use cygwin, but what about those who don't? (Again, this is a serious question. I don't use straight Windows enough to know the answer.)

A metadata-preserving archive format wouldn't make this harder, and it might make it easier.

As long as the metadata is cross-platform, which I'm not sure it is. We're trying to create a cross-platform source distribution, after all.

Eric.

On Fri, 19 Jun 2009 10:38:39 -0600, Zooko Wilcox-O'Hearn zooko@zooko.com wrote:

It isn't so easy for setuptools to know which things ought to have +x and which things ought not just based on their pathnames or other metadata.

It is very easy.... it already knows...

http://docs.python.org/distutils/setupscript.html#installing-scripts

The scripts option simply is a list of files to be handled in this way.

From the PyXML setup script:

setup(..., scripts=['scripts/xmlproc_parse', 'scripts/xmlproc_val'] )

Having an archive format that preserves such bits would probably be a good way to solve all such problems -- by making it the packagers problems to set the bits before packaging rather than setuptools's problem to figure out which bits ought to be set after installation.

It's totally impossible on windows...

Because you are talking about a unix specific problem...

Why not just solve the problem rather than spreading it to more platforms?

1.8 / 2.5 == 72%, so 28% of the .zip file size is saved by switching to .tar.xz. Of course, some packages may gain more or less than 28% (compared to ZIP compression) from LZMA compression.

Well maybe its good. Its a new one to me though...

David