Alberto Valverde wrote: [...]

...

No... which makes binary eggs unusable on Linux. I feel like there was something else that made binary packages on a Mac unreliable, but I can't remember. Windows binary eggs generally work fine. This is discussed some here: http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-eg...

I remember reading that... Perhaps a solution could be to"freeze" the tar.gzs too, beside the compiled eggs in case we're lucky, and hope that a compiler + libs are there when the install script is thawed. If the compiled egg won't cut it, then try to compile form source. I'm not sure how to tell easy_install to download the source distributions though.

Well, it'll make eggs from your tarballs anyway. Turning it into a build process is a bit of a nuisance... I was hoping for something that didn't require building, but just did installation. That said, it might work. Maybe PoachEggs (mentioned later) does what you want in this case. Or, maybe it can be slightly modified to do what you want (I think it might also unintentionally turn tarballs into eggs).

...

One nuisance is that people don't generally know how their Python was built (UCS2 or UCS4). I was thinking about making something very similar to eggfreezer (which I'm unlikely to do now that eggfreezer exists ;), You're patches are welcomed, In fact, If you want to include it inside virtualenv I would be most happy :).

Not so much virtualenv, but it might fit in PoachEggs: https://svn.openplans.org/svn/PoachEggs/trunk workingenv did installation, but I abandoned that when I cleaned it up as virtualenv, and now I'm inclined to keep them separate (though clearly complementary). I have thought about putting something in virtualenv to make relocating the environment easier. There's only a few things that need to be modified, I think. That might mitigate some of these issues.

...

and generating an "install" .py file that determines the platform and downloads the appropriate platform bundle.

Hmm, this "download" is precisely what I'm trying to avoid. My main use case is: A machine has gone down and I want to quickly put back everything together in another machine with a backup and something that contains *all* needed software. Sort of like an apt reposisory inside a dvd which lets you install debian on a machine with absolutely not net access. The "no-net" condition is just there to guarantee that all deps will be available no matter how old and discontinued they are (which if I think about it is rather ambitious... well, at least make it more likely than with the current situation)

With PoachEggs I've now got it working so you can build a working environment, create a "requirements" file that lists everything in that working environment, download all the eggs for that into a directory, and then later install from that directory and disallow network access. Well, more-or-less. It's not a single-file install like eggfreezer, but they are working toward similar goals. The single-file install including binaries is something I would really like for Deliverance, and specifically for lxml, but also to create a simple installation experience for people who don't know anything about Python build things (and maybe don't know Python).

I think that this multi-platform issue could be solved by bundling all the different binary versions of all binary packages. However, I'm not sure if pkg_resources could deal with the UCS2/UCS4 issue given that it doesn't distinguish it in the platform id. Maybe by hacking in an extra placeholder, before the .egg and after the ${platform}, that the script uses to distinguish and then remove it before giving it to easy_install? Though this smells like the root of the problem comes from setuptools and should be fixed there...

If you could select the appropriate binary at installation time you could include all of them in the bundle. It would be big, but at least personally that would be fine for me. It would be simpler to simply name the resulting file with a more accurate platform, but then people don't always know the right thing to get. At least a little check in the script itself would be helpful, so they get errors immediately instead of confusing errors at import time. I'm not sure how to detect UCS2/UCS4. The root (well, *one* root) of the problem is setuptools/distutils not getting the platform really right, but there's also all kinds of messy backward compatibility issues there, and no backward compatibility issues for eggfreezer. I'm not sure there aren't other issues. I'm also not sure that there isn't a finite number of resolvable issues. So maybe MacPorts and fink and system python on Macs are different. But that's just 3 platforms instead of 1, it's not an infinite number. And UCS2 Python is different from UCS4 on Linux, but that's really the one issue I know of where Linux Pythons differ. In theory other differences could occur, but in practice there's maybe 10 platforms instead of 3, and that's not unreasonable. Reading a comment on the philikon article (http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-eg...), I also notice that Enthought has done some work on this, it seems by fixing up the binary packages at install time. This seems to be related to an entirely different issue of the location of libraries and binary incompatibilities, which I only slightly understand. -- Ian Bicking : ianb@colorstudy.com : http://blog.ianbicking.org

Alberto Valverde wrote:

...

Alberto Valverde wrote: [...]

...

...

No... which makes binary eggs unusable on Linux. I feel like there was something else that made binary packages on a Mac unreliable, but I can't remember. Windows binary eggs generally work fine. This is discussed some here: http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-eg...

I remember reading that... Perhaps a solution could be to"freeze" the tar.gzs too, beside the compiled eggs in case we're lucky, and hope that a compiler + libs are there when the install script is thawed. If the compiled egg won't cut it, then try to compile form source. I'm not sure how to tell easy_install to download the source distributions though. Well, it'll make eggs from your tarballs anyway. Turning it into a build process is a bit of a nuisance... I was hoping for something that didn't require building, but just did installation.

Yes, installation is the primary concern. I was suggesting the possibility of bundling the source distributions of platform dependent eggs just in case we can compile them by any chance if there's no pre-compiled binary available. Just like easy_install does when downloading from an index, ut without net access.

Probably we're both having the same problem, which is that we're using easy_install to download packages, but then of course it also installs the packages, which is almost okay so long as you are okay with an egg. But if it just downloaded, that'd be better. I'm sure some digging would reveal a way to just download.

...

That said, it might work. Maybe PoachEggs (mentioned later) does what you want in this case. Or, maybe it can be slightly modified to do what you want (I think it might also unintentionally turn tarballs into eggs).

I took a look at it as a source of ideas when experimenting with all this stuff. One reason I discarded it was because I was under the impression that you need to manually provide a requirements list and I wanted something as automatic as possible, which you could say "get me TG + all it's deps" and it'll use easy_install to satisfy those deps. for you.

However, no that I've looked at it more closely it seems that it can piggy-back on virtualenv to build this list, right?

It lets you be more specific about packages than setup.py, but it doesn't require you to enumerate everything. If you install Pylons it'll pick up all of Pylons' requirements. The idea of PoachEggs, though, is that often a second-level requirement effects the stability of an application. So if you require Pylons==0.9.6, but Pylons doesn't require a specific version of Beaker, you might have problems when a newer version of Beaker appears. Relatedly, it's awkward to even put a requirement like Pylons==0.9.6 in your application, and you may not know one way or the other if Pylons 0.9.7 would work. So PoachEggs just lets you put all these requirements in a separate file, outside of any setup.py. You can also get a working environment, and then "freeze" the requirements to a set of exact versions of packages. It's equivalent to what people are doing with custom package indexes.

...

I have thought about putting something in virtualenv to make relocating the environment easier. There's only a few things that need to be modified, I think. That might mitigate some of these issues.

Hmm, like being able to just zip the whole virtual environment and unzip elsewhere and it just works? That would be awesome indeed, and make eggfreezer unnecessary except perhaps the little code to create an inline tarball.

Well, I don't know if it would just work. There are some paths that need to be fixed. I was thinking about a specific command you'd run, say "python -m virtualenvmove", and it'd rewrite a few things. But I suppose I could put a check in site.py, and do the move anytime site.py moves.

...

...

...

and generating an "install" .py file that determines the platform and downloads the appropriate platform bundle.

Hmm, this "download" is precisely what I'm trying to avoid. My main use case is: A machine has gone down and I want to quickly put back everything together in another machine with a backup and something that contains *all* needed software. Sort of like an apt reposisory inside a dvd which lets you install debian on a machine with absolutely not net access. The "no-net" condition is just there to guarantee that all deps will be available no matter how old and discontinued they are (which if I think about it is rather ambitious... well, at least make it more likely than with the current situation) With PoachEggs I've now got it working so you can build a working environment, create a "requirements" file that lists everything in that working environment, download all the eggs for that into a directory, and then later install from that directory and disallow network access. Well, more-or-less. It's not a single-file install like eggfreezer, but they are working toward similar goals.

Yes, almost the same goals although I'd like to keep eggfreezer simple by leaving all the hard work of finding the right distributions to easy_install as much as possible.

PoachEggs is mostly focused on finding the right distributions.

...

The root (well, *one* root) of the problem is setuptools/distutils not getting the platform really right, but there's also all kinds of messy backward compatibility issues there, and no backward compatibility issues for eggfreezer. I'm not sure there aren't other issues. I'm also not sure that there isn't a finite number of resolvable issues. So maybe MacPorts and fink and system python on Macs are different. But that's just 3 platforms instead of 1, it's not an infinite number. And UCS2 Python is different from UCS4 on Linux, but that's really the one issue I know of where Linux Pythons differ. In theory other differences could occur, but in practice there's maybe 10 platforms instead of 3, and that's not unreasonable.

I've already got an idea on how to handle these special flavors inside eggfreezer, the only thing I'm missing are the heuristics to detect the platform's flavor which I cannot really investigate ATM, nor in the near future, due to lack of testing machines and experience on them (I've never had the need to compile anything on windows and never done, for example).

So, if someone can point me to, or provide, something I can use to reliably detect a platform and its peculiar flavor I could implement this in short time, I guess.

I'm not sure, but maybe u'a'.encode('unicode_internal') shows UCS2/4? A quick test appears to saw yes -- the result is 'a\x00\x00\x00' on a UCS4 build, 'a\x00' on a UCS2 build.

...

Reading a comment on the philikon article (http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-eg...), I also notice that Enthought has done some work on this, it seems by fixing up the binary packages at install time. This seems to be related to an entirely different issue of the location of libraries and binary incompatibilities, which I only slightly understand.

Very interesting... some code doing this is not available anywhere for me to study/steal, right? :)

Probably, but you'd have to ask the Enthought guys. -- Ian Bicking : ianb@colorstudy.com : http://blog.ianbicking.org

Ian Bicking, on 2008-08-07:

Probably we're both having the same problem, which is that we're using easy_install to download packages, but then of course it also installs the packages, which is almost okay so long as you are okay with an egg. But if it just downloaded, that'd be better. I'm sure some digging would reveal a way to just download.

Hi, Allow me to chip in with a few thoughts. I have created z3c.recipe.eggbasket that may help there, if only as a code example. It is a recipe for zc.buildout, though you can use it outside of a buildout with some lines of python like this: import z3c.recipe.eggbasket.utils z3c.recipe.eggbasket.utils.create_source_tarball( egg = 'grok', versionfile = 'http://grok.zope.org/releaseinfo/grok-0.13.cfg') With those arguments it creates a tarball with grok and all its dependencies, using the versions as specified in the versionfile. The tarball contains what I would call 'source eggs', so the files downloaded from the cheese shop. As an extra it looks for Windows binary eggs for all those dependencies. This is the result: http://grok.zope.org/releaseinfo/grok-eggs-0.13.tgz (That tarball is used within grokproject -using another part of this same eggbasket recipe- to avoid install failures due to servers being down, thereby lessening the number of single points of failure. See downloader.py.) Look at utils.py in that package for how this is done. The important line is this one: zc.buildout.easy_install.download_cache(cache) So zc.buildout does the real work here. And it probably is just a thin layer around setuptools in this spot. BTW, this package was inspired by the zc.sourcerelease package by Jim Fulton, stealing some ideas there. Try that package if you want to giftwrap a complete buildout with all source eggs and any other downloaded stuff, including an install.py that you should run after unpacking the generated tarball and that takes care of building up the buildout again.

The idea of PoachEggs, though, is that often a second-level requirement effects the stability of an application. So if you require Pylons==0.9.6, but Pylons doesn't require a specific version of Beaker, you might have problems when a newer version of Beaker appears. Relatedly, it's awkward to even put a requirement like Pylons==0.9.6 in your application, and you may not know one way or the other if Pylons 0.9.7 would work.

So PoachEggs just lets you put all these requirements in a separate file, outside of any setup.py. You can also get a working environment, and then "freeze" the requirements to a set of exact versions of packages. It's equivalent to what people are doing with custom package indexes.

Sounds like we have some overlap between PoachEggs and z3c.recipe.eggbasket (and zc.buildout itself). Cheers, -- Maurits van Rees | http://maurits.vanrees.org/ Work | http://zestsoftware.nl/ "This is your day, don't let them take it away." [Barlow Girl]

zooko schrieb:

On Aug 5, 2008, at 10:07 AM, Ian Bicking wrote:

...

No... which makes binary eggs unusable on Linux. I feel like there was something else that made binary packages on a Mac unreliable, but I can't remember.

Perhaps it was http://bugs.python.org/setuptools/issue19 (Will setuptools on Mac Python accept fat eggs?).

Yes, that's what I meant. setuptools on Mac OS X 10.5 does not accept 'fat' binary eggs build on 10.4. Diez. R. Roggisch schrieb:

Why shouldn't they?

Or can you install all of the following on Leopard? http://files.turbogears.org/eggs/simplejson-1.9.1-py2.5-macosx-10.3-fat.egg http://files.turbogears.org/eggs/RuleDispatch-0.5a0.dev_r2306-py2.5-macosx-1... http://files.turbogears.org/eggs/PyProtocols-1.0a0dev_r2302-py2.5-macosx-10.... http://files.turbogears.org/eggs/Cheetah-2.0rc8-py2.5-macosx-10.3-fat.egg Chris