-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. I'm pleased to announce the eggbuild package. This is a proof of concept for some ideas I have in mind for Python packages distribution. The package name, the implemented setuptools command names and the API are unstable, and may change in the future. The project is currently available at: http://hg.mperillo.ath.cx/eggbuild. The README and TODO file contain some informations, and I will try to not replicate them here. Basically, the idea is to build a setuptools distribution format that, like PyInstaller, is *self contained* (containing all required packages and all external shared libraries). However, unlike PyInstaller, there are some important differences, listed in the README file. I have tested the eggbuild command with a patched psycopg2 package, on a Linux Debian Squeeze system. Just for information, the resulting egg is 1.4 Mb big, when a normal egg is 156 Kb big. The resulting EGG-INFO\native_libs file is available here: http://paste.pocoo.org/show/285677/ The patch for psycopg2 is available here: http://paste.pocoo.org/show/285662/ The code *seems* to work. When importing the psycopg2 module, the *correct* shared libraries are loaded. At least the correct libpq.so.5 library is used. The other libraries required by libpq.so.5 are searched in the system default location, and I need to check if this can be solved without having to patch each library (adding LD_RPATH). A simple solution is to set LD_LIBRARY_PATH environment variable (but there are some issues to be solved). Now, what I want to implement is a setuptools command that will build a Debian package, with the following features: * pre-install script will create a virtual environment, using virtualenv. The virtualenv should be created in /opt/<pkg-name>. * the package and required packages are stored in the deb data file, in egg binary format, and will be automatically copied to the virtual python installation. The main feature to implement is, however, a full MSI installer. However I don't know how MSI is supposed to work, and I will probably need some help (at least some good tutorial about MSI technology, what can be and what can not be done). I will appreciate comments and suggestions about the eggbuild design and implementation. Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzRxNwACgkQscQJ24LbaUQimACfRhi6kJUbPR8RkOz16Zcjni8T AA8AmQGqdakXHSuZTMc1oKbOaGPJKq2p =Pwqc -----END PGP SIGNATURE-----
On Wed, Nov 3, 2010 at 9:23 PM, Manlio Perillo <manlio.perillo@gmail.com> wrote:
Hi.
Ciao Manlio!
Basically, the idea is to build a setuptools distribution format that, like PyInstaller, is *self contained* (containing all required packages and all external shared libraries).
However, unlike PyInstaller, there are some important differences, listed in the README file.
What are the key *advantages* ? From what I can see the great difference is in packaging, since it aims at providing automatic multi-format output support; but there're already some efforts to create a .deb or a .rpm from an arbitrary, well formed python project; I'm not sure about MSI installers, but this still seems to carry some problems, as you say later. Maybe an automated support for pyinstaller packaging would be more useful, as it's already tested and being used around with success; otherwise you'd be duplicating a lot of effort from that project. Also, check zc.buildout: by properly setting the paths inside a "main dir", you can get a great jar-like effect and achieve a result which is very similar to your original intent, e.g. you can have an /opt/something dir holding your code, your scripts, and all downloads & dependency eggs. And it's *really* easy to automatically create an rpm once your project is setup that way, I think zc.buildout is already got a recipe for RPMs about that. -- Alan Franzoni -- contact me at public@[mysurname].eu
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 04/11/2010 16:52, Alan Franzoni ha scritto:
On Wed, Nov 3, 2010 at 9:23 PM, Manlio Perillo <manlio.perillo@gmail.com> wrote:
Hi.
Ciao Manlio!
Ciao Alan!
Basically, the idea is to build a setuptools distribution format that, like PyInstaller, is *self contained* (containing all required packages and all external shared libraries).
However, unlike PyInstaller, there are some important differences, listed in the README file.
What are the key *advantages* ?
PyInstaller and py2exe do have some problems. For a project using Qt I was unable to use py2exe, and I was unable to understand how to use PyInstaller (!). Moreover, I don't see reasons why the Python interpreter should be included in the distribution. Another interesting feature of eggbuild is that it produces standard eggs. Not only this means that these eggs can be made available from something like PyPi, but this also means that they can be easy installed. This make distribution more easy, IMHO.
From what I can see the great difference is in packaging, since it aims at providing automatic multi-format output support; but there're already some efforts to create a .deb or a .rpm from an arbitrary, well formed python project;
Yes, the problem is that (assuming you are speaking about plain distutils): 1) packages dependencies are not handled 2) "external" shared libraries are not handled
[...]
Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzS5PIACgkQscQJ24LbaURnYQCeI2esQYELtnsFVH8r7iEqgauc 88gAniMDD0qQc5+L21okaZrdYezcAXwV =V8St -----END PGP SIGNATURE-----
On 11/4/10 5:53 PM, Manlio Perillo wrote:
For a project using Qt I was unable to use py2exe, and I was unable to understand how to use PyInstaller (!).
That's quite a huge issue :-) but I think we've used PyInstaller internally ad it seemed to work. Have you asked for help on the pyinstaller ml and got no response? Ping Giovanni directly, then :P
Moreover, I don't see reasons why the Python interpreter should be included in the distribution.
If you're distributing an an app, the end user will probably just want it to be working without the hassle of installing it by himself - in this sense distributing the interpreter along the app is good. If you're distributing a library then you'd probably not need the eggbuild at all, since developers will very likely know their work. What specific problem are you trying to address?
Yes, the problem is that (assuming you are speaking about plain distutils):
1) packages dependencies are not handled
I must admit I've never tried auto-building Python extensions, but I've tried zc.buildout to deliver and it works. I can post an example of how I do use it and then you can take a peek if it's suitable for your use. I'll do it tomorrow, now I'm a bit in a rush.
2) "external" shared libraries are not handled
External shared libraries might be an issue. If there're enough deps you might just end up packaging half a system shared libraries in order to distribute your packages in a self-contained way. It's quite risky, my approach with RPMs is to let the packaging system handle system-wide deps, and use zc.buildout to handle python eggs and extensions. -- Alan Franzoni contact me at public@[mysurname].eu
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 04/11/2010 18:16, Alan Franzoni ha scritto:
On 11/4/10 5:53 PM, Manlio Perillo wrote:
For a project using Qt I was unable to use py2exe, and I was unable to understand how to use PyInstaller (!).
That's quite a huge issue :-) but I think we've used PyInstaller internally ad it seemed to work. Have you asked for help on the pyinstaller ml and got no response? Ping Giovanni directly, then :P
I just read the documentation and tried to use it.
Moreover, I don't see reasons why the Python interpreter should be included in the distribution.
If you're distributing an an app, the end user will probably just want it to be working without the hassle of installing it by himself - in this sense distributing the interpreter along the app is good.
If you're distributing a library then you'd probably not need the eggbuild at all, since developers will very likely know their work.
What specific problem are you trying to address?
Distribute a Python package on Windows. Suppose package A depends on pure Python package B and on PyQT. Right now I have two options: 1) create an installer using PyInstaller or py2exe 2) have the user install Python (not a problem), PyQt (the "all included" installer) and then easy_install my application I think this is a big problem. Even if we assume that it is ok to create a PyInstaller installer, I don't see why a Windows user must not be able to do: easy_install A and have B and PyQt (with external libraries) correctly installed on the system. eggbuild tries to solve this problem. It build eggs that "embeds" all required shared libraries (excluding shared libraries already required by the Python interpreter -- this usually allow me to forget about SxS mess with Visual Studio redistributable libraries).
Yes, the problem is that (assuming you are speaking about plain distutils):
1) packages dependencies are not handled
I must admit I've never tried auto-building Python extensions, but I've tried zc.buildout to deliver and it works. I can post an example of how I do use it and then you can take a peek if it's suitable for your use.
I have never used zc.buildout; however, as I wrote, what I'm trying to do is to solve the "external libraries" problem using standard eggs. Building an all included installer its the second step. For me it's already good if easy_install is able to install binary eggs with external libraries embedded! Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzS+pQACgkQscQJ24LbaUQ+PwCffNKkoaYfTig3uHucDaQ2mL84 GYcAn35I4PBVMAR1QnHwC/g9XXv0IM2l =kQCO -----END PGP SIGNATURE-----
On 11/4/10 7:25 PM, Manlio Perillo wrote:
What specific problem are you trying to address?
Distribute a Python package on Windows.
Suppose package A depends on pure Python package B and on PyQT. Right now I have two options:
1) create an installer using PyInstaller or py2exe 2) have the user install Python (not a problem), PyQt (the "all included" installer) and then easy_install my application
I think this is a big problem. Even if we assume that it is ok to create a PyInstaller installer, I don't see why a Windows user must not be able to do: easy_install A
and have B and PyQt (with external libraries) correctly installed on the system.
OK, I got it. I must admit that my knowledge of Windows is not advanced enough to tell you whether this could work or if there're obvious issues. This could be a problem on Linux as well, by the way. If I try installing, let's say, lxml, and libxml2 and libxml2-dev are not installed on my system, easy_install will fail. Those are my 2c for the "weak spots": - you need to check recursively for all the shared libraries needed by python modules and extensions, and all the shared libraries needed by those shared libraries, and so on. Eggbuilds might end up just like huge chroots. If you want to, let's say, distribute something built with PyQt on Linux, you'll need PyQt, the QT Runtime, then? most probably X libs, libc and something else. Do you *really* want to distribute everything? I've just checked for your EGG-INFO/native_libs: you don't seem to recurse libraries. Just an example I spotted: libgnutls requires zlib, but no libz.so.something is included in native libs. - you need to check for dlopen() calls as well (or LoadLibrary/Ex on Windows - but I don't know whether it's easy to look for them in PE executables or DLLs); this might not be trivial and will increase the number of libraries that need to be included. - there might be legal implications when redistributing shared libraries; it might not be easy to tell which license each library has, and a single non-redistributable library might prevent to release the whole package. GPL and LGPL mandates source distribution along binary distribution, are you ready to package and make all the sources available to the user? Remember that, as you will be the redistributor, this requirement must be met by you - you can't just tell people to fetch sources from this or that environment. - reproducible builds will be almost impossible - if a dep changes on your system, you might end up without a clear understanding of what happened between one build and another - apparently identical - build. You may resort to chroot-building tools like pbuilder or mock on Linux, but this will be far more difficult on Windows. My idea is that everything looks far too complex, and would work easily only for really simple cases. Everything else will just turn overcomplicated. You're really trying to port to Python what Java does through its classpath system and, to a greater extent, with Osgi; but Python philosophy is quite different, you'll end up distributing whole OSes! -- Alan Franzoni contact me at public@[mysurname].eu
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 04/11/2010 22:01, Alan Franzoni ha scritto:
On 11/4/10 7:25 PM, Manlio Perillo wrote:
What specific problem are you trying to address?
Distribute a Python package on Windows.
Suppose package A depends on pure Python package B and on PyQT. Right now I have two options:
1) create an installer using PyInstaller or py2exe 2) have the user install Python (not a problem), PyQt (the "all included" installer) and then easy_install my application
I think this is a big problem. Even if we assume that it is ok to create a PyInstaller installer, I don't see why a Windows user must not be able to do: easy_install A
and have B and PyQt (with external libraries) correctly installed on the system.
OK, I got it.
I must admit that my knowledge of Windows is not advanced enough to tell you whether this could work or if there're obvious issues.
I will do some tests on Windows when I find some free time. But it should work.
This could be a problem on Linux as well, by the way. If I try installing, let's say, lxml, and libxml2 and libxml2-dev are not installed on my system, easy_install will fail.
Yes. But on Linux systems at least we have package managers, and a compiler is almost always available,
Those are my 2c for the "weak spots":
- you need to check recursively for all the shared libraries needed by python modules and extensions, and all the shared libraries needed by those shared libraries, and so on.
This is what I do.
Eggbuilds might end up just like huge chroots.
If you want to, let's say, distribute something built with PyQt on Linux, you'll need PyQt, the QT Runtime, then? most probably X libs, libc and something else. Do you *really* want to distribute everything?
A simple solution is to specify, in the setup.cfg file, a list of "know" libraries that you can assume are always available on the system.
I've just checked for your EGG-INFO/native_libs: you don't seem to recurse libraries. Just an example I spotted: libgnutls requires zlib, but no libz.so.something is included in native libs.
The reason libz.so is not included is because it is already required by libpython. I assume that libraries required by libpython are available on the system, and I avoid to copy them in the egg.
- you need to check for dlopen() calls as well (or LoadLibrary/Ex on Windows - but I don't know whether it's easy to look for them in PE executables or DLLs); this might not be trivial and will increase the number of libraries that need to be included.
I assume only few projects use dlopen/LoadLibrary/Ex. This problem, moreover, is so complex that can not even be solved by advanced package managers; why should I try to solve it? :)
- there might be legal implications when redistributing shared libraries; it might not be easy to tell which license each library has, and a single non-redistributable library might prevent to release the whole package.
Yes, this is indeed a problem. It is resposibility of the egg builder to check this.
GPL and LGPL mandates source distribution along binary distribution, are you ready to package and make all the sources available to the user? Remember that, as you will be the redistributor, this requirement must be met by you - you can't just tell people to fetch sources from this or that environment.
- reproducible builds will be almost impossible - if a dep changes on your system, you might end up without a clear understanding of what happened between one build and another - apparently identical - build.
As far as I can tell, the same can happen with PyInstaller. And the same can happen, as an example, when riverbank builds the PyQt installer.
You may resort to chroot-building tools like pbuilder or mock on Linux, but this will be far more difficult on Windows.
My idea is that everything looks far too complex, and would work easily only for really simple cases. Everything else will just turn overcomplicated.
It is indeed a complex problem, but PyInstaller is far more complex.
You're really trying to port to Python what Java does through its classpath system and, to a greater extent, with Osgi; but Python philosophy is quite different, you'll end up distributing whole OSes!
I don't know Java :). Regards Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzTLycACgkQscQJ24LbaUSdoQCeJlSzjJ+iYR8gyqLu3RwKJFfk e4IAn0VrfK97p4p0TCsNyBxje26Qu74T =7xgA -----END PGP SIGNATURE-----
On 11/4/10 11:09 PM, Manlio Perillo wrote: Sorry for being late, I had a busy Friday.
If you want to, let's say, distribute something built with PyQt on Linux, you'll need PyQt, the QT Runtime, then? most probably X libs, libc and something else. Do you *really* want to distribute everything?
A simple solution is to specify, in the setup.cfg file, a list of "know" libraries that you can assume are always available on the system.
But you don't know if they are really available or if the *proper* version of them is available. This might make the whole project unuseful. And the behaviour of your project while specifying "known" libraries might just end up being very similar to PyInstaller - without the embedded interpreter. Another great issue with Python extensions is usually the ucs2/ucs4 problem; an extension compiled with ucs2 builtin won't work with a ucs4 interpreter and viceversa; that's probably a good reason alone to pack an interpreter along with the libs.
The reason libz.so is not included is because it is already required by libpython.
I think I can remember it is an optional dependency. If you assume a lib is available on your interpreter and the user's interpreter has been compiled in a different way you might just end up with many issues.
I assume only few projects use dlopen/LoadLibrary/Ex.
This might be an HUGE assumption. Have you verified that? You should at least look for dlopen() calls in your libraries and prevent building if that gets used. Otherwise your code might just unpredicably fail at runtime when the shared lib is opened.
- reproducible builds will be almost impossible - if a dep changes on your system, you might end up without a clear understanding of what happened between one build and another - apparently identical - build.
As far as I can tell, the same can happen with PyInstaller. And the same can happen, as an example, when riverbank builds the PyQt installer.
Sure it can happen.
My idea is that everything looks far too complex, and would work easily only for really simple cases. Everything else will just turn overcomplicated.
It is indeed a complex problem, but PyInstaller is far more complex.
But it's already there - why should you reinvent the wheel? Maybe you can just add some patches on top of it. Maybe you could just pick a scenario ( Windows XP? Vista? 7? 32-bit only?) and try to find out a narrow-scoped, quickly-available solution instead of going for the big, definitive, cross-platform solution. You might just say "hey, lib A B C are always available on Windows environments, pack everything else along." . I would keep advising packing the interpreter as well, by the way. Very important: I'm not a PyInstaller fan (I've used it a couple of times), I'd just like to prevent you from doing some work I think is unnecessary. -- Alan Franzoni contact me at public@[mysurname].eu
participants (2)
-
Alan Franzoni -
Manlio Perillo