something else to throw into the static metadata mix
Hi All, Sorry to bring this up if it makes things more difficult, if it's been solved or is covered by existing proposals, please point me in the right way. I've had a bit of a torrid time building psycopg2 from source of late, and in my dream world, being able to specify psycopg2 as a dependent egg would "just work" and pull down the right statically linked binary egg. This raises a problem that I haven't seen covered anywhere: not only would these eggs have to be platform dependent, they'd also need to be dependent on the required version of postgres. Does setuptools/buildout/any other existing or proposed packaging solution cater for this kind of requirement? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Chris Withers a écrit :
Hi All,
Sorry to bring this up if it makes things more difficult, if it's been solved or is covered by existing proposals, please point me in the right way.
I've had a bit of a torrid time building psycopg2 from source of late, and in my dream world, being able to specify psycopg2 as a dependent egg would "just work" and pull down the right statically linked binary egg.
This raises a problem that I haven't seen covered anywhere: not only would these eggs have to be platform dependent, they'd also need to be dependent on the required version of postgres.
Does setuptools/buildout/any other existing or proposed packaging solution cater for this kind of requirement?
You can with minitage [1]. If you are right into a minitage env. and you have specified that postgresql is a dependency of your project (minitage section or minibuild) and you have built it (eg: postgresql-8.4), the egg recipe (minitage.recipe.*) will know how to use and to point to it (update PATH/LDFLAGS/CFLAGS/RPATH/PKGCONFIG and so on )
cheers,
Chris
[1] http://www.minitage.org -- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF
kiorky wrote:
Does setuptools/buildout/any other existing or proposed packaging solution cater for this kind of requirement?
You can with minitage [1].
I'm afraid I'm more interested in what will be the "standard python way" of doing this. I might be wrong, but I get the impression Minitage tries to do too much to be interesting for me... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Chris Withers a écrit :
kiorky wrote:
Does setuptools/buildout/any other existing or proposed packaging solution cater for this kind of requirement?
You can with minitage [1].
I'm afraid I'm more interested in what will be the "standard python way" of doing this.
I might be wrong, but I get the impression Minitage tries to do too much to be interesting for me... Minitage recipes which deal with eggs use setuptools/buildout under the hood, The goal is not to reinvent the wheel but to isolate, integrate and automate. I'm afraid it use the python way to do and nothing else. Its actions are just integrations betweens all components to make a real project deployment complete in a nice way. This include for example: - setting an egg cache somewhere to share between projects. - having a well known layout to install stuff inside - compiling with -rpath for dynamic libraries - having hook architecture to make easy scripting around each installation step - having a good patching facility (on the fly egg patching for example)
Chris
-- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF
On Wed, Sep 30, 2009 at 5:00 PM, Chris Withers <chris@simplistix.co.uk> wrote:
Hi All,
Sorry to bring this up if it makes things more difficult, if it's been solved or is covered by existing proposals, please point me in the right way.
I've had a bit of a torrid time building psycopg2 from source of late, and in my dream world, being able to specify psycopg2 as a dependent egg would "just work" and pull down the right statically linked binary egg.
This raises a problem that I haven't seen covered anywhere: not only would these eggs have to be platform dependent, they'd also need to be dependent on the required version of postgres.
I think it's another issue, more related to the way easy_install or pip works with PyPI (using package_index.py) once those scripts have a list of dependencies, they are looking for it at PyPI using the simple index, and download the "best pick", and binary releases for your platform will come first. So, for pyscopg, if this page : pypi.python.org/simple/psycopg would contain the binary distribution for your platform, it would work. but there's only the source release, and the home url doesn't contain anything useful, so it grabs the source version for your case. zope.interface for example has binary releases for win32 so win32 users don't have to compile it: pypi.python.org/simple/zope.interface
Does setuptools/buildout/any other existing or proposed packaging solution cater for this kind of requirement?
cheers,
Chris
-- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
Tarek Ziadé wrote:
I think it's another issue, more related to the way easy_install or pip works with PyPI (using package_index.py)
I think you're missing my point: how would the package be named and/or how would you express in static metadata, that a particular version of the package is for postgres 8 rather than postgres 7, or 8.4 rather than 8.3? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Wed, Sep 30, 2009 at 6:28 PM, Chris Withers <chris@simplistix.co.uk> wrote:
Tarek Ziadé wrote:
I think it's another issue, more related to the way easy_install or pip works with PyPI (using package_index.py)
I think you're missing my point: how would the package be named and/or how would you express in static metadata, that a particular version of the package is for postgres 8 rather than postgres 7, or 8.4 rather than 8.3?
I am not sure to understand, you mean within the psycopg2 project metadata ? Since it's a system requirement, it's out of the metadata scope. So unless the project provides a binary version your system will have to be able to compile source version. I think psycopg2 deals like python-ldap for this : they depend on system libraries, and a given source distribution is supposed to compile against one given version of each of these libraries. It's up to the project to have a name or a version that corresponds to the lib it wraps. I think a good practice is to have a version number partially synchronized with the lib the project wraps. I know that python-ldap version numbers have nothing to do with open ldap version, and that compilation might fail if you don't have the right open ldap, and it's hard to know which version matches. Now for the static metadata, if you add 'pyscopg2' to the dependencies, if this project provides binary releases with statically compiled libraries, that matches your platform, it'll work. lxml is a good example for this, it ships with libxml and libxslt in binary releases now : http://pypi.python.org/simple/lxml But it's out of the scope of the static metadata to describe these system dependencies : the dependencies are only *python* dependencies, so the micro-language we will provide does only concerns the Python environment.
cheers,
Chris
-- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
-- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
Quoting Tarek Ziadé <ziade.tarek@gmail.com>:
On Wed, Sep 30, 2009 at 6:28 PM, Chris Withers <chris@simplistix.co.uk> wrote:
Tarek Ziadé wrote:
I think it's another issue, more related to the way easy_install or pip works with PyPI (using package_index.py)
I think you're missing my point: how would the package be named and/or how would you express in static metadata, that a particular version of the package is for postgres 8 rather than postgres 7, or 8.4 rather than 8.3?
But it's out of the scope of the static metadata to describe these system dependencies : the dependencies are only *python* dependencies, so the micro-language we will provide does only concerns the Python environment.
I agree with Tarek that the system dependency handling is not something that should be dealt with in the python package metadata. The best thing that can be hoped for is that packages which rely on a specific version of a system dependency will throw a descriptive error indicating this particular problem. The MySQL_python project is one example of a project that is already doing this, since they throw an ImportError with a descriptive title in this case. -Chris
cheers,
Chris
-- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
-- Tarek Ziadé | http://ziade.org | オープンソースはすごい! _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Wed, 30 Sep 2009 12:26:15 -0700, <cgalvan@mail.utexas.edu> wrote:
I agree with Tarek that the system dependency handling is not something that should be dealt with in the python package metadata. The best thing that can be hoped for is that packages which rely on a specific version of a system dependency will throw a descriptive error indicating this particular problem. The MySQL_python project is one example of a project that is already doing this, since they throw an ImportError with a descriptive title in this case.
The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py) -srid
2009/9/30 Sridhar Ratnakumar <sridharr@activestate.com>
On Wed, 30 Sep 2009 12:26:15 -0700, <cgalvan@mail.utexas.edu> wrote:
I agree with Tarek that the system dependency handling is not something
that should be dealt with in the python package metadata. The best thing that can be hoped for is that packages which rely on a specific version of a system dependency will throw a descriptive error indicating this particular problem. The MySQL_python project is one example of a project that is already doing this, since they throw an ImportError with a descriptive title in this case.
The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py)
Just to throw more in the mix -- this option is totally ad hoc in lxml, which means it's hard to handle in an installer like easy_install or pip. That is, there's no good way to do "easy_install --static lxml" (it's more possible in pip, but still quite awkward). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker
On Wed, 30 Sep 2009 13:58:56 -0700, Ian Bicking <ianb@colorstudy.com> wrote:
The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py) Just to throw more in the mix -- this option is totally ad hoc in lxml, which means it's hard to handle in an installer like easy_install or pip. That is, there's no good way to do "easy_install --static lxml" (it's more possible in pip, but still quite awkward).
Maybe we can standardize the ``--static`` flag for setup.py and installers? The benefit of this is that builders and installers can support a static build feature without having to hack build scripts for specific packages (Currently PyPM's backend maintains custom code for building such special packages). Examples of usage:: $ python setup.py build --static install $ easy_install --static lxml $ pip install --install-option="--static" pip -srid
Sridhar Ratnakumar wrote:
On Wed, 30 Sep 2009 13:58:56 -0700, Ian Bicking <ianb@colorstudy.com> wrote:
The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py) Just to throw more in the mix -- this option is totally ad hoc in lxml, which means it's hard to handle in an installer like easy_install or pip. That is, there's no good way to do "easy_install --static lxml" (it's more possible in pip, but still quite awkward).
Maybe we can standardize the ``--static`` flag for setup.py and installers? The benefit of this is that builders and installers can support a static build feature without having to hack build scripts for specific packages (Currently PyPM's backend maintains custom code for building such special packages).
Examples of usage::
$ python setup.py build --static install $ easy_install --static lxml $ pip install --install-option="--static" pip
I like your thoughts here, but there currently isn't a way to distinguish between an egg that contains binaries that were linked statically vs. dynamically. It seems like in that case we would need some distinguishing flag in an egg name to signify it was linked statically, and at that point it seems like we are trying to do too much. Thoughts? -Chris
-srid
Chris Galvan wrote:
I like your thoughts here, but there currently isn't a way to distinguish between an egg that contains binaries that were linked statically vs. dynamically. It seems like in that case we would need some distinguishing flag in an egg name to signify it was linked statically, and at that point it seems like we are trying to do too much.
Interestingly, I read that as "we currently use the egg name as a source of static metadata, and it's getting 'too full' to be used for that" ;-) I'm guessing the static metadata that we've been vigorously discussing would be the place to keep this kind of information, and the filename should no longer matter? This would certainly help with things like PIL, which is constantly being repackaged because setuptools didn't like the name that the Effbot chose... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Python installers/package managers are not there to install system dependencies. For me, it s totally out of scope and can lead to security holes. Eg, you include a lib "a" (dynamic or static), and it has a hole, to fix it, you must re-release you egg and the users must reinstall it on their side. But if you use the system provided "library", no matter that it was provided by the host system or an isolation software like minitage: If you choose to use dynamic linking when building the egg: * You must only rebuild the library, all is dynamic. If you choose to use static linking when building the egg: * You must rebuild the implied egg too and in the meantime you are exposed to the security hole. Another reasons not to include system dependencies in the python dependencies expressions is that the system dependencies are: - To use the system package manager, you must have great privileges - Installation is not portable, there is plenty of package managers outside - Even if you have support for the underlying package manager, the user may want not to use the system sofware but a custom version of the library built somewhere. And it will be a PÏTA and for the metadata parser to have the knowlegde about it. static is not a solution, even not an option for me except in very rare particular cases like having the binutils (/bin/ls, /bin/mount ...) static for recover puposes... Sridhar Ratnakumar a écrit :
On Wed, 30 Sep 2009 13:58:56 -0700, Ian Bicking <ianb@colorstudy.com> wrote:
The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py) Just to throw more in the mix -- this option is totally ad hoc in lxml, which means it's hard to handle in an installer like easy_install or pip. That is, there's no good way to do "easy_install --static lxml" (it's more possible in pip, but still quite awkward).
Maybe we can standardize the ``--static`` flag for setup.py and installers? The benefit of this is that builders and installers can support a static build feature without having to hack build scripts for specific packages (Currently PyPM's backend maintains custom code for building such special packages).
Examples of usage::
$ python setup.py build --static install $ easy_install --static lxml $ pip install --install-option="--static" pip
-srid _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF
And another thing is that to support multi-installation with differents versions/packages for projects on the same machine. If you just have a "global" system installation as a base, it is a problem. You must have a way to hook to use your software and you must have means to make multi-installations side by side isolated from each other. kiorky a écrit :
Python installers/package managers are not there to install system dependencies.
For me, it s totally out of scope and can lead to security holes. Eg, you include a lib "a" (dynamic or static), and it has a hole, to fix it, you must re-release you egg and the users must reinstall it on their side.
But if you use the system provided "library", no matter that it was provided by the host system or an isolation software like minitage: If you choose to use dynamic linking when building the egg: * You must only rebuild the library, all is dynamic.
If you choose to use static linking when building the egg: * You must rebuild the implied egg too and in the meantime you are exposed to the security hole.
Another reasons not to include system dependencies in the python dependencies expressions is that the system dependencies are: - To use the system package manager, you must have great privileges - Installation is not portable, there is plenty of package managers outside - Even if you have support for the underlying package manager, the user may want not to use the system sofware but a custom version of the library built somewhere. And it will be a PÏTA and for the metadata parser to have the knowlegde about it.
static is not a solution, even not an option for me except in very rare particular cases like having the binutils (/bin/ls, /bin/mount ...) static for recover puposes...
Sridhar Ratnakumar a écrit :
On Wed, 30 Sep 2009 13:58:56 -0700, Ian Bicking <ianb@colorstudy.com> wrote:
The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py) Just to throw more in the mix -- this option is totally ad hoc in lxml, which means it's hard to handle in an installer like easy_install or pip. That is, there's no good way to do "easy_install --static lxml" (it's more possible in pip, but still quite awkward). Maybe we can standardize the ``--static`` flag for setup.py and installers? The benefit of this is that builders and installers can support a static build feature without having to hack build scripts for specific packages (Currently PyPM's backend maintains custom code for building such special packages).
Examples of usage::
$ python setup.py build --static install $ easy_install --static lxml $ pip install --install-option="--static" pip
-srid _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
------------------------------------------------------------------------
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF
Chris Withers wrote:
I think you're missing my point: how would the package be named and/or how would you express in static metadata, that a particular version of the package is for postgres 8 rather than postgres 7, or 8.4 rather than 8.3?
Just to add my 2 cents: It seems to me that there are two sets of metadata getting confused here. One is the existing set including the Python dependencies. This set is meant to describe what the package does and relies on conceptually. The other is a set of system dependencies that describe in what precise environment a binary egg happens to have been built. The latter are currently expressed only in the file name of the egg or not at all. The set of system dependencies can be taken to be empty for a source egg and includes the target platform and architecture, some of the options Python was built with (what flavour of unicode, which optional libraries compiled in), and a set of third-party system libraries with or without version restrictions. Of the system dependencies, only the platform and architecture are currently accessible (through the egg file name) when downloading an egg, which is one of the reasons why releasing binary eggs for Linux is so unpopular - you might just end up with an unforeseeable mismatch of unicode flavour between your Python and the binary egg. In a dream world, a package index would have an API that allows an installer to query what system dependencies are relevant to binary eggs of a given package and for which combinations of values binary eggs actually exist. The installer could then figure out which of them fits its actual environment best or whether to get the source egg and roll its own binary. -- Thomas
participants (8)
-
cgalvan＠mail.utexas.edu -
Chris Galvan
-
Chris Withers -
Ian Bicking -
kiorky -
Sridhar Ratnakumar -
Tarek Ziadé -
Thomas Lotze