Chris Withers a écrit :

kiorky wrote:

...

...

Does setuptools/buildout/any other existing or proposed packaging solution cater for this kind of requirement?

You can with minitage [1].

I'm afraid I'm more interested in what will be the "standard python way" of doing this.

I might be wrong, but I get the impression Minitage tries to do too much to be interesting for me... Minitage recipes which deal with eggs use setuptools/buildout under the hood, The goal is not to reinvent the wheel but to isolate, integrate and automate. I'm afraid it use the python way to do and nothing else. Its actions are just integrations betweens all components to make a real project deployment complete in a nice way. This include for example: - setting an egg cache somewhere to share between projects. - having a well known layout to install stuff inside - compiling with -rpath for dynamic libraries - having hook architecture to make easy scripting around each installation step - having a good patching facility (on the fly egg patching for example)

Chris

-- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF

Tarek Ziadé wrote:

I think it's another issue, more related to the way easy_install or pip works with PyPI (using package_index.py)

I think you're missing my point: how would the package be named and/or how would you express in static metadata, that a particular version of the package is for postgres 8 rather than postgres 7, or 8.4 rather than 8.3? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

Quoting Tarek Ziadé :

On Wed, Sep 30, 2009 at 6:28 PM, Chris Withers wrote:

...

Tarek Ziadé wrote:

...

I think it's another issue, more related to the way easy_install or pip works with PyPI (using package_index.py)

I think you're missing my point: how would the package be named and/or how would you express in static metadata, that a particular version of the package is for postgres 8 rather than postgres 7, or 8.4 rather than 8.3?

But it's out of the scope of the static metadata to describe these system dependencies : the dependencies are only *python* dependencies, so the micro-language we will provide does only concerns the Python environment.

I agree with Tarek that the system dependency handling is not something that should be dealt with in the python package metadata. The best thing that can be hoped for is that packages which rely on a specific version of a system dependency will throw a descriptive error indicating this particular problem. The MySQL_python project is one example of a project that is already doing this, since they throw an ImportError with a descriptive title in this case. -Chris

...

cheers,

Chris

-- Simplistix - Content Management, Batch Processing & Python Consulting           - http://www.simplistix.co.uk

-- Tarek Ziadé | http://ziade.org | オープンソースはすごい! _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

2009/9/30 Sridhar Ratnakumar

On Wed, 30 Sep 2009 12:26:15 -0700, wrote:

I agree with Tarek that the system dependency handling is not something

...

that should be dealt with in the python package metadata. The best thing that can be hoped for is that packages which rely on a specific version of a system dependency will throw a descriptive error indicating this particular problem. The MySQL_python project is one example of a project that is already doing this, since they throw an ImportError with a descriptive title in this case.

The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py)

Just to throw more in the mix -- this option is totally ad hoc in lxml, which means it's hard to handle in an installer like easy_install or pip. That is, there's no good way to do "easy_install --static lxml" (it's more possible in pip, but still quite awkward). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker

Sridhar Ratnakumar wrote:

On Wed, 30 Sep 2009 13:58:56 -0700, Ian Bicking wrote:

...

The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py) Just to throw more in the mix -- this option is totally ad hoc in lxml, which means it's hard to handle in an installer like easy_install or pip. That is, there's no good way to do "easy_install --static lxml" (it's more possible in pip, but still quite awkward).

Maybe we can standardize the ``--static`` flag for setup.py and installers? The benefit of this is that builders and installers can support a static build feature without having to hack build scripts for specific packages (Currently PyPM's backend maintains custom code for building such special packages).

Examples of usage::

$ python setup.py build --static install $ easy_install --static lxml $ pip install --install-option="--static" pip

I like your thoughts here, but there currently isn't a way to distinguish between an egg that contains binaries that were linked statically vs. dynamically. It seems like in that case we would need some distinguishing flag in an egg name to signify it was linked statically, and at that point it seems like we are trying to do too much. Thoughts? -Chris

-srid

Python installers/package managers are not there to install system dependencies. For me, it s totally out of scope and can lead to security holes. Eg, you include a lib "a" (dynamic or static), and it has a hole, to fix it, you must re-release you egg and the users must reinstall it on their side. But if you use the system provided "library", no matter that it was provided by the host system or an isolation software like minitage: If you choose to use dynamic linking when building the egg: * You must only rebuild the library, all is dynamic. If you choose to use static linking when building the egg: * You must rebuild the implied egg too and in the meantime you are exposed to the security hole. Another reasons not to include system dependencies in the python dependencies expressions is that the system dependencies are: - To use the system package manager, you must have great privileges - Installation is not portable, there is plenty of package managers outside - Even if you have support for the underlying package manager, the user may want not to use the system sofware but a custom version of the library built somewhere. And it will be a PÏTA and for the metadata parser to have the knowlegde about it. static is not a solution, even not an option for me except in very rare particular cases like having the binutils (/bin/ls, /bin/mount ...) static for recover puposes... Sridhar Ratnakumar a écrit :

On Wed, 30 Sep 2009 13:58:56 -0700, Ian Bicking wrote:

...

The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py) Just to throw more in the mix -- this option is totally ad hoc in lxml, which means it's hard to handle in an installer like easy_install or pip. That is, there's no good way to do "easy_install --static lxml" (it's more possible in pip, but still quite awkward).

Maybe we can standardize the ``--static`` flag for setup.py and installers? The benefit of this is that builders and installers can support a static build feature without having to hack build scripts for specific packages (Currently PyPM's backend maintains custom code for building such special packages).

Examples of usage::

$ python setup.py build --static install $ easy_install --static lxml $ pip install --install-option="--static" pip

-srid _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF

Chris Withers wrote:

I think you're missing my point: how would the package be named and/or how would you express in static metadata, that a particular version of the package is for postgres 8 rather than postgres 7, or 8.4 rather than 8.3?

Just to add my 2 cents: It seems to me that there are two sets of metadata getting confused here. One is the existing set including the Python dependencies. This set is meant to describe what the package does and relies on conceptually. The other is a set of system dependencies that describe in what precise environment a binary egg happens to have been built. The latter are currently expressed only in the file name of the egg or not at all. The set of system dependencies can be taken to be empty for a source egg and includes the target platform and architecture, some of the options Python was built with (what flavour of unicode, which optional libraries compiled in), and a set of third-party system libraries with or without version restrictions. Of the system dependencies, only the platform and architecture are currently accessible (through the egg file name) when downloading an egg, which is one of the reasons why releasing binary eggs for Linux is so unpopular - you might just end up with an unforeseeable mismatch of unicode flavour between your Python and the binary egg. In a dream world, a package index would have an API that allows an installer to query what system dependencies are relevant to binary eggs of a given package and for which combinations of values binary eggs actually exist. The installer could then figure out which of them fits its actual environment best or whether to get the source egg and roll its own binary. -- Thomas