On vendoring binaries and headers with pypi packages
Hi,
When distributing python packages that depend on non-python components, the typical way of doing this with general-purpose package managers (linux distributions, conda) is to place binaries, headers, configuration and other artefacts for these components in their respective natural locations under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/, PREFIX/etc/, PREFIX/share/).
Now, if I have a python package that depends on such a thing (e.g. pyzmq vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as package data.
For the case of the header files, there is the distutils `install_headers` directive, but the target directory is (typically) under PREFIX/include/PythonX.Y, which is probably only a good choices for headers of a C extension linked with this specific python installation.
Would it make sense, when vendoring packages packages that don't depend on the python version in a wheel to use `data_files` instead, and target respectively the `bin`, `include`, `etc` and `share` subdirectories of the prefix?
Specifically, for the headers, one could do
data_files = [ ('include/foobar', [list of the foobar header files])), ],
Note: I was just using zmq as an example.
Thanks,
Sylvain
The goal in this approach would be to place headers, and binaries in the same location as if they were installed with a separate package.
On Mon, May 15, 2017 at 7:29 PM, Sylvain Corlay sylvain.corlay@gmail.com wrote:
Hi,
When distributing python packages that depend on non-python components, the typical way of doing this with general-purpose package managers (linux distributions, conda) is to place binaries, headers, configuration and other artefacts for these components in their respective natural locations under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/, PREFIX/etc/, PREFIX/share/).
Now, if I have a python package that depends on such a thing (e.g. pyzmq vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as package data.
For the case of the header files, there is the distutils `install_headers` directive, but the target directory is (typically) under PREFIX/include/PythonX.Y, which is probably only a good choices for headers of a C extension linked with this specific python installation.
Would it make sense, when vendoring packages packages that don't depend on the python version in a wheel to use `data_files` instead, and target respectively the `bin`, `include`, `etc` and `share` subdirectories of the prefix?
Specifically, for the headers, one could do
data_files = [ ('include/foobar', [list of the foobar header files])), ],Note: I was just using zmq as an example.
Thanks,
Sylvain
What are you trying to accomplish by putting include files into virtualenvs?
On May 15, 2017 10:29 AM, "Sylvain Corlay" sylvain.corlay@gmail.com wrote:
Hi,
When distributing python packages that depend on non-python components, the typical way of doing this with general-purpose package managers (linux distributions, conda) is to place binaries, headers, configuration and other artefacts for these components in their respective natural locations under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/, PREFIX/etc/, PREFIX/share/).
Now, if I have a python package that depends on such a thing (e.g. pyzmq vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as package data.
For the case of the header files, there is the distutils `install_headers` directive, but the target directory is (typically) under PREFIX/include/PythonX.Y, which is probably only a good choices for headers of a C extension linked with this specific python installation.
Would it make sense, when vendoring packages packages that don't depend on the python version in a wheel to use `data_files` instead, and target respectively the `bin`, `include`, `etc` and `share` subdirectories of the prefix?
Specifically, for the headers, one could do
data_files = [ ('include/foobar', [list of the foobar header files])), ],Note: I was just using zmq as an example.
Thanks,
Sylvain
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Let's say that you have an anaconda installation, and you install the latest pyzmq from pypi instead of from the conda recipe.
I would like the vendored libzmq to be placed under `PREFIX/bin` and the headers under `PREFIX/include`, just as if I had either
- installed zmq with `cmake -D CMAKE_INSTALL_PREFIX=PREFIX` and `make install` - or installed the zeromq from the conda package.
Similarly, for a system-wide install on linux distribution, the same would hold with PREFIX=/usr
On Mon, May 15, 2017 at 9:47 PM, Nathaniel Smith njs@pobox.com wrote:
What are you trying to accomplish by putting include files into virtualenvs?
On May 15, 2017 10:29 AM, "Sylvain Corlay" sylvain.corlay@gmail.com wrote:
Hi,
When distributing python packages that depend on non-python components, the typical way of doing this with general-purpose package managers (linux distributions, conda) is to place binaries, headers, configuration and other artefacts for these components in their respective natural locations under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/, PREFIX/etc/, PREFIX/share/).
Now, if I have a python package that depends on such a thing (e.g. pyzmq vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as package data.
For the case of the header files, there is the distutils `install_headers` directive, but the target directory is (typically) under PREFIX/include/PythonX.Y, which is probably only a good choices for headers of a C extension linked with this specific python installation.
Would it make sense, when vendoring packages packages that don't depend on the python version in a wheel to use `data_files` instead, and target respectively the `bin`, `include`, `etc` and `share` subdirectories of the prefix?
Specifically, for the headers, one could do
data_files = [ ('include/foobar', [list of the foobar header files])), ],Note: I was just using zmq as an example.
Thanks,
Sylvain
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
No, I'm trying to ask, like, why you want that. Presumably you have some end goal in mind that you think this will help accomplish, and I think it would be helpful to know what that is :-).
On May 15, 2017 2:42 PM, "Sylvain Corlay" sylvain.corlay@gmail.com wrote:
Let's say that you have an anaconda installation, and you install the latest pyzmq from pypi instead of from the conda recipe.
I would like the vendored libzmq to be placed under `PREFIX/bin` and the headers under `PREFIX/include`, just as if I had either
- installed zmq with `cmake -D CMAKE_INSTALL_PREFIX=PREFIX` and `make
install`
- or installed the zeromq from the conda package.
Similarly, for a system-wide install on linux distribution, the same would hold with PREFIX=/usr
On Mon, May 15, 2017 at 9:47 PM, Nathaniel Smith njs@pobox.com wrote:
What are you trying to accomplish by putting include files into virtualenvs?
On May 15, 2017 10:29 AM, "Sylvain Corlay" sylvain.corlay@gmail.com wrote:
Hi,
When distributing python packages that depend on non-python components, the typical way of doing this with general-purpose package managers (linux distributions, conda) is to place binaries, headers, configuration and other artefacts for these components in their respective natural locations under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/, PREFIX/etc/, PREFIX/share/).
Now, if I have a python package that depends on such a thing (e.g. pyzmq vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as package data.
For the case of the header files, there is the distutils `install_headers` directive, but the target directory is (typically) under PREFIX/include/PythonX.Y, which is probably only a good choices for headers of a C extension linked with this specific python installation.
Would it make sense, when vendoring packages packages that don't depend on the python version in a wheel to use `data_files` instead, and target respectively the `bin`, `include`, `etc` and `share` subdirectories of the prefix?
Specifically, for the headers, one could do
data_files = [ ('include/foobar', [list of the foobar header files])), ],Note: I was just using zmq as an example.
Thanks,
Sylvain
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
OK thanks!
I want to create a pypi package for xtensor and xtensor-python, and would like it to place the headers in the same location as with the conda package or Debian packages.
(Which are essentially built with a regular `cmake -D CMAKE_INSTALL_PREFIX=PREFIX; make install`)
The conda package puts things under `CONDA_PREFIX/include/xtensor[-python]` and the Debian package `/usr/include/xtensor[-python]`.
It seems that wheel's data_files can achieve that very simply, and was wondering if there was any drawback in this approach.
Sylvain
On Mon, May 15, 2017 at 11:45 PM, Nathaniel Smith njs@pobox.com wrote:
No, I'm trying to ask, like, why you want that. Presumably you have some end goal in mind that you think this will help accomplish, and I think it would be helpful to know what that is :-).
On May 15, 2017 2:42 PM, "Sylvain Corlay" sylvain.corlay@gmail.com wrote:
Let's say that you have an anaconda installation, and you install the latest pyzmq from pypi instead of from the conda recipe.
I would like the vendored libzmq to be placed under `PREFIX/bin` and the headers under `PREFIX/include`, just as if I had either
- installed zmq with `cmake -D CMAKE_INSTALL_PREFIX=PREFIX` and `make
install`
- or installed the zeromq from the conda package.
Similarly, for a system-wide install on linux distribution, the same would hold with PREFIX=/usr
On Mon, May 15, 2017 at 9:47 PM, Nathaniel Smith njs@pobox.com wrote:
What are you trying to accomplish by putting include files into virtualenvs?
On May 15, 2017 10:29 AM, "Sylvain Corlay" sylvain.corlay@gmail.com wrote:
Hi,
When distributing python packages that depend on non-python components, the typical way of doing this with general-purpose package managers (linux distributions, conda) is to place binaries, headers, configuration and other artefacts for these components in their respective natural locations under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/, PREFIX/etc/, PREFIX/share/).
Now, if I have a python package that depends on such a thing (e.g. pyzmq vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as package data.
For the case of the header files, there is the distutils `install_headers` directive, but the target directory is (typically) under PREFIX/include/PythonX.Y, which is probably only a good choices for headers of a C extension linked with this specific python installation.
Would it make sense, when vendoring packages packages that don't depend on the python version in a wheel to use `data_files` instead, and target respectively the `bin`, `include`, `etc` and `share` subdirectories of the prefix?
Specifically, for the headers, one could do
data_files = [ ('include/foobar', [list of the foobar header files])), ],Note: I was just using zmq as an example.
Thanks,
Sylvain
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 15 May 2017 at 22:42, Sylvain Corlay sylvain.corlay@gmail.com wrote:
Let's say that you have an anaconda installation, and you install the latest pyzmq from pypi instead of from the conda recipe.
One of the key features of conda over pip is that it manages not just Python packages, but also non-Python dependencies. So you seem to be asking "if I choose to use the approach that doesn't handle non-Python dependencies, how can I make it handle non-Python dependencies in the same way as the solution which does, but which I chose not to use"...
So I'd have to ask, why aren't you able to use the conda build? Making pip handle non-Python dependencies is a major piece of work that's been discussed on various occasions on this list, but typically the debate has boiled down to "given that pip has limited developer resource, why should we spend it on implementing something that isn't core to pip and is already handled well by tools like conda?"
Paul.
So I actually do use conda (and also maintain dozens of conda-forge recipes). but I also need to package things on pypi and would like it to be as consistent as possible with what general purpose package managers do.
I started a proposal to add additional install paths to wheel. Maybe something like this would be useful for your application, would you be interested in finishing the proposal? If I was still working on it then I would decouple it from wheel, come up with a good API for finding the installed files afterwards, and possibly come up with a way to define custom path prefixes using string templates. https://www.python.org/dev/peps/pep-0491/#install-paths
Several people broke data_files by accident, long story, setup.py once was able to reliably install things anywhere but between virtualenv and wheel etc. this feature has been broken for many years, in other words you can try to install things with data_files but they will be installed in unpredictable locations depending on the installer. Unfortunately the current less flexible data_files is sometimes regarded as a feature.
On Mon, May 15, 2017 at 6:49 PM Sylvain Corlay sylvain.corlay@gmail.com wrote:
So I actually do use conda (and also maintain dozens of conda-forge recipes). but I also need to package things on pypi and would like it to be as consistent as possible with what general purpose package managers do. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
participants (4)
-
Daniel Holth -
Nathaniel Smith -
Paul Moore -
Sylvain Corlay