deprecating pip install --target
This is fairly broken - it doesn't handle platlib vs purelib (see pip PR 3450), doesn't handle data files, or any other layout. Donald says pip uses it to maintain the _vendor subtrees only, which doesn't seem like a particularly strong use case. Certainly the help description for it is misleading - since what we're actually doing is copying only a subset of what the package installed - so at a minimum we need to be much clearer about what it does. But, I think it would be better to deprecate it and remove it... so I'm pinging here to see if anyone can explain a sensible use case for it in the context of pip :) -Rob -- Robert Collins <rbtcollins@hpe.com> Distinguished Technologist HP Converged Cloud
On 11 February 2016 at 19:07, Robert Collins <robertc@robertcollins.net> wrote:
But, I think it would be better to deprecate it and remove it... so I'm pinging here to see if anyone can explain a sensible use case for it in the context of pip :)
I have used it in the past to bundle libraries in an application I'm building using zipapp. So I'd write my __main__.py, create a "libs" subdirectory, and "pip install --target libs" any dependencies I have. Because it's a zipped application, I'm only including pure-Python libraries so as long as they don't have data files they are fine. That's basically the same use case as vendoring. It's not something I do a lot, but I'd like there to be at a minimum, a documented replacement approach (that's roughly as convenient - "install your dependencies somewhere and copy them" or "download a wheel and unzip it manually" aren't really what I have in mind). I do think that creating zipped applications *should* be a reasonable use case - I think it's a woefully under-used deployment option, and I'd hate to see more obstacles put in the way of people using it. Alternatively, it should be possible to *detect* the problem cases, so why not do that, and reject them? Effectively, reduce the scope of --target to pure-python wheels only. Paul
Robert Collins <robertc <at> robertcollins.net> writes:
This is fairly broken - it doesn't handle platlib vs purelib (see pip PR 3450), doesn't handle data files, or any other layout. Donald says pip uses it to maintain the _vendor subtrees only, which doesn't seem like a particularly strong use case.
Certainly the help description for it is misleading - since what we're actually doing is copying only a subset of what the package installed - so at a minimum we need to be much clearer about what it does.
But, I think it would be better to deprecate it and remove it... so I'm pinging here to see if anyone can explain a sensible use case for it in the context of pip :)
I use it in pretty much the same way as Paul mentioned - I wouldn't like it to go unless something equivalent is available. Updating the help / documentation for it to better reflect what it does would be uncontroversial for me, but I see no strong reason for deprecation and removal. As Paul suggests, it can get stricter about what it'll handle. Regards, Vinay Sajip
I was also planning to use it in an upcoming project that has to "do its own" package management. The aim was to install different versions of packages in different directories and use sys.path modifications to resolve them at runtime (kind of like what setuptools did in the older days). An alternative would be great, though I can probably fake things somehow for my purposes. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Vinay Sajip" <vinay_sajip@yahoo.co.uk> Sent: 2/11/2016 15:18 To: "Distutils-Sig@Python.Org" <Distutils-Sig@Python.Org> Subject: Re: [Distutils] deprecating pip install --target Robert Collins <robertc <at> robertcollins.net> writes:
This is fairly broken - it doesn't handle platlib vs purelib (see pip PR 3450), doesn't handle data files, or any other layout. Donald says pip uses it to maintain the _vendor subtrees only, which doesn't seem like a particularly strong use case.
Certainly the help description for it is misleading - since what we're actually doing is copying only a subset of what the package installed - so at a minimum we need to be much clearer about what it does.
But, I think it would be better to deprecate it and remove it... so I'm pinging here to see if anyone can explain a sensible use case for it in the context of pip :)
I use it in pretty much the same way as Paul mentioned - I wouldn't like it to go unless something equivalent is available. Updating the help / documentation for it to better reflect what it does would be uncontroversial for me, but I see no strong reason for deprecation and removal. As Paul suggests, it can get stricter about what it'll handle. Regards, Vinay Sajip _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
My setup-requires wrapper, adding pre-setup.py dependency installation to setup.py, relies on this feature. It needs to install something in a directory that is only added to PYTHONPATH during the installation and does not interfere with the normal environment. On Fri, Feb 12, 2016 at 9:55 AM Steve Dower <python@stevedower.id.au> wrote:
I was also planning to use it in an upcoming project that has to "do its own" package management. The aim was to install different versions of packages in different directories and use sys.path modifications to resolve them at runtime (kind of like what setuptools did in the older days).
An alternative would be great, though I can probably fake things somehow for my purposes.
Cheers, Steve
Top-posted from my Windows Phone ------------------------------ From: Vinay Sajip <vinay_sajip@yahoo.co.uk> Sent: 2/11/2016 15:18 To: Distutils-Sig@Python.Org Subject: Re: [Distutils] deprecating pip install --target
Robert Collins <robertc <at> robertcollins.net> writes:
This is fairly broken - it doesn't handle platlib vs purelib (see pip PR 3450), doesn't handle data files, or any other layout. Donald says pip uses it to maintain the _vendor subtrees only, which doesn't seem like a particularly strong use case.
Certainly the help description for it is misleading - since what we're actually doing is copying only a subset of what the package installed - so at a minimum we need to be much clearer about what it does.
But, I think it would be better to deprecate it and remove it... so I'm pinging here to see if anyone can explain a sensible use case for it in the context of pip :)
I use it in pretty much the same way as Paul mentioned - I wouldn't like it to go unless something equivalent is available. Updating the help / documentation for it to better reflect what it does would be uncontroversial for me, but I see no strong reason for deprecation and removal. As Paul suggests, it can get stricter about what it'll handle.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 13 February 2016 at 04:12, Daniel Holth <dholth@gmail.com> wrote:
My setup-requires wrapper, adding pre-setup.py dependency installation to setup.py, relies on this feature. It needs to install something in a directory that is only added to PYTHONPATH during the installation and does not interfere with the normal environment.
Python packages that create (and need) scripts, or use data_files, won't work with --target. Using a --prefix and setting PYTHONPATH *and* PATH appropriately would be better. -Rob -- Robert Collins <rbtcollins@hpe.com> Distinguished Technologist HP Converged Cloud
A new, important, use of --target is in creating installation bundles for AWS Lambda functions using a Python runtime. AWS expects a zip file containing the primary source file and all dependent packages. 'pip install -t' creates exactly the structure needed. Further, because a 'pip install -t .' has the same behavior as a 'npm install' we can easily use the same packaging code for Python and Node Lambda function bundles. Thanks. -Mark This e-mail message and all attachments transmitted with it may contain privileged and/or confidential information intended solely for the use of the addressee(s). If the reader of this message is not the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, forwarding or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete this message, all attachments and all copies and backups thereof.
On 13 February 2016 at 03:54, Steve Dower <python@stevedower.id.au> wrote:
I was also planning to use it in an upcoming project that has to "do its own" package management. The aim was to install different versions of packages in different directories and use sys.path modifications to resolve them at runtime (kind of like what setuptools did in the older days).
An alternative would be great, though I can probably fake things somehow for my purposes.
Sounds similar to Daniel's need - and again, --prefix + setting PATH and PYTHONPATH would be better. -Rob -- Robert Collins <rbtcollins@hpe.com> Distinguished Technologist HP Converged Cloud
On 16 February 2016 at 22:52, Robert Collins <robertc@robertcollins.net> wrote:
An alternative would be great, though I can probably fake things somehow for my purposes.
Sounds similar to Daniel's need - and again, --prefix + setting PATH and PYTHONPATH would be better.
Note that if I read the help for --prefix correctly, "pip install --target x foo" puts foo in x, whereas "pip install --prefix x foo" puts foo in x/lib. So how would setting prefix allow me to put foo in x, and not in a subdirectory? That is specifically my requirement (and the vendoring requirement in general). I *know* that means there's no obvious place to put data files or extensions or whatever, and that's fine by me. It seems that if we want to go down this route, we need to include the full set of --install-purelib, --install-platlib, --install-scripts etc arguments to pip. But that's probably the wrong solution - if we want to start playing with the various install location parameters to pip install (--target, --prefix, --root) we should probably do a "proper" job and just find a way to allow user-defined schemes. Paul
Another alternative is making "pip the library" as has been discussed in the past. Certainly my needs would be satisfied by a library that can get me as far as wheel files given package name/s and version spec (and index URL I guess). Unpacking wheels isn't especially difficult and in my case I know there are no existing installs to worry about. Top-posted from my Windows Phone -----Original Message----- From: "Paul Moore" <p.f.moore@gmail.com> Sent: 2/17/2016 2:10 To: "Robert Collins" <robertc@robertcollins.net> Cc: "Steve Dower" <python@stevedower.id.au>; "Python Distutils SIG" <Distutils-Sig@python.org> Subject: Re: [Distutils] deprecating pip install --target On 16 February 2016 at 22:52, Robert Collins <robertc@robertcollins.net> wrote:
An alternative would be great, though I can probably fake things somehow for my purposes.
Sounds similar to Daniel's need - and again, --prefix + setting PATH and PYTHONPATH would be better.
Note that if I read the help for --prefix correctly, "pip install --target x foo" puts foo in x, whereas "pip install --prefix x foo" puts foo in x/lib. So how would setting prefix allow me to put foo in x, and not in a subdirectory? That is specifically my requirement (and the vendoring requirement in general). I *know* that means there's no obvious place to put data files or extensions or whatever, and that's fine by me. It seems that if we want to go down this route, we need to include the full set of --install-purelib, --install-platlib, --install-scripts etc arguments to pip. But that's probably the wrong solution - if we want to start playing with the various install location parameters to pip install (--target, --prefix, --root) we should probably do a "proper" job and just find a way to allow user-defined schemes. Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 17.02.2016 11:09, Paul Moore wrote:
On 16 February 2016 at 22:52, Robert Collins <robertc@robertcollins.net> wrote:
An alternative would be great, though I can probably fake things somehow for my purposes.
Sounds similar to Daniel's need - and again, --prefix + setting PATH and PYTHONPATH would be better.
Note that if I read the help for --prefix correctly, "pip install --target x foo" puts foo in x, whereas "pip install --prefix x foo" puts foo in x/lib. So how would setting prefix allow me to put foo in x, and not in a subdirectory? That is specifically my requirement (and the vendoring requirement in general).
I just discovered a way that lets you do just that, but before using it for my own code I'd like to know whether you would consider it a weird hack that will probably not work in the future or something reasonable that could be used in production code? Here's the trick: write a temporary distutils setup.cfg file in the current working directory with the content: [install] install-lib=abspath/to/target_dir then run pip from that directory like so: pip install packagexy --prefix abspath/to/target_dir Of note, combining a local setup.cfg file and --prefix like this isn't documented and to me it wasn't even clear whether the file would be expected in the current working directory that pip gets run in or in the downloaded package. What is more, the local setup.cfg file can also be used to specify a complete installation scheme for use without the --prefix option. As I said I'm really interested in your opinions, Wolfgang
On 29.02.2016 17:00, Wolfgang Maier wrote:
I just discovered a way that lets you do just that, but before using it for my own code I'd like to know whether you would consider it a weird hack that will probably not work in the future or something reasonable that could be used in production code? Here's the trick:
write a temporary distutils setup.cfg file in the current working directory with the content:
[install] install-lib=abspath/to/target_dir
then run pip from that directory like so:
pip install packagexy --prefix abspath/to/target_dir
Of note, combining a local setup.cfg file and --prefix like this isn't documented and to me it wasn't even clear whether the file would be expected in the current working directory that pip gets run in or in the downloaded package.
Answering my own question: its a messy hack that will not work from a virtualenv because distutils will ignore the local cfg file then.
participants (8)
-
Daniel Holth -
Mark Niedzielski -
Paul Moore -
Robert Collins -
Steve Dower -
Steve Dower -
Vinay Sajip -
Wolfgang Maier