PyPi’s predictable download url
Hello, I’m writing installation instructions for a package and have a question: what is the correct predictable URL for the package? I found couple. Which one of these guaranteed won’t change again in the future? - https://pypi.python.org/packages/source/v/virtualenv/virtualenv-15.1.0.tar.g... (used to work in the past, but doesn’t seem to work now, copied from virtualenv’s docs <https://virtualenv.pypa.io/en/latest/installation/>) - https://pypi.io/packages/source/v/virtualenv/virtualenv-15.1.0.tar.gz - https://pypi.org/packages/source/v/virtualenv/virtualenv-15.1.0.tar.gz - https://files.pythonhosted.org/packages/source/v/virtualenv/virtualenv-15.1.... Is there a predictable URL, which will return latest version of the package? Thank you. — Regards, *Aleks* *Bunin* -- *DISCLAIMER* This communication is intended only for the personal use of the recipient(s) named above and may contain information and/or attachments that are privileged, confidential, proprietary and protected from disclosure. If you are not the intended recipient of this message, you are hereby notified that any dissemination, distribution or copying of this message, and any attachments thereto, is strictly prohibited. If you receive this message in error, please promptly delete it without further distribution and reply to the sender that you have received the message in error. This communication or any attachment thereto is not intended, and should not be construed, as a investment advice or and does not constitute an offer or commitment, a solicitation of an offer, or any advice or recommendation, to enter into or conclude any transaction. This communication or any attachment thereto is not intended, and should not be construed, as legal, tax, public accounting or auditing advice or opinions. You should consult your legal counsel, accountants and/or tax advisors prior to making any decisions or taking any action concerning the matters in this communication. Attachments hereto may have additional important disclosures and disclaimers, which you should read.
On Tue, Jul 25, 2017 at 1:45 PM, Aleks Bunin <b@enlnt.com> wrote:
I’m writing installation instructions for a package and have a question: what is the correct predictable URL for the package?
I would guess you need to start from the API (e.g. https://wiki.python.org/moin/PyPIJSON) and go wherever it says. If writing your own automated thing, you could access the API yourself, or just delegate it to pip. For user-facing instructions, I would highly suggest having them do "pip install X" rather than have them download a tarball, extract it, find and run setup.py. If you want to pin it at a version, you can do something like: pip install "virtualenv==15.1.0" pip install "django>=1.11,<1.12" # for a range of versions Nick
On Tuesday, July 25, 2017, Nick Timkovich <prometheus235@gmail.com> wrote:
On Tue, Jul 25, 2017 at 1:45 PM, Aleks Bunin <b@enlnt.com <javascript:_e(%7B%7D,'cvml','b@enlnt.com');>> wrote:
I’m writing installation instructions for a package and have a question: what is the correct predictable URL for the package?
I would guess you need to start from the API (e.g. https://wiki.python.org/moin/PyPIJSON) and go wherever it says.
$ pip download pkgname><ver Determines the package URL. https://pip.pypa.io/en/stable/reference/pip_download/#examples https://github.com/pypa/pip/blob/master/pip/commands/download.py Would this solve?: - [ ] ENH: pip.commands.download: --only-print-url
If writing your own automated thing, you could access the API yourself, or just delegate it to pip. For user-facing instructions, I would highly suggest having them do "pip install X" rather than have them download a tarball, extract it, find and run setup.py. If you want to pin it at a version, you can do something like:
pip install "virtualenv==15.1.0" pip install "django>=1.11,<1.12" # for a range of versions
Nick
On Tue, Jul 25, 2017 at 3:33 PM, Nick Timkovich <prometheus235@gmail.com> wrote:
On Tue, Jul 25, 2017 at 1:45 PM, Aleks Bunin <b@enlnt.com> wrote: .. I would guess you need to start from the API (e.g. https://wiki.python.org/moin/PyPIJSON) and go wherever it says.
Unfortunately it currently says "http://pypi.python.org/pypi/<package_name>/json", which does not work. It looks like https is now required. With the http: prefix curl silently returns nothing.
On Tue, Jul 25, 2017 at 3:07 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
On Tue, Jul 25, 2017 at 3:33 PM, Nick Timkovich <prometheus235@gmail.com> wrote:
On Tue, Jul 25, 2017 at 1:45 PM, Aleks Bunin <b@enlnt.com> wrote: .. I would guess you need to start from the API (e.g. https://wiki.python.org/moin/PyPIJSON) and go wherever it says.
Unfortunately it currently says "http://pypi.python.org/pypi/<package_name>/json", which does not work. It looks like https is now required. With the http: prefix curl silently returns nothing.
That's because curl is kinda annoying and doesn't follow redirects by default: $ curl -i http://pypi.python.org/pypi/virtualenv/json HTTP/1.1 301 Moved Permanently ...
On Tue, Jul 25, 2017 at 4:18 PM, Nick Timkovich <prometheus235@gmail.com> wrote: ..
That's because curl is kinda annoying and doesn't follow redirects by default:
$ curl -i http://pypi.python.org/pypi/virtualenv/json HTTP/1.1 301 Moved Permanently ...
Well, http://pypi.org/.. which is presumably the home of the latest PyPI returns 403: $ curl -i http://pypi.org/pypi/virtualenv/json HTTP/1.1 403 SSL is required ... This suggests that redirects are considered to be legacy and may not be supported in the future.
On Tuesday, July 25, 2017, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
On Tue, Jul 25, 2017 at 4:18 PM, Nick Timkovich <prometheus235@gmail.com <javascript:;>> wrote: ..
That's because curl is kinda annoying and doesn't follow redirects by default:
$ curl -i http://pypi.python.org/pypi/virtualenv/json HTTP/1.1 301 Moved Permanently ...
Well, http://pypi.org/.. which is presumably the home of the latest PyPI returns 403:
$ curl -i http://pypi.org/pypi/virtualenv/json HTTP/1.1 403 SSL is required ...
This suggests that redirects are considered to be legacy and may not be supported in the future.
Here are the warehouse routes: https://github.com/pypa/warehouse/blob/master/warehouse/routes.py Why do you need an http to https redirect?
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org <javascript:;> https://mail.python.org/mailman/listinfo/distutils-sig
On Jul 25, 2017, at 2:15 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Tuesday, July 25, 2017, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote: On Tue, Jul 25, 2017 at 4:18 PM, Nick Timkovich <prometheus235@gmail.com> wrote: ..
That's because curl is kinda annoying and doesn't follow redirects by default:
$ curl -i http://pypi.python.org/pypi/virtualenv/json HTTP/1.1 301 Moved Permanently ...
Well, http://pypi.org/.. which is presumably the home of the latest PyPI returns 403:
$ curl -i http://pypi.org/pypi/virtualenv/json HTTP/1.1 403 SSL is required ...
This suggests that redirects are considered to be legacy and may not be supported in the future.
Here are the warehouse routes: https://github.com/pypa/warehouse/blob/master/warehouse/routes.py
Why do you need an http to https redirect?
To explain this: pypi.org is on the HSTS preload list so all major browsers will automatically use HTTPS for it no matter what. cURL does not support this feature. --Noah
On 07/25/2017 05:25 PM, Noah Kantrowitz wrote:
On Jul 25, 2017, at 2:15 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Tuesday, July 25, 2017, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote: On Tue, Jul 25, 2017 at 4:18 PM, Nick Timkovich <prometheus235@gmail.com> wrote: ..
That's because curl is kinda annoying and doesn't follow redirects by default:
$ curl -i http://pypi.python.org/pypi/virtualenv/json HTTP/1.1 301 Moved Permanently ...
Well, http://pypi.org/.. which is presumably the home of the latest PyPI returns 403:
$ curl -i http://pypi.org/pypi/virtualenv/json HTTP/1.1 403 SSL is required ...
This suggests that redirects are considered to be legacy and may not be supported in the future.
Here are the warehouse routes: https://github.com/pypa/warehouse/blob/master/warehouse/routes.py
Why do you need an http to https redirect?
To explain this: pypi.org is on the HSTS preload list so all major browsers will automatically use HTTPS for it no matter what. cURL does not support this feature.
Seems like having an unconditional HTTP->HTTPS redirect in place would be a "good neighbor" kind of thing (and belt-and-suspenders, as well). Tres. -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
On Jul 25, 2017, at 6:06 PM, Tres Seaver <tseaver@palladion.com> wrote:
On 07/25/2017 05:25 PM, Noah Kantrowitz wrote:
On Jul 25, 2017, at 2:15 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Tuesday, July 25, 2017, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote: On Tue, Jul 25, 2017 at 4:18 PM, Nick Timkovich <prometheus235@gmail.com> wrote: ..
That's because curl is kinda annoying and doesn't follow redirects by default:
$ curl -i http://pypi.python.org/pypi/virtualenv/json HTTP/1.1 301 Moved Permanently ...
Well, http://pypi.org/.. which is presumably the home of the latest PyPI returns 403:
$ curl -i http://pypi.org/pypi/virtualenv/json HTTP/1.1 403 SSL is required ...
This suggests that redirects are considered to be legacy and may not be supported in the future.
Here are the warehouse routes: https://github.com/pypa/warehouse/blob/master/warehouse/routes.py
Why do you need an http to https redirect?
To explain this: pypi.org is on the HSTS preload list so all major browsers will automatically use HTTPS for it no matter what. cURL does not support this feature.
Seems like having an unconditional HTTP->HTTPS redirect in place would be a "good neighbor" kind of thing (and belt-and-suspenders, as well).
Warehouse purposely only redirects “UI” pages from HTTP to HTTPS, API pages hard fail on HTTP. The rationale here is that UI pages are most likely going to be visited by browsers/people which may not support the HSTS preload list so we don’t want to display an error for those people (and if they support HSTS at all, future visits will be HTTPS). However for API views the most typical case is for someone to hardcode an URL in a client/script/configuration somewhere, and there the HTTP -> HTTPS redirect actually does them a disservice, because it silently allows them to be insecure (since a network attacker can intercept the HTTP request and just never redirect) and most automated tooling does not support HSTS (so future requests won’t be secure either). Thus this is a trade off and for browsers the trade off is to make it work, and for automated tooling the trade off is to make it correct. — Donald Stufft
On Jul 25, 2017, at 3:06 PM, Tres Seaver <tseaver@palladion.com> wrote:
On 07/25/2017 05:25 PM, Noah Kantrowitz wrote:
On Jul 25, 2017, at 2:15 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Tuesday, July 25, 2017, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote: On Tue, Jul 25, 2017 at 4:18 PM, Nick Timkovich <prometheus235@gmail.com> wrote: ..
That's because curl is kinda annoying and doesn't follow redirects by default:
$ curl -i http://pypi.python.org/pypi/virtualenv/json HTTP/1.1 301 Moved Permanently ...
Well, http://pypi.org/.. which is presumably the home of the latest PyPI returns 403:
$ curl -i http://pypi.org/pypi/virtualenv/json HTTP/1.1 403 SSL is required ...
This suggests that redirects are considered to be legacy and may not be supported in the future.
Here are the warehouse routes: https://github.com/pypa/warehouse/blob/master/warehouse/routes.py
Why do you need an http to https redirect?
To explain this: pypi.org is on the HSTS preload list so all major browsers will automatically use HTTPS for it no matter what. cURL does not support this feature.
Seems like having an unconditional HTTP->HTTPS redirect in place would be a "good neighbor" kind of thing (and belt-and-suspenders, as well).
Those redirects lead to a false sense of security. As pypi.org is new and we know there are no legacy links to it out there, it does not make sense to allow http://pypi.org as a thing. There is no such website as http://pypi.org. --Noah
On Tue, Jul 25, 2017 at 4:25 PM, Noah Kantrowitz <noah@coderanger.net> wrote:
On Tuesday, July 25, 2017, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote: $ curl -i http://pypi.org/pypi/virtualenv/json HTTP/1.1 403 SSL is required ...
To explain this: pypi.org is on the HSTS preload list so all major browsers will automatically use HTTPS for it no matter what. cURL does not support this feature.
--Noah
Also sounds like a good idea so anyone that writes a client and forgets that S will be forced to correct it rather than silently relying on a redirect (slower, opportunity for attack) Nick
On 26 July 2017 at 04:45, Aleks Bunin <b@enlnt.com> wrote:
Hello,
I’m writing installation instructions for a package and have a question: what is the correct predictable URL for the package? I found couple. Which one of these guaranteed won’t change again in the future?
https://pypi.python.org/packages/source/v/virtualenv/virtualenv-15.1.0.tar.g... (used to work in the past, but doesn’t seem to work now, copied from virtualenv’s docs) https://pypi.io/packages/source/v/virtualenv/virtualenv-15.1.0.tar.gz https://pypi.org/packages/source/v/virtualenv/virtualenv-15.1.0.tar.gz https://files.pythonhosted.org/packages/source/v/virtualenv/virtualenv-15.1....
Is there a predictable URL, which will return latest version of the package?
No, and it isn't particularly straightforward to go from a downloaded tarball to a correctly installed component either. Direct downloads will also bypass any pre-published wheel files. While https://www.python.org/dev/peps/pep-0503/ defines the current API for tools to retrieve this information from the service, the most robust option is to instead let a dedicated client tool like pip handle the problem: * pip install: retrieve *and* install a component * pip download: just retrieve the files without installing them locally Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jul 26, 2017 at 10:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
.. the most robust option is to instead let a dedicated client tool like pip handle the problem:
* pip install: retrieve *and* install a component * pip download: just retrieve the files without installing them locally
This is a sound advise in general, but the specific issue here is how to bootstrap virtualenv for python 2.7 on the systems with missing/broken pip? Note that "ensurepip" may not be an option for a user without root privileges. The bootstrap options offered at <https://virtualenv.pypa.io/en/stable/installation/> don't work. There is an open issue about this on Github: <https://github.com/pypa/virtualenv/issues/1042>. A comment on that issues suggests "curl https://pypi.python.org/simple/virtualenv-15.0.1.tar.gz", but that does not work either.
On 26 July 2017 at 15:52, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Wed, Jul 26, 2017 at 10:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
.. the most robust option is to instead let a dedicated client tool like pip handle the problem:
* pip install: retrieve *and* install a component * pip download: just retrieve the files without installing them locally
This is a sound advise in general, but the specific issue here is how to bootstrap virtualenv for python 2.7 on the systems with missing/broken pip? Note that "ensurepip" may not be an option for a user without root privileges.
The bootstrap options offered at <https://virtualenv.pypa.io/en/stable/installation/> don't work. There is an open issue about this on Github: <https://github.com/pypa/virtualenv/issues/1042>. A comment on that issues suggests "curl https://pypi.python.org/simple/virtualenv-15.0.1.tar.gz", but that does not work either.
I didn't get the impression that this was the problem the OP was hitting, and it's certainly not what I'd recommend for him, but you should be able to just clone the virtualenv repo and run ```python virtualenv.py``` to get started. Paul
On Wednesday, July 26, 2017, Paul Moore <p.f.moore@gmail.com> wrote:
On Wed, Jul 26, 2017 at 10:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
.. the most robust option is to instead let a dedicated client tool
On 26 July 2017 at 15:52, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote: like pip
handle the problem:
* pip install: retrieve *and* install a component * pip download: just retrieve the files without installing them locally
This is a sound advise in general, but the specific issue here is how to bootstrap virtualenv for python 2.7 on the systems with missing/broken pip? Note that "ensurepip" may not be an option for a user without root privileges.
The bootstrap options offered at <https://virtualenv.pypa.io/en/stable/installation/> don't work. There is an open issue about this on Github: <https://github.com/pypa/virtualenv/issues/1042>. A comment on that issues suggests "curl https://pypi.python.org/simple/virtualenv-15.0.1.tar.gz", but that does not work either.
I didn't get the impression that this was the problem the OP was hitting, and it's certainly not what I'd recommend for him, but you should be able to just clone the virtualenv repo and run ```python virtualenv.py``` to get started.
A shallow clone would minimize the bandwidth cost: $ git clone --depth=1 https://github.com/pypa/virtualenv If all you need to do is bootstrap a latest pip and virtualenv, there's always get-pip.py? - Docs: https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py - Src: https://bootstrap.pypa.io/get-pip.py $ python get-pip.py --user $ ~/.local/bin/pip install --user virtualenv virtualenvwrapper
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Wes Turner kirjoitti 27.07.2017 klo 02:36:
On Wednesday, July 26, 2017, Paul Moore <p.f.moore@gmail.com <mailto:p.f.moore@gmail.com>> wrote:
On 26 July 2017 at 15:52, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote: > On Wed, Jul 26, 2017 at 10:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: >> .. the most robust option is to instead let a dedicated client tool like pip >> handle the problem: >> >> * pip install: retrieve *and* install a component >> * pip download: just retrieve the files without installing them locally > > This is a sound advise in general, but the specific issue here is how > to bootstrap virtualenv for python 2.7 on the systems with > missing/broken pip? Note that "ensurepip" may not be an option for a > user without root privileges. > > The bootstrap options offered at > <https://virtualenv.pypa.io/en/stable/installation/ <https://virtualenv.pypa.io/en/stable/installation/>> don't work. > There is an open issue about this on Github: > <https://github.com/pypa/virtualenv/issues/1042 <https://github.com/pypa/virtualenv/issues/1042>>. A comment on that > issues suggests "curl > https://pypi.python.org/simple/virtualenv-15.0.1.tar.gz <https://pypi.python.org/simple/virtualenv-15.0.1.tar.gz>", but that > does not work either.
I didn't get the impression that this was the problem the OP was hitting, and it's certainly not what I'd recommend for him, but you should be able to just clone the virtualenv repo and run ```python virtualenv.py``` to get started.
A shallow clone would minimize the bandwidth cost:
$ git clone --depth=1 https://github.com/pypa/virtualenv
Doesn't work for projects using setuptools_scm, as they require the latest tag to get the version info.
If all you need to do is bootstrap a latest pip and virtualenv, there's always get-pip.py?
- Docs: https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py - Src: https://bootstrap.pypa.io/get-pip.py
$ python get-pip.py --user $ ~/.local/bin/pip install --user virtualenv virtualenvwrapper
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 27 July 2017 at 11:06, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
I didn't get the impression that this was the problem the OP was hitting, and it's certainly not what I'd recommend for him, but you should be able to just clone the virtualenv repo and run ```python virtualenv.py``` to get started.
A shallow clone would minimize the bandwidth cost:
$ git clone --depth=1 https://github.com/pypa/virtualenv
Doesn't work for projects using setuptools_scm, as they require the latest tag to get the version info.
None of this is a general solution. The general solution (for the OP) is: 1. Assume your users have a working pip. 2. pip install myproject The rest of the discussion is (as far as I can tell) how to avoid having to deal with step (1). And in particular, how to avoid having to deal with step (1) *if you're trying to install virtualenv*. None of the solutions really make much sense in practice, as it's pretty easy to get a working pip. Any recent version of Python comes with pip installed, if your distro doesn't then you should install pip with your system package manager, if you don't have admin privileges then https://pip.pypa.io/en/stable/installing/ explains how to use get-pip.py (you need --user if you don't have the privileges to modify the Python installation you're using - that's maybe not as obvious as it could be from the page, although it shouldn't be *too* hard to work out if system installs are failing). Without knowing why "pip install" isn't sufficient for the OP, other than the fact that the original question was asking for a "predictable URL" (which doesn't exist) rather than describing what the ultimate objective was, it's hard to see what the point of this discussion is. Paul
On 27 July 2017 at 00:52, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Wed, Jul 26, 2017 at 10:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
.. the most robust option is to instead let a dedicated client tool like pip handle the problem:
* pip install: retrieve *and* install a component * pip download: just retrieve the files without installing them locally
This is a sound advise in general, but the specific issue here is how to bootstrap virtualenv for python 2.7 on the systems with missing/broken pip? Note that "ensurepip" may not be an option for a user without root privileges.
python -m ensurepip --user should still work in that case (since it installs inside the user's home directory and leaves the system site-packages alone). As far as direct *file* downloads go, no, those URLs aren't readily calculated at all - automated clients are required to look them up via the index page for the package (e.g. https://pypi.org/simple/virtualenv/ for virtualenv). For virtualenv, the full URL for the 15.1.0 wheel file for example is https://files.pythonhosted.org/packages/6f/86/3dc328ee7b1a6419ebfac7896d882f... To get a *human* to the right set of files, though, you'll want to link them to https://pypi.org/project/virtualenv/#files Cheers, NIck. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (10)
-
Aleks Bunin -
Alex Grönholm -
Alexander Belopolsky -
Donald Stufft -
Nick Coghlan -
Nick Timkovich -
Noah Kantrowitz -
Paul Moore -
Tres Seaver -
Wes Turner