Mailman 3 August 2018 - Distutils-SIG

Editable requirement parsing in pip
by Tzu-ping Chung Aug. 31, 2018

Aug. 31, 2018

Hi, I was verifying my Pipenv resolver work, and found a quirk in pip when handling -e requirements. With a requirements.txt: # Works as expected colorama ; os_name == "nt" # Works as expected ./colorama ; os_name == "nt" # Drops markers; always installs disregarding the OS (!) -e ./colorama ; os_name == "nt" This by itself is okay, since it doesn’t really make sense to me to specify a requirement as editable without intending it to always work, although somehow surprising and (AFAICT) undocumented. The behavior in the command line is a little more preplexing: # Works as expected $ pip install "./colorama ; os_name == 'nt'" # Does not work $ pip install -e "./colorama ; os_name == 'nt'" ./colorama ; os_name == "nt" should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+ # This either $ pip install "-e ./colorama ; os_name == 'nt'" ./colorama ; os_name == "nt" should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+ Since pip’s documentation says a requirements.txt “is just a list of pip install arguments placed in a file” [1], this is surprising to me. [1]: https://pip.pypa.io/en/stable/user_guide/#id1 So my questions are * Is this behaviour intentional, or is it an oversight? * Is there specification or documentation on how pip is supposed to work? Given that PEP 508 (although pip does not actually use its URL lookup syntax) allows requirements to specify markers in all cases, I would expect pip to allow and honour them in all cases. Or, if that’s not the case (since editable is not really a standardised thing), I’d expect pip to at least be consistent with itself :) -- Tzu-ping Chung (@uranusjr) uranusjr(a)gmail.com https://uranusjr.com

1 0

Clarification on string arguments in PEP 517 hooks
by Paul Moore Aug. 28, 2018

Aug. 28, 2018

I'm in the process of implementing PEP 517 for pip, and I've hit a question. I'm pretty sure I know the answer, but I want to be clear, as whatever the answer is will require some fixing up. The various hooks take directory paths as arguments, and typically return a filename (e.g., build_wheel). The returned filename is always explicitly noted as being *a unicode string*. However, argumnents (metadata_directory in build_wheel/prepare_metadata_for_build_wheel) are *not* explicitly mandated to be Unicode. My assumption is that the intent is that *all* strings, whether arguments or return values, must be Unicode. The reason that I've hit this is that the current setuptools PEP 517 backend passes the metadata_directory direct to distutils which, on Python 2, checks for a (non-Unicode) string type and errors if Unicode is supplied. Fixing that is going to be annoyingly tricky, so I want to be sure that's the requirement before I start making fixes to setuptools. Paul

3 4

AofzaX.TH
by youtube.com/[whatItSaysHere] [whatItSaysHere] Aug. 25, 2018

Aug. 25, 2018

Amazing Thailand

1 0

(no subject)
by bunkeryou Aug. 24, 2018

Aug. 24, 2018

ส่งจากสมาร์ทโฟน Samsung Galaxy ของฉัน

1 0

Re: pipenv and pip
by Tzu-ping Chung Aug. 21, 2018

Aug. 21, 2018

Hi, Dan and I had been doing most of the maintenance work for Pipenv recently, and as Dan mentioned, we have been working on some related projects that poke into pip internals significantly, so I feel I should voice some opinions. I have significantly less experience messing with pip than Dan, and might be able to offer a slightly different perspective. Pipenv mainly interacts with pip for two things: install/uninstall/upgrade packages, and to gain information about a package (what versions are available, what dependencies does a particular version has, etc.). For the former case, we are currently using it with subprocesses, and it is likely the intended way of interaction. I have to say, however, that the experience is not flawless. pip has a significant startup time, and does not offer chances for interaction once it is started on running, so we really don’t have a good way to, for example, provide installation progress bar for the user, unless we parse pip’s stdout directly. These are not essential to Pipenv’s functionality, however, so they are more like an annoyance rather than glaring problems. The other thing Pipenv uses pip for—getting package information—is more troubling (to me, personally). Pipenv has a slightly different need from pip regarding dependency resolution. pip can (and does) freely drop dependencies that does not match the current environment, but Pipenv needs to generate a lock file for an abstract platform that works for, say, both macOS and Windows. This means pip’s resolver is not useful for us, and we need to implement our own. Our own resolver, however, still needs to know about packages it gets, and we are left with two choices: a. try re-implement the same logic, or b. use pip internals to cobble something together. We tried to go for a. for a while, but as you’d easily imagine, our own implementation is buggy, cannot handle edge cases nearly as well, and fielded a lot of complaints along the lines of “I can do this in pip, why can’t I do the same in Pipenv”. One example is how package artifacts are discovered. At my own first glance, I thought to myself this wouldn’t be that hard—we have a simple API, and the naming conventions are there, so as long as we specify sources in Pipfile (we do), we should be able to discover them no problem. I couldn’t be more wrong. There are find_links, dependency_links, pip.conf for the user, for the machine, all sorts of things, and for everything quirk in pip we don’t replicate 100%, issues are filed urging use to fix it. In the end we gave up and use pip’s internal PackageFinder instead. This is a big problem going forward, and we are fully aware of that. The strategy we are taking at the moment is to try to limit the surface area of pip internals usage. Dan mentioned we have been building a resolver for Pipenv[1], and we took the chance to work toward centralising things interfacing with pip internals. Those are still internals, of course, but we now have a relatively good idea what we actually need from pip, and I’d be extremely happy if some parts of pip can come out as standalone with official blessing. The things I am particularly interested in (since they would be beneficial for Pipenv) are: * VcsSupport * PackageFinder * WheelBuilder (and everything that comes with it like the wheel cache, preparer, unpack_url, etc.) Sorry for the very long post, but I want to get everything out so it might be easier to paint a complete picture of the state we are currently in. [1]: https://github.com/sarugaku/passa <https://github.com/sarugaku/passa> Yours, TP -- Tzu-ping Chung (@uranusjr) uranusjr(a)gmail.com https://uranusjr.com > On 21/8, 2018, at 00:00, distutils-sig-request(a)python.org wrote: > > Send Distutils-SIG mailing list submissions to > distutils-sig(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > or, via email, send a message with subject or body 'help' to > distutils-sig-request(a)python.org > > You can reach the person managing the list at > distutils-sig-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Distutils-SIG digest..."Today's Topics: > > 1. Re: pipenv and pip (Dan Ryan) > 2. Re: pipenv and pip (Dan Ryan) > > From: Dan Ryan <dan(a)danryan.co> > Subject: [Distutils] Re: pipenv and pip > Date: 20 August 2018 at 22:04:11 GMT+8 > To: Chris Jerdonek <chris.jerdonek(a)gmail.com> > Cc: distutils sig <distutils-sig(a)python.org> > > > The truth is that it’s basically impossible to gauge bugs in pip vs bugs in our patches to it which are often a lot more likely — reproductions of edge cases can be impossible but there are specific things I know we broke (like parsing certain kinds of extras, previously) — mostly bugs land in pips issue tracker before we report them or we will direct people there. We have like 2 active maintainers and we are maintaining like 15 pipenv related projects so we normally just point people at pip rather than file an issue. I am usually on irc as well if needed, and I often ask for clarification there > > Dan Ryan // pipenv maintainer > gh: @techalchemy > >> On Aug 20, 2018, at 4:32 AM, Chris Jerdonek <chris.jerdonek(a)gmail.com> wrote: >> >> Thanks. Is the state of affairs as you described them what you're >> planning for the future as well, or do you anticipate any changes >> worthy of note? >> >> Also, are any of the bugs filed in pipenv's tracker due to bugs or >> rough spots in pip -- is there a way to find those, like by using a >> label? It would be good to be able to know about those so pip can >> improve and become more useful. It doesn't seem like any bugs have >> been filed in pip's tracker in the past year by any of pipenv's top >> contributors. That seems a bit surprising to me given pipenv's heavy >> reliance on pip (together with the fact that I know pip has its share >> of issues), or is there another way you have of communicating >> regarding things that interconnect with pip? >> >> Thanks, >> --Chris >> >> >> >>> On Mon, Aug 20, 2018 at 12:51 AM, Dan Ryan <dan(a)danryan.co> wrote: >>> Sure I can grab that— we patch pip because we use some internals to handle resolution and we have some bugs around that currently. They aren’t upstreamed because they aren’t actually present in pip, only in pipenv. Pipenv crosses back and forth across the virtualenv boundary during the process. Pipenv relies on piptools and vendors a patched version of pip to ensure consistency as well as to provide a few hacks around querying the index. We do have a bit of reimplementation around some kinds of logic, with the largest overlap being in parsing of requirements. >>> >>> As we handle some resolution, which isn’t really something pip does, there is no cli interface to achieve this. I maintain a library (as of last week) which provides compatibility shims between pip versions 8-current. It is a good idea to use the cli, but we already spend enough resources forking subprocesses into the background that it is a lot more efficient to use the internals, which I track quite closely. The preference toward cli interaction is largely to allow internal api breakage which we don’t mind. >>> >>> For the most part, we have open channels of communication as necessary. We rely as heavily as we can on pip, packaging, and setuptools to connect the dots, retrieve package info, etc. >>> >>> Dan Ryan // pipenv maintainer >>> gh: @techalchemy >>> >>>> On Aug 20, 2018, at 2:41 AM, Chris Jerdonek <chris.jerdonek(a)gmail.com> wrote: >>>> >>>> Hi, >>>> >>>> Can someone explain to me the relationship between pipenv and pip, >>>> from the perspective of pipenv's maintainers? >>>> >>>> For example, does pipenv currently reimplement anything that pip tries >>>> to do, or does it simply call out to pip through the CLI or through >>>> its internal API's? Does it have any preferences or future plans in >>>> this regard? How about upstreaming to pip fixes or things that would >>>> help pipenv? >>>> >>>> I've been contributing to pip more lately, and I had a look at >>>> pipenv's repository for the first time today. >>>> https://github.com/pypa/pipenv >>>> >>>> Given that pip's code was recently made internal, I was a bit >>>> surprised to see that pipenv vendors and patches pip: >>>> https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip >>>> Before I had always assumed that pipenv used pip's CLI (because that's >>>> what pip says you should do). >>>> >>>> I also noticed that some bugs in pipenv's tracker seem closely related >>>> to pip's behavior, but I don't recall seeing any bugs or PR's in pip's >>>> tracker reported from pipenv maintainers. >>>> >>>> Without knowing a whole lot more than what I've stated, one concern I >>>> have is around fragmentation, duplication of work, and repeating >>>> mistakes (or introducing new ones) if a lot of work is going into >>>> pipenv without coordinating with pip. Is this in any way similar to >>>> the beginning of what happened with distutils, setuptools, and >>>> distribute that we are still recovering from? >>>> >>>> --Chris >>>> -- >>>> Distutils-SIG mailing list -- distutils-sig(a)python.org >>>> To unsubscribe send an email to distutils-sig-leave(a)python.org >>>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ >>>> Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… > > > > From: Dan Ryan <dan(a)danryan.co> > Subject: [Distutils] Re: pipenv and pip > Date: 20 August 2018 at 22:09:23 GMT+8 > To: Paul Moore <p.f.moore(a)gmail.com> > Cc: Distutils <distutils-sig(a)python.org> > > > How would I (said library) maintain compatibility? I’m pretty clever. The shim library doesn’t actually do anything besides provide import paths. If I shim something that didn’t exist, it shims None for any pip versions where it doesn’t exist. So for example if you are running pip 9 and you import RequirementTracker from the shims library you just import None > > Dan Ryan // pipenv maintainer > gh: @techalchemy > >> On Aug 20, 2018, at 8:15 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: >> >>> On Mon, 20 Aug 2018 at 12:25, Wes Turner <wes.turner(a)gmail.com> wrote: >>> >>> On Monday, August 20, 2018, Paul Moore <p.f.moore(a)gmail.com> wrote: >> >>>> I know "security by obscurity" doesn't work, but I'm happier if >>>> details of this library *aren't* widely known - it's not something I'd >>>> want to encourage people using, nor is it supported by pip, as it's >>>> basically a direct interface into pip's internal functions, papering >>>> over the name changes that we did in pip 10 specifically to dissuade >>>> people from doing this. >>> >>> >>> If someone was committing to identifying useful API methods, parameters, and return values; >>> writing a ~PEP; >>> implementing said API; >>> and maintaining backwards compatible shims for some reason; >>> would something like `pip.api` be an appropriate namespace? >>> (now that we're on version 18 with a faster release cycle)? >> >> I'm not quite sure I know what you mean here. The key point is that >> pip 18.0 might have an internal function pip._internal.xxx, and in pip >> 18.1 there's no such function, and the functionality doesn't even >> exist any more. How would a 3rd party project maintain backwards >> compatible shims in the face of that? Agreed it's not likely in >> practice - but we're not going to guarantee it. >> >> To be honest I don't see the point of discussing pip's internal API. >> It's just that - internal. I'd rather discuss useful (general) >> packaging libraries, that tools can build on - pip can vendor those >> and act as (just) another consumer, rather than getting into debates >> about support and internal APIs. >> >> Paul >> -- >> Distutils-SIG mailing list -- distutils-sig(a)python.org >> To unsubscribe send an email to distutils-sig-leave(a)python.org >> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ >> Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… > > > -- > Distutils-SIG mailing list -- distutils-sig(a)python.org > To unsubscribe send an email to distutils-sig-leave(a)python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/

6 9

Accessing extras during install command
by Joni Orponen Aug. 21, 2018

Aug. 21, 2018

Hello, I'd like to run different external programs during the install command which are provided by (potentially) installed extras, but I cannot find where the currently selected extras are stored. The only thing I quickly found by going through the distutils and setuptools codebases is an old TODO from easy_install indicating it might not have been implemented yet: https://github.com/pypa/setuptools/blob/afba2d89c4436451ad4d7e19228584246a9… I can hack this by try excepting on ImportError, but is there a more elegant (and intended) way? -- Joni Orponen

2 1

pipenv and pip
by Chris Jerdonek Aug. 20, 2018

Aug. 20, 2018

Hi, Can someone explain to me the relationship between pipenv and pip, from the perspective of pipenv's maintainers? For example, does pipenv currently reimplement anything that pip tries to do, or does it simply call out to pip through the CLI or through its internal API's? Does it have any preferences or future plans in this regard? How about upstreaming to pip fixes or things that would help pipenv? I've been contributing to pip more lately, and I had a look at pipenv's repository for the first time today. https://github.com/pypa/pipenv Given that pip's code was recently made internal, I was a bit surprised to see that pipenv vendors and patches pip: https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip Before I had always assumed that pipenv used pip's CLI (because that's what pip says you should do). I also noticed that some bugs in pipenv's tracker seem closely related to pip's behavior, but I don't recall seeing any bugs or PR's in pip's tracker reported from pipenv maintainers. Without knowing a whole lot more than what I've stated, one concern I have is around fragmentation, duplication of work, and repeating mistakes (or introducing new ones) if a lot of work is going into pipenv without coordinating with pip. Is this in any way similar to the beginning of what happened with distutils, setuptools, and distribute that we are still recovering from? --Chris

5 13

Re: Cannot upload distribution archives using twine using functioning username and login for pypi.org
by Chung Tzu-ping Aug. 16, 2018

Aug. 16, 2018

According to the documentation[1], TestPyPI is a complete independent instance from PyPI, and they don’t share databases at all. You need to register a separate account there. [1]: https://packaging.python.org/guides/using-testpypi/ -- Tzu-ping Chung (@uranusjr) uranusjr(a)gmail.com Sent from my iPhone > On 16 Aug 2018, at 23:55, tom(a)tombaker.org wrote: > > When I follow the instructions at > https://packaging.python.org/tutorials/packaging-projects/ > to upload my distribution archives with > > $ twine upload --repository-url https://test.pypi.org/legacy/ dist/*` > > I get prompted for my username and password, but the upload aborts with > `HTTPError: 403 Client Error` (see below) even though I can successfully log > into https://pypi.org using the same username and password. > > Can anyone advise? > > Tom > > ``` > Uploading distributions to https://test.pypi.org/legacy/ > Enter your username: tombaker > Enter your password: > Uploading mklists-0.1.1-py3-none-any.whl > 100%|*************************************************************************| 10.6k/10.6k [00:00<00:00, 15.8kB/s] > HTTPError: 403 Client Error: Invalid or non-existent authentication information. for url: https://test.pypi.org/legacy/ > ```

1 0

Cannot upload distribution archives using twine using functioning username and login for pypi.org
by tom＠tombaker.org Aug. 16, 2018

Aug. 16, 2018

When I follow the instructions at https://packaging.python.org/tutorials/packaging-projects/ to upload my distribution archives with $ twine upload --repository-url https://test.pypi.org/legacy/ dist/*` I get prompted for my username and password, but the upload aborts with `HTTPError: 403 Client Error` (see below) even though I can successfully log into https://pypi.org using the same username and password. Can anyone advise? Tom ``` Uploading distributions to https://test.pypi.org/legacy/ Enter your username: tombaker Enter your password: Uploading mklists-0.1.1-py3-none-any.whl 100%|*************************************************************************| 10.6k/10.6k [00:00<00:00, 15.8kB/s] HTTPError: 403 Client Error: Invalid or non-existent authentication information. for url: https://test.pypi.org/legacy/ ```

3 2

Is ensurepip still a thing?
by Chris Barker Aug. 12, 2018

Aug. 12, 2018

I'm updating some instructions for my students, in which the first thing I do is have them run ensurepip: $ python3 -m ensurepip --upgrade which resulted in: $ python3 -m ensurepip --upgrade Looking in links: /var/folders/ym/tj87fc850yd6526nbrn14rxm0000gn/T/tmpwc8nd6oj Requirement already up-to-date: setuptools in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (39.0.1) Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (10.0.1) (this is after a brand-new python 3.7 install on OS-X from python.org) All good. But then I use pip, and get (after a successful install): You are using pip version 10.0.1, however version 18.0 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Huh? shouldn't ensurepip have updated it for me already?? Or should I simply suggest the pip install --upgrade pip command and not bother with ensurepip anymore? BTW -- shouldn't that be: python3 -m pip install --upgrade pip to make sure they get the "right" pip? KInda hard to keep up with the latest ... Thanks, -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

6 8