Mailman 3 August 2017 - Distutils-SIG

Re: [Distutils] PEP 517 again
by xoviat Aug. 29, 2017

Aug. 29, 2017

Oh, BDFL, will you settle whether we should raise an exception or return Notimplemented? On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ewing(a)canterbury.ac.nz> wrote: xoviat wrote: > I mean how is opening a file different than attempting to build an sdist? > 1. Opening a file is a very common operation. 2. Most file opens are expected to succeed, and if one doesn't, the appropriate place to deal with that is almost never at the site of the open call, but somewhere further up. In contrast, there's probably only about one place in any given frontend where the backend's build_sdist method gets invoked, and it's unlikely the ability to let a not-implemented exception bubble up from that point would be of great use. -- Greg _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org https://mail.python.org/mailman/listinfo/distutils-sig

12 57

PEP 517 - a plan of action
by Paul Moore Aug. 29, 2017

Aug. 29, 2017

I feel that the discussion on PEP 517 is nearing an end. The outstanding questions are pretty detail-level, and there's little progress being made on them because the situations involved are mostly theoretical in the absence of actual implementations. So I propose the following plan of action: 1. Accept PEP 517 (at least provisionally). 2. Thomas to complete and publish his library of helper functions for PEP 517 consumers (optional - but I think he said he had such a thing in progress). 3. Build PEP 517 interfaces for flit and setuptools. 4. Implement PEP 517 support in pip. 5. Review what we learned from the above. 6. If needed, revise the PEP in view of actual implementation experience, and finalise it. Further frontends and backends can wait for the results of this process if they are concerned about tracking a provisional PEP. Hopefully, it shouldn't be too long a process (I'd like to see PEP 517 in the next release of pip, if possible). Notes ----- 1. On any remaining questions for the PEP, let's just go for the simplest solution for now. Let the PEP authors decide what that is, with Nick as BDFL arbitrating if there's disagreement. 2. In my view, the PEP 517 support in pip should fully embrace the principle in the PEP that it's the backend's job to ensure that sdist->wheel and build_wheel produce equivalent results (when both are possible), and should therefore *only* use build_wheel, at least until experience demonstrates that we have to protect users against badly written backends. 3. My suspicion is that the setuptools backend will pretty clearly demonstrate whether it's hard or easy to conform to the requirements in the PEP, which can inform pip's stance on a "pure build_wheel" approach. 4. If pip goes for direct wheel builds, this does mean that the build_sdist and get_requires_for_build_sdist hooks won't be exercised in any actual code during this process. IMO, that's OK, as those hooks cover important (if currently theoretical) cases for validating the "publish my project" workflow, and tools that want to address those cases (such as tox) are likely to need the sdist hooks later. If this seems like a reasonable plan, I'd suggest that people add links to issues or PRs for PEP 517 implementations in flit, setuptools and pip to this thread - as I'm sure all the participants here will want to review the concrete implementations. Please understand - I'm not trying to steamroller the process here, or force particular decisions. It's just that I know I'm burned out on the discussions, to the extent that I'm skimming mails and largely ignoring many of the hypothetical arguments, and I'm sure others are too. So we need to move to a point where we can discuss actual, in-development code rather than possibilities, and I believe the above plan gets us to that point as fast as possible. Paul

4 6

PBR/distutils2 with pep517 Support Was: Conditionless setup.py
by Thomas Güttler Aug. 28, 2017

Aug. 28, 2017

Am 25.08.2017 um 15:00 schrieb Paul Moore: > > One thought - are the PBR and/or distutils2 teams looking at providing > PEP 517 support? Assuming they are, have they had a change to review > the PEP to ensure it suits their needs? And if they aren't, what is it > about the PEP that makes them unwilling to do so? Hi Paul, I think this is a very good question! I started a new thread to make this more visible. Are there PBR/distutils2 hackers on this list here? If yes: Do you support this pep? If no: where can you meet PBR/distutils2 hackers? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/

4 4

Re: [Distutils] PEP 517 again
by xoviat Aug. 26, 2017

Aug. 26, 2017

Should probably take Guido out of this email chain now. And does everyone agree with what Nathaniel said on sys.path? On Aug 26, 2017 11:00 AM, "Daniel Holth" <dholth(a)gmail.com> wrote: I'm bored with that On Sat, Aug 26, 2017, 11:59 xoviat <xoviat(a)gmail.com> wrote: > Is everyone on board with that? > > On Aug 26, 2017 4:29 AM, "Paul Moore" <p.f.moore(a)gmail.com> wrote: > >> On 26 August 2017 at 03:17, Guido van Rossum <guido(a)python.org> wrote: >> > In pretty much any other context, if you have an operation that returns >> an >> > regular value or an error value, the error value should be None. >> (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> Paul >> > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG(a)python.org > https://mail.python.org/mailman/listinfo/distutils-sig >

1 0

Conditionless setup.py
by Thomas Güttler Aug. 26, 2017

Aug. 26, 2017

The setup.py of numpy looks big and complicated: https://github.com/numpy/numpy/blob/master/setup.py I guess there are even more complicated setup.py files. Do you think it will be possible in the future to make setup.py conditionless (no "if", no "else", no function calls)? I mean that the setuptools/distutils/whatever libraries will handle all needed conditions? This would mean setup.py would be a pure data structure which could be encoded in json or yaml. What do you think? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/

9 13

PEP 517 again
by Thomas Kluyver Aug. 26, 2017

Aug. 26, 2017

Nathaniel seems to be busy with other things at the moment, so I hope he won't mind me passing on this list of things he'd like to resolve with the draft PEP. I'll quote his comments and put my responses inline. > - needs some sort of NotImplemented(Error) handling specified We've gone round return-value vs exception a few times, but I don't think that debate is particularly heated, so it probably just needs someone to say "this is what we're doing". I can be that someone if needs be. But does anyone feel strongly about it? > - The out-of-tree build example (that makes an sdist and then builds > it) seems like exactly the kind of implementation that Donald says he > doesn't want to see? OTOH I think Nick said he wants to see a more > elaborated specification of out-of-tree build strategies with this > specifically as one of them. > - Nick has suggested that the to-be-defined NotImplemented(Error) > handling can be used by build_wheel to indicate that it can't do > out-of-tree builds, so maybe the frontend should try an in-tree build > instead. I don't know how to convert this into a real spec, though -- > like in general, if I'm a frontend and I call `hook(*args, **kwargs)` > and it says "can't do that", then how do I know what the problem is > and what I should try instead? > - What happens if prepare_build_metadata returns NotImplemented / > raises NotImplementedError? > - I don't understand how out-of-tree builds and prepare_build_metadata > are supposed to interact. > - Also, AFAICT the out-of-tree build feature has no motivation > anymore. The only mentioned use case is to support incremental build > features in automatic build pipelines like Travis, but there are a > number of these build pipelines, and lots of existing build systems > that support out-of-tree builds, and AFAICT no-one actually uses them > together like this. (I think it's because most build systems' > out-of-tree build features are designed on the assumption that there's > a human running the show who will deal with any edge cases.) I believe the out-of-tree build option Nathaniel is referring to is the build_directory parameter to build_wheel(). His proposed APIs remove that parameter, and elsewhere in his email he describes that no-one seems to want it: everyone thinks someone else wants it. By my understanding, the reasons for including the build_directory parameter are: 1. Without an explicit build directory, the developers of pip are concerned that build backends will modify the source tree and cause issues which users report as bugs to pip. 2. A frontend-controlled build directory could be used for caching intermediate build artifacts. This was a secondary argument after we had the idea, and we've never really fleshed out how we expect it to work. There's also a concern that if the first round of frontends always use an empty tempdir, backends will end up relying on that. I think this second argument is a weak one unless we spend the time to figure out the details. Do other people see the situation in a similar way? How might we move forwards on this? > - needs some sort of conclusion to the srcdir-on-sys.path issue. While Nathaniel is in the minority regarding srcdir-on-sys.path, he argues that most of us are assuming a default position without really thinking through it, which is certainly true of me. I don't feel strongly about this topic, but I do want to come to a conclusion. As Nathaniel does feel strongly about it, does anyone object to either: A. Leaving the source dir off sys.path when loading the hooks, and making a special backend which exposes hooks from the CWD. B. Adding another key in the TOML file to control whether hooks can be loaded from the source dir. I can live with either, but I prefer A, because it means fewer options to describe in the spec. Thomas

8 45

Fwd: Re: PEP 517 again
by xoviat Aug. 25, 2017

Aug. 25, 2017

---------- Forwarded message ---------- From: "xoviat" <xoviat(a)gmail.com> Date: Aug 24, 2017 8:39 AM Subject: Re: [Distutils] PEP 517 again To: "Thomas Kluyver" <thomas(a)kluyver.me.uk> Cc: That's actually the general argument against exceptions and why golang doesn't have them. I have not seen notimplemented used in the wild ever though. Also, regarding the build_dir argument, it won't be used in pip. The pip devs may have said that but it's too much work to support now and most of them seem to have a shortage of time. Pip will be copying to a temporary directory for a long time. On Aug 24, 2017 8:28 AM, "Thomas Kluyver" <thomas(a)kluyver.me.uk> wrote: > On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote: > > I mean is this golang or Python? In Python, you raise notimplementederror. > > > But there's a NotImplemented singleton in Python as well. The argument for > using a return value is that the hook code has to deliberately return that, > whereas a NotImplementedError could bubble up from some internal call, in > which case it should really be registered as an error. >

8 51

Fwd: Fwd: Re: PEP 517 again
by xoviat Aug. 24, 2017

Aug. 24, 2017

---------- Forwarded message ---------- From: xoviat <xoviat(a)gmail.com> Date: 2017-08-24 14:26 GMT-05:00 Subject: Re: [Distutils] Fwd: Re: PEP 517 again To: Thomas Kluyver <thomas(a)kluyver.me.uk> And also, are Nick and Paul okay with this? 2017-08-24 14:24 GMT-05:00 xoviat <xoviat(a)gmail.com>: > So in summary: > - NotImplementedError > - Remove build_dir and specify that no modifications can be made to the > source directory > > On the third issue: > > I understand where Nathaniel is coming from in regards to sys.path. But > there should at least be an option for this because of for example, numpy: > numpy builds itself with it's own build system that is defined in the > source directory. If you remove '' from sys.path, then numpy cannot build > itself because there is no other numpy.distutils package. > > 2017-08-24 14:00 GMT-05:00 Thomas Kluyver <thomas(a)kluyver.me.uk>: > >> On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote: >> >> It looks like a lot of trouble for a feature that is designed to solve a >> problem for a very thin intersection of the Venn diagram of people who: >> >> a. wants to quickly iterate while hacking a package >> >> b. doesn't want to bother learning how to trigger the incremental build >> system of that package or how to install it in editable mode. >> >> For better or worse, editable installs exist and don't need to be defined >> by PEP 517 just to handle this case. I'm not convinced it's worth the >> additional complexity of the spec just for that. >> >> >> This is a good point. Given that developers can use the build tool >> directly for development purposes, rather than going through the PEP 517 >> interface, we don't need to prioritise that use case for the spec. >> >> So for the time being, I'd prefer to say that the build_wheel() hook >> should not create or modify files in the source directory. Backends may >> keep their own caches elsewhere to speed up repeated builds, but any more >> provisions for developer use cases should be left for a future PEP. And >> developers can invoke enscons/flit/... directly in the meantime. >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG(a)python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >

1 0

Mirroring PyPI JSON Locally
by Cooper Ry Lees Aug. 24, 2017

Aug. 24, 2017

Hi all, First time emailer, so please be kind. Also, if this is not the right mailing list for PyPA talk, I apologize. Please point me in the right direction if so (Brett Canon pointed me here). The main reason I have emailed here is I believe it may be PEP time to standardize the JSON metadata that PyPI makes available, like what was done for the `'simple API` described in PEP503. I've been doing a bit of work on `bandersnatch` (I didn't name it), which is a PEP 381 mirroring package and wanted to enhance it to also mirror the handy JSON metadata PyPI generates and makes available @ https://pypi.python.org/pypi/PKG_NAME/json. I've done a PR on bandersnatch as a POC that mirrors both the PyPI directory structure (URL/pypi/PKG_NAME/json) and created a standardizable URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some other proxy). I'm also contemplating naming the directory 'metadata' rather than JSON so if some new hotness / we want to change the format down the line we're not stuck with json as the dirname. This PR can be found here: https://bitbucket.org/pypa/bandersnatch/pull-requests/33/save-json- metadata-to-mirror My main use case is to write a very simple async 'verifier' tool that will crawl all the JSON files and then ensure the packages directory on each of my internal mirrors (I have a mirror per region / datacenter) have all the files they should. I sync centrally (to save resource on the PyPI infrastructure) and then rsync out all the diffs to each region / datacenter, and under some failure scenarios I could miss a file or many. So I feel using JSON pulled down from the authoritative source will allow an async job to verify the MD5 of all the package files on each mirror. What are peoples thoughts here? Is it worth a PEP similar to PEP503 going forward? Can people enhance / share some thoughts on this idea. Thanks, Cooper Ry Lees me(a)cooperlees.com <me(a)copperlees.com> https://cooperlees.com/

6 13

PEP 517: Bootstrapping setuptools
by xoviat Aug. 22, 2017

Aug. 22, 2017

Previously, the attempt to move setuptools off of vendored dependencies failed because it was not done correctly: install_requires was set to the vendored dependencies but not setup_requires, which would have been required to correctly specify dependencies. However, setup_requires would have introduced circular dependencies that are impossible to correctly resolve, so that would have simply shifted an impossible problem onto pip. The principle issue then, is that setuptools requires setuptools to build itself. And although, this problem was previously not worth solving because of 'setup.py install', it is now not a difficult problem to solve if we rethink slightly what should be included in the setuptools respository: - Rather than re-generating egg_info on each setup.py invocation, we can cache dist-info in the setuptools respository. - We can implement a simple entry_points script that updates dist-info based on the contents of setup.py, which is only runnable of course after setuptools is installed - We can implement the reference build backend for setuptools: simply copy the files and dist-info into a new compliant zip archive Viola! Setuptools is no longer required to build setuptools, and the installation process is fully compliant with Python PEPs.

8 16